Think measurement is necessary? These authors say think again. In a blog post on the Stanford Social Innovation Review, Mary Kay Gugerty & Dean Karlan argue for a more tempered approach to measurement:
Ultimately, impact evaluations should be comparative. It isn’t just about whether something works, but also how to do the most good with scarce resources. Take school attendance for example. Anecdotal claims suggest that providing basic necessities, such as uniforms and scholarships, makes it more likely that children will attend school. But for the money, it turns out the that giving kids deworming pills is 28 times more effective than school uniforms and 56 times more effective than scholarships in increasing school attendance. However, getting credible information on impact is not easy—many organizations struggle to measure the results of their work, and often use methods and data that paint unreliable pictures of program success. We applaud the focus on impact, when feasible. But sometimes impact simply is not measurable in a credible way, and yet people (organizations, or perhaps their donors) push to measure it anyhow.
Instead of this wasteful data collection, the authors say, organizations should work to build appropriately-sized data-collection strategies and systems that demonstrate accountability to funders and provide decision makers with timely and actionable operational data.
For a forthcoming book, called The Goldilocks Problem, we developed a set of principles that all organizations—regardless of their ability to assess impact—can use to build strong systems of data collection. We call these principles the CART—credible, actionable, responsible, and transportable data collection.