Program evaluations provide answers to a key question: “Did this program work in a particular population and setting?” For funders who haven’t been trained in reading evaluations, wading through such reports can be confusing and time-consuming. So, to get the most out of program evaluations, here are 7 questions to ask when reading them:
1) How objective are these findings? The most rigorous evaluations are conducted by neutral third parties. Data collected by an external group, but analyzed in-house, is less rigorous but still informative. At a minimum, the evaluation should include in-house data monitoring and analysis.
2) Were baseline measurements taken? Recording baseline characteristics of program participants and related comparison groups allows for a “before” and “after” picture. For example, what was the prevalence of malaria in a given community before a program was started, and what is that prevalence after the program? If baseline data is not available, regional data may also be used as a benchmark.
3) How strong is the comparison? Rigorous research design compares program participants to people in a control group, who have similar characteristics but do not receive the program. This allows evaluators to answer the question: “If all other factors are held constant except for participation in the program, what impact do I see among participants?”
4) Was the sample size appropriate? The size and composition of a sample helps determine what researchers call “statistical significance.” Choice of sample size depends in part on the size of the effect anticipated from the program, as well as who or what the program is meant to affect; the larger the effect size, the fewer communities or individuals are required for it to be found statistically significant. If the effect size is large enough, even a small pilot of a dozen communities might be enough to help practitioners improve practice and determine whether the program is worth scaling up.
5) How objective and meaningful are the measures used? The most rigorous program evaluations measure out to ultimate impacts (e.g. child mortality), use both quantitative and qualitative research methods, and use surveys and indicators that have been externally assessed for their ability to measure the intended factor consistently across the affected populations. At a minimum, evaluations should measure project outputs (e.g. medications distributed) and quality of services, as well as identify possible limitations or biases if only one evaluation exists.
6) Are the study assumptions reasonable? Many if not most studies are done in the absence of perfect or complete information. Researchers should be clear where assumptions were made, and indicate why the assumptions can be considered reasonable and reliable. If different assumptions are both reasonable and likely to affect results, a good evaluation often includes a range, so that readers understand that actual results are likely to fall somewhere in the middle.
7) What evidence is there to suggest replicability? The more program implementations are measured in a diversity of sites with consistently positive results, the more confidence evaluators and funders can have that a program can be scaled up to new areas or groups.
Remember, even if an evaluation doesn’t meet the most rigorous standards, it doesn’t mean that it still can’t be useful. Even evaluations that are less rigorous can provide helpful information to funders, such as the quality of services provided or whether the program, as designed, is consistently producing the intended outputs. Many nonprofits often don’t have the resources – including dedicated funding, sufficient capacity, or existing data to build upon – to conduct such evaluations. If the program evaluation you seek doesn’t yet exist, perhaps it’s time to consider funding one.
For more on measuring impact and supporting program evaluations, see Beyond Compliance: Measuring to Learn, Improve, and Create Positive Change. For a downloadable ‘cheat sheet’ on what to look for when reading program evaluations, click here: Cheat Sheet- Reading Evaluations