Evaluating for impact: 3ie comes to ICRAF

Billions of dollars are spent on development projects every year, but what difference has all of that money actually made? The answer is less clear-cut than you might think.

Questions about the real impact of aid money were the driving force behind the establishment of the International Initiative for Impact Evaluation (3ie), Dr. Jyotsna Puri, the organization’s Deputy Executive Director and Head of Evaluation, said at a seminar held on ICRAF’s campus on 3 September. (Slides are available here.)

Puri said that 3ie, which was founded in 2008, has already become the world’s biggest funder of impact evaluations. In its first seven years, the organization  - which gets its funding from the Bill & Melinda Gates Foundation, the British government, and the William & Flora Hewitt Foundation – has awarded more than $84 million in grants, covering projects in more than 50 countries. All of those grants are meant to help realize 3ie’s vision, which is to “improve lives through impact evaluation.”

Nuanced analysis of programme impacts is critical to maximizing the effectiveness of aid money, Puri said. She offered an example of a study that she had conducted in northern Thailand, where she assessed whether the establishment of protected areas had actually reduced rates of deforestation.

It might seem obvious that setting up a conservation area would help to protect trees, but reality, Puri said, is more complicated. In northern Thailand, the forests that are chosen for protection are usually those that are unsuitable for farming, which is the main driver of deforestation in the region. Thus, when she accounted for variables like soil quality, landscape slope, and distance to markets, it turned out that many of the forests that have been protected were never threatened by deforestation in the first place.

“It’s incredibly important to account for selection bias when you’re assessing the effectiveness of protected areas,” she said.

Working with farmers

Dr. Diana Lopez, 3ie’s Evaluation Specialist, next discussed some of the lessons the organization has learned from working with smallholder farmers in sub-Saharan Africa and South Asia (from slide 30). Some of the challenges they have faced, she said, include accounting for the ‘contaminating’ effects of nearby projects; measuring the synergies of different interventions; and ensuring that women are adequately represented.

Lopez stressed the importance of doing qualitative studies before beginning an intervention. Such research can help the project managers understand how their work will impact the targeted group, and help them predict potential spillover effects. A baseline pilot survey can also be useful, she said, for refining the design of data collection instruments and identifying the constraints that farmers may face in participating in the project.

Tracking impact

Stuti Tripathi, 3ie’s Senior Policy and Evidence Uptake Officer, next offered a presentation on how 3ie measures the impact of its own work (starting from slide 36). 3ie focuses on generating evidence that’s useful, she said; in order to be useful, the information must be relevant, contextual, demand driven, and clear, and it must present its audience with a feasible option.

“We don’t want the research we fund to sit in a file that’s only ever read by five people,” she said. “3ie funds high-quality, policy-relevant studies.”

Stuti described the kinds of influence that 3ie tracks: expansion of successful programmes, closure of unsuccessful programmes, changes in programme design, informing dialogues or the design of other programmes.

Karl Hughes, ICRAF’s Head of Monitoring and Evaluation, told the group that he sees three clear areas of synergy between 3ie and ICRAF’s work: using geospatial data to conduct assessments, bringing research into the development process, and tracking the influence and impact of research.