Mixing methods for holistic project evaluations

What is the value of integrating qualitative analyses in quantitative project evaluations? According to a group of researchers from Bioversity International, World Fish and Nepalese NGO LI-BIRD, qualitative research is not only valuable, but in fact essential for holistic project evaluations.

In the brief ‘Mixing Methods for Holistic Project Evaluations: Revisiting Nepal’s Home Garden Project through a Qualitative Lens’, researchers Marlène Elias and Miranda Morgan argue that whereas quantitative methods shed information on trends expressed among project beneficiaries and non-beneficiaries, qualitative methods are needed to explain why and how a project has achieved the observed impacts.

The researchers explain that qualitative (process) evaluations help contextualize projects within their larger systems, pointing to various factors external to the projects that come together to facilitate changes in livelihoods or in the environment. In the case of the ‘Home Garden’ project in Nepal, these factors included the village’s acquisition of electricity, water pumps and mills, male outmigration and girls’ education, among others—without which the changes observed may not have come about.

The qualitative analysis further illuminated unanticipated project outcomes, which participants themselves perceived as meaningful. For instance, both male and female participants expressed that some of the biggest changes they experienced were associated with relations across castes (socio-religious groups in Hindu societies), which had not been anticipated and hence not measured using pre-defined indicators. The qualitative evaluation revealed how and why the project was particularly significant to certain participants, such as women from marginalized castes, and captured the importance of the project in participants’ own terms. The project’s intellectual and emotional benefits then emerged as even more significant to participating women and men than the material outcomes of the project. The qualitative research also showed that transformations in gender and social relations often occur when the various groups enmeshed in unequal relations (e.g. women and men, members of different ethno-religious or socio-economic groups) are involved in projects together and encouraged to interact.

The authors conclude that despite the limited resources and sample size available for project evaluations, it is possible to elicit critical qualitative information on the impacts of a project by conducting a limited number of in-depth key informant interviews and focus groups with project participants and non-participants. They stress the importance of working with both women and men from different social groups—ages, socio-economic, ethnic or socio-religious groups—to gain perspective on the often distinct experiences of these groups. When doing so, the combination of quantitative and qualitative methods allows for a clearer understanding of a project’s impacts and effectiveness. If the purpose of evaluations is not only accountability but also learning, mixing methods is needed to enable the wide and deep understanding needed to can guide future interventions.

Read the full brief ‘Mixing Methods for Holistic Project Evaluations: Revisiting Nepal’s Home Garden Project through a Qualitative Lens’

The first phase of the 'Home Gardens' project was conducted by Bioversity International, LI-BIRD and the Swiss Agency for Development and Cooperation between 2002 and 2005. In 2014, researchers from Bioversity International’s CGIAR Research Programme on Forests, Trees and Agroforestry, WorldFish researchers from CGIAR Research Programme on Aquatic Agricultural Systems and scientists LI-BIRD headed to the field in Nepal’s Gulmi District to examine the processes of change in gender relations that can result from biodiversity research for development projects.

^{Photo: A group of men take part in fieldwork, Nepal. Credit: Bioversity International/M. Elias}