Title

Towards a multi-dimensional model of impact evaluation quality: assessing development impact evaluation methods with respect to context

Abstract

In this thesis, I give an account of the quality of impact evaluation methods for development interventions that goes beyond internal validity considerations to also incorporate the transferability of findings to new contexts. To investigate how well different impact evaluation methods facilitate transferability, I adapt tools from realist synthesis to extract and synthesise the programme theories that underpin a set of evaluations of the same intervention-outcome pair. From this synthesis of programme theory, I derive the markers of intervention causation in context (MICCs) that an impact evaluation of an intervention of that type would have to report to facilitate the transferability of findings. I systematically build a complete set of impact evaluations for two intervention-outcome pairings, and apply my method to these two cases. This generates case-specific insights such as identifying evidence gaps where minimal further data generation offers large gains in understanding. Further, the analysis generates cross-case insights such as the tendency to better report causally significant features of intervention implementation than causally significant features of context. Most importantly, the analysis suggests there is no association between method choice and the facilitation of transferability, in theory or in practice. I argue that we can nonetheless improve on ‘there is no gold standard’ by showing how generating a middle-range theory of intervention causation capable of underpinning the list of MICCs for a type of intervention provides a guide to evaluation method choice and to transferring results between contexts.

In parallel, to render my main results more useful to the relevant experts and therefore more likely to influence practice, I conduct semi-structured interviews with development intervention evaluation experts. I identify a broad discursive trend in favour of theory-based evaluation and a nascent interest in the use of middle-range theories to underpin transferability. I therefore frame my main results in these terms.

Download PDF