Introduction

The gold standard of applying fully experimental methods is not open to all, or many, impact evaluations, especially in social settings. Where there are barriers to RCTs, it is widely possible to use quasi- experimental designs (QEDs) applying well- chosen comparison groups instead of control groups. As their name implies, quasi- experimental approaches are not as rigid as RCTs in their requirements and set- up. With fewer constraints to the circumstances where they can apply, they offer much greater choice and flexibility for purposeful evaluation.

Making that choice, however, comes with a need for sound judgement across the many QED options open to impact evaluators. QEDs may lack the counterfactual precision of fully experimental methods, but sound choice and application of a robust QED approach can offer a level of proof of effect sufficient to satisfy many user demands for quantifying attribution. In practice, different options present different trade- offs between flexibility, simplicity and reliability, and contrasting pros and cons. This chapter introduces many of those options, their merits and challenges to help evaluators make a ‘best choice’ of this high- utility approach to counterfactual insight and analysis.

What are ‘natural experiments’?

QEDs are sometimes grouped together as ‘natural’ experiments. This term cannot cover all QED options, but it serves to separate the constructed ‘controls’ (that is, non- treatment groups) of RCTs, from the use of available ‘natural’ comparators, often adopted from outside an intervention. An evaluation asked to look at the effect of a pilot intervention on a disadvantaged group in one location might look for a comparison from a similar needs group in a close- fit ‘comparator’ geographical area where the intervention was not taking place.