Alan Gerber, Donald Green, and Edward Kaplan begin their volume, Field Experiments and Their Critics, with a forceful argument in favor of field experiments. They derive the “Illusion of Observational Learning Theorem” and then discuss the implications for the efficient allocation of research resources. They conclude that it generally makes sense to allocate all of one’s research resources to experimentation. Only when the costs (financial or ethical) of experimentation are extremely high and good-quality observational data is readily available should researchers invest in observational research. They throw down the gauntlet: Either you’re doing experiments, or you’re probably wasting your time.

The remaining authors mostly ignore the barb in this argument. None of them deny that field experiments can give us an excellent combination of internal and external validity—if they are appropriate and well-done. Hence, the core debate of the book is about the circumstances under which field experiments are appropriate and the extent to which they can be done well.

Abhijit Banerjee and Esther Duflo (Chapter 4) thoughtfully defend field experiments from several common critiques and outline how field experiments (mostly program evaluations) have advanced knowledge in development economics. But others in the volume are less convinced. Susan Stokes (Chapter 2) and Christopher Barrett and Michael Carter (Chapter 3) question how well experiments can actually be done and make note of problems with the interpretation of their results. Angus Deaton (Chapter 6) echoes these concerns and asks whether focusing on experimental methods causes us to focus on the wrong questions. Dawn Teele (Chapter 5) raises hard questions about ethics and the appropriateness of field experimentation, and though I do not agree with every point she makes, this chapter should be required reading for all field experimentalists.

The final three chapters try to mediate the conflict. Andrew Gelman (Chapter 7) touches on a number of important points: Uncovering causation isn’t the only reason to look at data, and randomized controlled trials aren’t the only research designs that use experimentation. Kosuke Imai, Gary King, Elizabeth and Stuart (Chapter 8) outline the ways in which observational and experimental researchers misunderstand each other and causal inference, and they offer helpful, practical advice. Finally, Ian Shapiro (Chapter 9) closes on an ecumenical note. All methods have strengths and weaknesses. Methodological pluralism is and always will be our best approach because it keeps the focus on the questions we are asking.

Even as a strong advocate for field experiments, Shapiro’s essay resonated deeply with me. Good questions form the foundation of social science research. Methods are crucial, but they must be secondary to questions or else we risk wasting time in a drunkard’s search. In this way, the observational vs. experimental debate echoes the old qualitative vs. quantitative debate. What is the best method? It depends on the question. Even Gerber, Green, and Kaplan ultimately give that answer, albeit somewhat grudgingly.

My main critique of the volume is that it neglects the practical aspect of field experiments that helps place them in methodological context. Randomized controlled trials are exceptional at answering the following question: Did this exact intervention change outcomes? That part is straightforward. The hard part about experimentation is designing interventions in such a way that the answer to the question is interesting regardless of what it is. Yes, it is technically true that you can learn something from any careful, well-powered, experimental intervention. But obviously the goal is to learn something useful.

I don’t bring this up to legitimize the flippant argument that one can only do experiments on topics that don’t matter. That is clearly not true. I bring it up because field experimentalists need to be honest about the debt they owe to observation as they design their interventions.

Pursuing an experimental research program is like engaging in a very expensive game of “Twenty Questions.” You present an experiment with a clear and precise hypothesis—a best guess generally rooted in existing observational research. If all goes well, the experiment answers back with a clear and precise yes or no. In the next round of the game, you know a little bit more (usually about what doesn’t work!), but you still have to come up with a new intervention and hypothesis. During this costly iterative process, experimentalists may be skeptical of observational findings, but they would be foolish to simply dismiss or ignore them.

In short, the best field experiments are creative and carefully implemented, but they are also well-informed by theory and cultural context. Theory and cultural context mostly remain the domain of observational research. Thus, taking an inclusive approach to methods is practical and wise for those engaging in field experimentation.