This book reflects a magnificent debate between highly regarded, serious researchers. It provides space for the authors to develop careful arguments in each chapter and the chapters provide a coherent sense for the weaknesses of experimental work circa 2009. As a greater preponderance of the chapters are leery of experimental work, this book could be a critical read for those scholars who do want to engage an experimental toolbox. It could be incorporated with a series of other readings such as Paul Rosenbaum’s Observational Studies, Alan Gerber and Donald Green’s Field Experiments, and papers by pro-experimentalists such as Guido Imbens (“Better Late Than Nothing”) to produce an interesting course on research design.

The arguments in the book are slightly dated, as the original conversation began from a debate that arose within the Yale University social science community in 2009. Seven years later, there is more consensus within the experimental community about the role experiments play in addressing a research question. For example, experimentalists have formed a professional organization titled Experiments in Governance and Policy (EGAP) which houses a study registry, allowing experimentalists to test interactions in a more principled way. Additionally, a growing number of researchers have studied ways to incorporate social networks into randomized experiments, addressing concerns voiced by Susan Stokes that “there can be no stopping rule, no point at which one has tested for all possible interactions” (p. 49). Indeed, the marriage of “radical skeptics” (Stokes’ term for the advocates of an experimental agenda above an observational agenda) to traditional political science has generated a host of new tools to address many of the concerns espoused in these chapters. The quality of inferences from experimental work has been increasing, moving to develop new statistical techniques to address some of the threats to causal inference pointed out in various chapters (i.e., heterogeneity, interference). That is, I would argue, “radical skeptics” have significantly improved our ability to draw inferences. Many more scholars are aware of the causal inference techniques, for example, described in Chapter 8 “Misunderstandings Between Experimentalists and Observationalists about Causal Inference.”

This is because the academic community is increasingly concerned about causal inference. In their chapter Alan Gerber, Donald Green and Edward Kaplan state that, “the risk of bias is typically much greater in observational research” (p. 11) and indeed while this is true, there has been a growing body of literature that has leveraged natural randomization (lotteries, natural disasters, etc.) to seize the opportunity to produce a causally-identified parameter from what is otherwise an observational study. In the discussion of experimental work, Dawn Teele emphasizes “experiments” as a methodology and writes with some concern about whether experimental researchers are inappropriately evangelizing and colonizing the discipline (i.e. “experimentalists’ tracts” p. 7). This mischaracterizes the adoption of experimental work: it’s proselytizers testify to the importance of causal identification, with randomized trial experimental work seen as one component of a revolution that recognizes endogeneity as one of the critical challenges of human subjects research.

The book is successful at bringing the issue of ethics to the table; an issue that experimentalists are still wrestling with and have yet to form a consensus on. As Teele describes carefully in her chapter on the ethics of field experiments, the current framework for evaluating whether an experiment is ethical is insufficient. Institutional Review Boards (IRB) are focused only on protecting subjects, not non-subjects. Teele’s illustration of this point engaged Marianne Bertrand et al.’s (2009) experiment on bribes for the procurement of illegal driver’s licenses in India. In this instance, the IRB did not consider whether the experiment, by providing an incentive for unqualified drivers, might harm other drivers or pedestrians. Experimental social science surely needs a set of more clearly delineated principles for the ethics of experimental work. These include cases where the current restrictions should probably be significantly relaxed, such as those where academics are working in partnerships with organizations or companies who are already engaged in experimental work. They also include cases where there should be more careful consideration of the broader impacts of experimentation, such as those instances where voter mobilization or persuasion experiments could affect the outcome of an election. Teele offers one particularly interesting suggestion, “to bring the participants in on the study’s purpose after the trial is over and to share research results after the report is finished” (p. 136). This could be accomplished by providing ungated access to papers published from randomized field experiments, for example.

The book highlights the central tension been authors: good research questions versus good research designs. The very best research in our field is able to find good questions to answer with research designs that provide some measure of causal identification. Yet, there are many instances where researchers must choose to emphasize one over the other, and in that case there are real stakes. Observational work, as traditionally conducted, has real limitations where there is very likely bias introduced from endogeneity. Biased findings can be totally wrong. Wrong findings can cause great harm and lead to poor social policies. The experimentalists provide a strategy to mitigate impact from “unobservable confounders” (p. 35), even when we allow ourselves to be “radical skeptics” and limit the scope of their power. The alternative, however, of choosing only causally identifiable questions is also limiting. We might sometimes prefer to study questions where we adopt heroic assumptions and risk bias rather than risk not answering these important questions. Medical science has sometimes made this choice; imagine a world where we had never studied the relationship between smoking and lung cancer, or children’s health and lead paint, because of the infeasibility of conducting a randomized trial. We have similar issues in the social sciences as well. As Christopher Barrett and Michael Carter write, “macroeconomic and political economy questions that many believe to be of first-order importance in development are clearly not candidates for randomization” (p. 73).

The punch line of this book might be: building positive knowledge is tough. Each of these authors takes seriously the charge that establishing meaningful findings in research needs careful and well-considered research design. Indeed, it appears there is more consensus in these chapters than would initially appear, and that many of these authors would agree with Stokes when she writes that “we need rich and variegated evidence, rigorously developed and analyzed, and considered in light of theories—which, as in all fields of science, are in part deductive in nature—if we are to gain knowledge about the workings of the social world” (p. 54). While experimental work has gained a strong foothold in political science, there are still those who have deep misgivings about its role. This book brings many of these concerns to the table in a clear and coherent fashion. From 2009 to 2016, we have seen experimental work play an increasingly prominent role in political science. This book challenges us to continue to think critically, radically, and skeptically about the role of experiments and the need for causal identification in research design.