In Novum Organum (1620), Francis Bacon offered a new method of “true induction, with the assistance of experiment” (p. 117) based upon a series of tables (of presence, of absence, and of comparison) which were early versions of John Stuart Mill’s “Joint Method of Agreement and Differences.” Through his method he hoped to reveal “forms, … nothing else than those laws and regulations of simple action which arrange and constitute any simple nature” (p. 148). Bacon’s use of his tables to understand the different forms of heat make amusing reading today (e.g., “rays of the sun,” “all shaggy substances, as wool, contain some heat,” and “horse dung, and the like excrement from other animals, when fresh” [pp. 121–22]), but his call for experimentation and induction helped to set western science on a new track away from scholasticism mired in syllogisms and reverence for Aristotle.

Have the modern proponents of field experiments considered in this book found the New Atlantis and the true method of induction? Like Bacon, they offer us a recipe for the advancement of science: random assignment of cases to a treatment and control group, manipulation of a putative cause in the treatment group, and withholding the treatment from the control group to see what happens in the “counterfactual” world without the treatment. The result, they argue, will be reliable identification of true causal effects in real-world settings, just as Bacon promised. Will it?

The all-star cast of Field Experiments and Their Critics provide quite different answers to four basic questions about field experiments.

Is randomization and experimentation the only reliable way to do inference? Two articles—one by Alan Gerber, Donald Green, and Edward Kaplan (GGK) and the other by Abhijit Banerjee and Esther Duflo (BD)—argue that only experiments can reliably control for confounding factors that will bedevil and ultimately discredit observational studies. GGK present an interesting but “cooked” mathematical result that begins by assuming that experimental studies are unbiased while observational studies always suffer from some unknown bias. Not surprisingly, they conclude that experimental studies are better than observational studies. BD run through all the ways experimenters can deal with their critics through the simple expedient of fielding even more experiments. Experimentation certainly has its virtues.

Are randomized control trials (RCTs) the gold standard, but perhaps usefully supplemented by other forms of research? Andrew Gelman calls RCTs the “gold standard” and plumps for “doing more field experiments,” but he goes on to say that “simple comparisons and regressions are and should be here to stay” (p. 185). He adduces two reasons. First even observational studies can be useful because “description is an important part of social science, both in its own right and in providing foundations upon which to build formal models” (pp. 185–86). This point deserves emphasis: Watson and Crick’s famous paper on the structure of DNA was merely description, but it was extraordinary science. Similarly, much of social science uses statistical methods (from the sample survey to regression analysis) to provide rich and reliable descriptions of the world that, just like Watson and Crick’s work, both suggest and restrict what is causally possible. Second, Gelman argues that it is wasteful and foolish to restrict the analysis of experimental data to just those analyses (often a simple difference of means test) justified by the randomized design. Certainly, Gelman argues, more information can be extracted by doing more statistical analysis, even if it is suspect.

Are experiments about as equally flawed as observational studies? Dawn Teele shows that experiments are gravely limited because they sometimes require manipulations that are unethical or undoable so that there is no way to avoid observational studies. Susan Stokes, Angus Deaton, and Ian Shapiro believe that experiments are flawed because they only provide “local average treatment effects” which do not take into account the heterogeneity in the world which means that individual cases might react quite differently to the same treatment. As a result, Stokes and others in the volume argue that the radical skeptic who distrusts all observational studies should be equally (radically) skeptical of experiments because their results can be biased. An experiment can show that on average a drug does not work because (unbeknown to the researcher) it interacts with some individual characteristic and makes half its recipients better and half of them worse off. Yet if we could identify the relevant characteristic, we could use the drug to improve people’s health. There is every reason to believe that heterogeneity like this is rife in social science. Finally, as Deaton argues forcefully, experiments inevitably require isolation and simplification that does not exist in the real world so that projecting a result beyond an experiment requires observational studies that consider general equilibrium effects, real-world implementation problems, and behavioral adaptations by people to changes in the world. In short, experiments may be undoable, misleading, or unprojectable.

Are there ways other than experimentation to do useful research? Stokes, Deaton, Kosuke Imai-Gary King-Elizabeth Stuart (IKS), and Christopher Barrett-Michael Cater (BC) certainly think so. Stokes states the essential issue: Radical skeptics believe that there are endless numbers of confounders that can undo observational studies, but she argues that the range of plausible confounders and interactions is not infinite. BC argue that substantive knowledge can rule out many possible confounding variables, and we should “retreat from radical skepticism and let theory and careful observation…guide an understanding of the causal process and name the potential confounds that can cripple inference from both observational data and from RCTs” (p. 76). Deaton agrees, and IKS provide a useful smorgasbord of alternative designs, tallying their strengths and weaknesses.

Where would Bacon stand on this? Would he applaud field experimenters as the true prophets of reliable induction? I don’t think so. Bacon depicted scientists as either “empirics or dogmatical,” and “the former like ants only heap up and use their store, the latter like spiders spin out their own webs” (p. 76). In this entomological competition, Bacon favored the bee who “extracts matter from the flowers of the garden and the field” and “works and fashions by its own efforts” (p. 76). The bee does not just collect facts or results like the pure empiric, and she does not simply spin out web after web of theoretical speculation like the scholastics. The bee extracts material from the world and refashions it. “The true labor of philosophy resembles hers, for it neither relies entirely or principally on the powers of the mind, nor yet lays up in the memory the matter afforded by the experiments of natural history and mechanics in its raw state, but changes and works it in the understanding” (p. 76). Bacon believed that scientists must link empirical data and theory to develop useful inductions. Science requires models and theory to discipline the ant-like heaping up of experimental facts. It requires facts to discipline the spider-like spinning out of theories. By extension, reliable empirical research requires both experiments and observational studies that blend empirics with theory. I’m with Bacon, science requires busy bees. And by the way, this is a honey of a book.