1. WHAT EVIDENCE FOR EXTERNAL VALIDITY?

A peculiar feature of MEE is the amount of space it devotes to a problem that is not very high in the list of economists’ and philosophers’ priorities. The book is organized in two parts (‘Inferences within the experiment’ and ‘Inferences from the experiment’), corresponding roughly to problems of internal and external validity. But while the literature on the former topic is enormous in experimental methodology and philosophy of science, the latter has received relatively little attention. Have I devoted excessive space to a pseudo-problem?

I think not, and other authors who have recently published on the same topic concur that external validity is a genuine problem for the social sciences.Footnote ¹ But others remain unconvinced.Footnote ² The disagreement has to do in part with the thesis, defended in MEE, that ‘experiments cannot be validated by experimental means’ (248).Footnote ³ ‘Valid’ here means externally valid: laboratory data are insufficient, on their own, to establish that the results of an experiment apply also in different circumstances.

I still stand by this claim, but unlike Jones I do not think that it is a radical view. Taking it seriously would not have revolutionary consequences for the way experimental economics is done for example. In fact I think that this view captures experimental economists’ best practice. So clearly Jones and I disagree on what this practice is.

In MEE I argue that all external validity inferences – the inferences economists actually make, not the ones they should make according to some abstract philosophical theory – rely on a combination of laboratory and non-laboratory (field) data. To substantiate this claim I illustrate and analyse various concrete cases that exemplify how different sources of evidence complement each other (Guala Reference Guala2005: ch. 7–10). Since Jones does not explicitly criticize my reconstruction of these cases, I assume that he agrees that combining field and experimental data is often useful. His point is that it is not always necessary: in some cases experimental data can take you all the way from the laboratory to the outside world.

Jones points out correctly that many external validity worries are addressed by ‘incorporating the missing factors’ in a new experimental design. Suppose an experiment has demonstrated that, beyond reasonable doubt, X causes Y in circumstances C1. Still, one may be worried that in other circumstances (C2) X may fail to cause Y, due to the interaction of a factor (let us call it K) that disrupts the causal link. An obvious way to dispel this worry is to run a new experiment adding K to the original design, and see what happens. If nothing changes, then the result of the original experiment has been shown to be valid outside the narrow context C1. The external validity of ‘X causes Y’ has been demonstrated, by purely experimental means.

My disagreement concerns the final gloss: ‘by purely experimental means’. The issue is where K (the missing factor) is coming from. Experimental economists spend a lot of time thinking about what to include in their designs. Since many experiments are variations of previous studies, they also spend a lot of time thinking about what others have excluded from their experiments. In doing so, they have to identify factors that might matter in a given situation. Sometimes theory tells them that some K is likely to make a difference. Economic theory abstracts many causal factors away, however, so experimenters must also rely on common sense or on alternative theories (from psychology, sociology, etc.) that mention potential difference-makers. But wherever it comes from, no ‘missing factor’ is taken seriously, if economists believe that it is unlikely to be instantiated in real-world circumstances that constitute the intended domain of application of the theory or the experiment.

Consider an example from decision theory, discussed in MEE (Guala Reference Guala2005: 224–229). At one point, during the long controversy on preference reversals, some experimenters demonstrated that repeated arbitrage drastically reduces the frequency of reversals (Chu and Chu Reference Chu and Chu1990). This result is interesting – and received attention – because arbitrageurs are at work in several important contexts that fall in the domain of economic theory (e.g. financial markets). So the experiment suggests that in these markets the preference reversal phenomenon is unlikely to be frequently instantiated. But reaching this conclusion requires a complex set of premises: these include Chu and Chu's experimental results of course, and also, crucially, the claim that arbitrage takes place in a number of real-world markets. The latter claim, presumably, is supported by independent observations of these markets. So external validity claims are always supported by a combination of experimental and field data.

One may still say that this example is unrepresentative: there are cases where no field data are required. I predict that such cases would fall in one of two categories. Either (1) the existence of the ‘missing factor’ in (some) real world situations is so obvious and acknowledged that it does not require any argument in support. Or (2) the factor is considered a theoretical possibility, even though it has not been observed in a specific domain yet. By adding it to her design, the experimenter is just checking whether it is something that we should worry about, if one day we will apply the experimental results in a real-world context.

This is what I called ‘robustness testing’ in MEE (ch. 10), a procedure that contributes a body of background knowledge for policy intervention or application (the ‘library of phenomena’ as I have called it). Jones’s argument may be interpreted as challenging my distinction between robustness and external validity: in both cases the experiment generates knowledge that is potentially applicable, and in both cases it draws on background information concerning the properties of real-world systems. At such a general level of analysis, I agree that the distinction may appear to be just verbal. But I insist that these two activities – robustness and external validity checking – are different enough to deserve different names. Establishing that a particular experimental result is non-robust to factor K (as in Chu and Chu's case) is not the same as proving that it is externally invalid: preference reversals may still occur in a wide set of circumstances, for example, where K does not hold. Conversely, establishing that an experimental result is robust to K does not imply that it has external validity: the experiment may exclude some other factors that are always instantiated in the real world. The distinction between robustness and external validity is useful to the extent that it reminds us of these simple logical points, and it prevents hasty inferences from the laboratory to the field.

2. ANALOGY AND COMPLETENESS

The structure of external validity inferences was modelled in MEE as a species of analogical reasoning. Daniel Steel (Reference Steel2008, Reference Steel2010) has shown that the analogical account can be expressed entirely – and more rigorously – in the formalism of Bayes’ nets theory. I think that this is the best account of external validity that is currently available (especially in Steel's version) so I will try to defend it from Jones’s critiques.

Jones finds two main problems with the analogical account. First, every external validity inference is necessarily local: if it involves an analogy between a particular experiment and a particular field system, it will be impossible to extend its conclusion to another field context. In Jones’ words: ‘the original analogy between the experiment and the field study tells us nothing about the other part of the external world without bringing in another analogy’ (266).

Unlike Jones, I do not find this conclusion particularly repugnant. In fact I think that caution in generalizing scientific results is one of the important lessons of the philosophy of experimental practice (Guala Reference Guala2005: 200–202). Solid scientific knowledge is built slowly, step by step, rather than by grand theoretical conjectures. Rushing to make general claims from experimental data is a grave mistake, in my view, which has already caused some experimental economists unnecessary trouble (see e.g. Guala Reference Guala2012). The experimental method is a powerful method, which works best in partnership with other methodologies such as theoretical modelling, simulations, econometrics and historical analysis.

Jones’s second argument against analogical inference targets the completeness principle. Completeness requires that all potential confounding factors should be controlled before drawing any conclusion. In MEE I mention the completeness requirement while discussing an argument known as ‘blame the theory’ (Starmer Reference Starmer1999): if an experiment implements all the initial conditions of a theoretical model, the experiment cannot be blamed for ‘missing a factor’; the theory is incomplete, and should be blamed for it. In MEE I say that such an argument sets unrealistic standards, because no theoretical model can be expected to include all causally relevant factors. Experiments are useful precisely because models are often incomplete; but then experiments can be blamed for missing factors just as much as theoretical models. No one is immune from ‘missing factor’ critiques.

Jones charges me of inconsistency, for I allegedly appeal to completeness when I analyse external validity inferences in terms of analogical reasoning: if you reject completeness when you discuss ‘blame the theory’, then you cannot introduce it when you model external validity inferences. But analogical reasoning requires completeness, so analogical reasoning cannot work (266). However, as I shall explain shortly, I do not endorse the completeness requirement, so the argument does not take off.

This is not the first time that I have been accused of requiring completeness (Steel Reference Steel2008: 149–153; Bardsley et al. Reference Bardsley, Cubitt, Loomes, Moffatt, Starmer and Sugden2010: 223; see also my reply to Steel in Guala Reference Guala2010), so I must take responsibility for the misunderstanding. In MEE I claim that extrapolation inferences ‘cannot be strong unless experimental and field evidence have been generated by systems that are similar in all relevant respects or, in other words, unless all sources of external validity error have been taken care of by means of accurate design’ (Guala Reference Guala2005: 199; emphasis added). The key word here is ‘relevant’. Jones (259: n. 6) reads it as ‘referring to all factors covered by the ceteris paribus conditions’. But the intended meaning was ‘relevant given what is known about the target domain’.

The confusion regarding what is ‘relevant’ can be resolved looking at the theory of induction defended in Part One of MEE. A key idea in Part One is that it is useful to distinguish between objective and subjective notions of support.Footnote ⁴ An objective analysis outlines the conditions for a piece of evidence e to support a hypothesis H independently of what is known or believed about e, H and the background conditions K that help us draw the inference from e to H. Its main contribution is normative – it allows the identification of what ought to be done and ought to be known in order to design a good experiment or to make a strong inductive inference. But when we judge whether an inference is warranted or not, a subjective element is necessarily involved: in practice, one can only check the factors that are deemed relevant by one's current background knowledge.Footnote ⁵

Studies of scientific practice reveal that experimental results are accepted by the scientific community when experimenters have shown that all the factors that may lead to error have been controlled for. But the universal quantifier should not be read metaphysically as ‘all possible sources of errors’ (some of them may not be even known) but rather as ‘all the errors that the community of scientists working in that area consider worth checking’. The list is necessarily finite and to an extent will be determined by conventional and sociological factors (as well as by past experience, of course; see Guala Reference Guala2005: 130–132). Interpreted objectively, the requirement that the experimental and target systems ought to be similar in all causally relevant respects would indeed be unreasonable. But interpreted subjectively (or ‘sociologically’), the completeness requirement looks much more reasonable indeed.Footnote ⁶

So I do not advocate double standards for external and internal validity inferences. On the contrary, it is important that they are both consistent with the same general principles of inductive reasoning, as I say explicitly in MEE:

3. ECONOMICS AS A NON-LABORATORY SCIENCE

The expression ‘non-laboratory science’ was coined after Ian Hacking's (Reference Hacking and Pickering1992) characterization of experimental physics, chemistry and biology as ‘laboratory sciences’. In a laboratory science, research can be driven entirely by questions that arise in the lab and are answered by data collected in the lab, with little regard for what happens in the ‘outside world’. Notice that Hacking does not say that physicists or biologists or chemists are not concerned with non-laboratory phenomena – he merely claims that it can happen: ‘pure’ experimentation is a legitimate style of research. Physicists do not blame each other for not sticking their noses out of the laboratory.

In sociology, anthropology and political science, in contrast, you can be blamed for that. In economics you were indeed blamed until recently, as the first generation of experimental economists can testify. Experimenters of the second and third generation have become a lot more specialized, so economics may be in the course of becoming a laboratory science. The interesting question is whether this is a progressive move or not.

Physics is not just ‘pure’ theoretical and experimental physics, but also applied science and engineering. A sharp separation is sustainable as long as pure science informs applied science every now and then, and the latter continues to produce efficacious technology. Presumably Hittorf, Goldstein and Thomson had no idea that cathode rays would allow people to watch football on TV, but there is no doubt that we owe them something for this wonderful technology.

However, the step from pure science to application is relatively small in this case: an old-fashioned TV set is little more than a cathode ray tube in a box. The transfer of experimental knowledge is easier when laboratory devices can be shielded, boxed and sold at the counter. Unfortunately not all science is like that. Jones cites the example of cosmology, claiming that knowledge gained in experiments is routinely used to explain phenomena in the ‘outside universe’ (264). This is true but not relevant, since I have never claimed that experimental knowledge in physics or biology always remains confined within the laboratory. Like Hacking, I only said that this can happen because physicists or biologists (unlike economists) are not constantly asked to demonstrate the external validity of their experimental results.

Jones presumably would say that the attitude of physicists is sensible, and that my position places too heavy a burden on experimental economists. I agree with him that ‘pure’ science is not to be criticized as such, but at the same time I invite economists to take physicists’ and biologists’ grand claims with a pinch of salt. The application of ‘pure’ theoretical and experimental knowledge is usually more complicated than scientists suggest in press releases and popular books, and it typically relies on a large amount of information collected in the field. The chapter devoted to the design of the FCC auctions was meant to illustrate these complications in detail, Footnote ⁷ but the point holds outside of economics too. Physicists know that superconductivity is hard to come by in non-artificial contexts, and biologists are well aware that the Drosophila melanogaster of most experiments is not the same as the fruit fly floating over the bananas in your kitchen. This is particularly important when it comes to application, because the stakes are higher: scientists at NASA do not send probes into space based just on theoretical models and a few laboratory experiments.Footnote ⁸

Economists’ power of shielding and intervention is probably close to that of astrophysicists or evolutionary biologists. Social scientists have to work with large and unconstrained systems, heterogeneous subjects and politically charged issues. For this reason, it is useful to keep reminding ourselves that ‘pure’ laboratory science is just a preliminary step in the path towards useful scientific knowledge. We must keep in mind that applied science is a messy and context-specific business, and that experimental results are transferred in the field with much labour and difficulty. This will make sure that experimental research will not lose contact with the problems economists are expected (and remunerated) to solve. It will also prevent economists from making premature claims about the applicability of experimental findings in real-world contexts. And it will encourage the creation of teams of theorists, experimenters and applied scientists (‘engineers’) that constitute the file and rank of successful scientific disciplines, but which have been culpably ignored by most philosophers until now.