Experimental Economics is a discipline that has reached a critical point in its development. It has reached maturity in the sense that it is widely accepted within the economics discipline and its results inform large areas of economic analysis. However, it is still unclear what the results of given experiments actually mean and how far we can trust them. There has been a substantial amount of methodological analysis but much of it has been defensive, aimed at justifying experiments to economists. Furthermore, much of the extant methodological analysis is philosophically outdated since it relies on a theory-testing paradigm that has little relation to how economists (and other scientists) do experiments and how results are established.
For these reasons Ana Cordeiro dos Santos’ book is a welcome newcomer to the debate. It uses current ideas in the philosophy of experiment and tries to extend them to account for the social aspects of experimental debate. While this approach has its problems, I believe that most of the fundamental points in the book are basically correct and any methodological debate should take account of its arguments.
According to Santos, experimental facts are generated through two general processes: a material process and a social process. The material process is the confrontation of the experimenter's ideas with evidence generated through experimental procedures. This is actually a complex three-way process involving the ‘phenomenal model’ – the experimenter's picture of how the world works, the ‘instrumental model’ – the experimenter's view of how the experimental apparatus performs and the ‘material procedure’ – the actual, physical experiment and its results. The latter includes the experimental subjects as well as the instruments used such as computers and instructions to the subjects.
The problem, from the experimenter's point of view, is that the material world (especially, in economic experiments, the subjects) ‘resists’ the experimenter's preconceptions both of how the world works and how the experimental apparatus performs. The experimenter is forced to intervene or to manipulate the material in the experiment to try to understand how the experiment is actually working. This involves removing artefacts from the experiment while trying to allow causes of experimental phenomena to remain. This means that one not only changes the material procedure but one must also adjust one's phenomenal model and instrumental model to account for why these changes have to be made. Eventually this process of change will reach a type of equilibrium where there is ‘three-way coherence’ between the three parts of the experiment.
One immediate implication of material resistance is that the more potential there is for ‘resistance’ then the more an experiment tells us about the real world. This is referred to as the ‘materiality’ of the experiment. Conversely, the more controlled is an experiment then the less informative it is. In experimental economics this implies that subjects should have as free a choice as is possible within the confines of the experiment. If this happens then belief in an experiment's results will be well-founded or, in other words, the experiment is ‘epistemically superior’.
The social process involves what happens after the material process has reached coherence. Following this, experimental results may still be biased because of the prior expectations of the experimenters. This is because there may be multiple possible interpretations of the results or they may seemingly clash with other results. This means that researchers have to present their results to the wider community, marshalling arguments to support their own point of view. Other researchers will involve themselves in criticism of these arguments until all questions have been answered and a consensus achieved.
This process of wider criticism is a process of ‘social resistance’ that aims to overcome the experimenter's innate subjective biases and to make the final result more objective. In general, the wider the backgrounds of those criticizing the experiments then the more objective the results that emerge. Once this social validation has taken place the experimental results become established as facts.
Santos also introduces another notion in response to Francesco Guala's (Reference Guala2005) ideas on External Validity. In her view, Guala's conditions for external validity are too tight for experimentation to be a useful technique. Very few experiments, in her view, would correspond as tightly with concrete external situations as would be necessary. In response she formulates a looser idea of ‘generic inference’: while experiments may not match up with concrete situations in the outside world as specified by Guala, they may match up to a class of situations. Experiments still tell us something about the world because of this looser matching up even if they don't match all details.
The framework described above is used to make some powerful points about experiments in economics. One is that the level of materiality in experiments is important. In many experiments (such as double auction market experiments) subjects are highly constrained in the choices they can make. In such experiments market demand and supply are already defined and the only uncertainty is whether subjects reach equilibrium in the right place. Such experiments have low materiality since they are heavily controlled and should not be seen as providing strong evidence in favour of market efficiency.
The framework also casts doubt on at least some experimental techniques. Many experimentalists (e.g. Plott Reference Plott, Arrow, Columbatto, Perlman and Schmidt1995; Binmore Reference Binmore1999) have claimed that stationary repetition is an essential part of an experiment as this enables subjects to gain information about the situation. However this technique also pushes subjects in a particular direction by imposing incentives repetitively. In such a situation the subject is effectively being taught how to do the experiment ‘properly’ in a situation (repeated trials of the same choices) that often simply doesn't exist in the outside world. This reduces the materiality of the experiment and hence its epistemic value. In effect the imposition of repeated trials means that the experiment is less useful in telling us about how people choose.
Also useful is the related set of criteria that the author gives for making sure that we do not control an anomaly out of existence. These are worth setting down: (i) An experimenter must elicit the relevant set of motives that drive subjects’ choices; (ii) The range of choices must allow the expression of these motives; (iii) The range of choices must be effective in bringing about the intended outcomes. An experiment that excludes certain motives altogether, which imposes large penalties for having these motives or which makes these motives meaningless is not a useful behavioural experiment.
Finally Santos makes the useful point that the so-called trade-off between external and internal validity (see Schram Reference Schram2005 for an example) is not as obvious as might be thought. The idea behind this trade-off is that the controls imposed as a result of trying to elicit the desired results i.e. creating internal validity, also creates restrictions on an individual's choices that would not exist externally. However, as Santos points out, this falsely sees controls as purely restrictive. In fact controls are used to create institutions within the laboratory and, insofar as they mimic the external world, actually increase external validity. In fact external validity requires internal validity as a prerequisite. Anything that is not internally valid cannot be externally valid either.
Insofar as the author emphasizes these points, this is a very valuable book and is important for these reasons. It is in this light that I would like to make the following criticisms, which should not be seen as detracting from the overall quality of the work.
My first worry is about the positive aspects of the Social Epistemology of Experimental Economics. I agree with the author that experimental results should be materially and socially robust. However this doesn't seem to be factually correct. As the author herself has pointed out, there are many examples in experimental economics where experimental results are materially quite weak. Even worse, it is hard to see in many cases whether social robustness has any real bite.
Economics is a notoriously conservative discipline and this conservatism is itself a bias. The problem is that this conservatism could overwhelm the experimental enterprise and result in over-controlled experiments that simply confirm results that theorists have previously deduced. Indeed it is arguable that the so-called ‘Discovered Preference Hypothesis’ of Plott (Reference Plott, Arrow, Columbatto, Perlman and Schmidt1995) and Binmore (Reference Binmore1999) is simply an exercise in over-controlling until the ‘correct’ results are obtained. In effect, the lack of social robustness has resulted in a lack of materiality.
This is a quite a general phenomenon across all sciences. One set of results may be seen as valid because the preconceptions of the scientists involved are in accord with them, while another set of results is rejected because they are not. If these biases are predominant, or overwhelming, within a discipline then it is hard to see how a test for social robustness can be applied. Since outsiders may not have the technical knowledge to challenge the results, it may be that some patently unsound results may survive, and be given a high degree of epistemic value. Formally, Santos’ social robustness test may have been applied in that an experiment is exposed to criticism but the results may still be wrong.
A second criticism is related to Santos’ treatment of Guala. It is obvious that she does not agree with Guala's conclusions on the external validity of experiments. However, as has already been mentioned, she locates the source of the problem in the alleged tightness of the requirements for external validity. External validity is only assured when experiments are analogous in all crucial respects with concrete target situations. Her solution is to bypass the issue by creating the new category of ‘generic inference’ that more loosely corresponds with the external world.
In my view this simply doesn't give enough credit to Guala's main argument. For Guala, the reason why external validity is so difficult is because laboratory experiments are always going to be incomplete in terms of the factors explaining choices in experiments. This problem means that experiments cannot be created which mimic the outside world by introducing all relevant factors – there will always be excluded factors. This means that one cannot establish external validity by experimental means as extra, excluded factors may change behaviour in the external world. Instead, in order to establish external validity one has to make an analogy with a field study in the external world. The field study then gives the experiment the necessary external validity.
Santos’ proposal simply does not deal with this problem. It is not clear how generic inference avoids the problem of incompleteness. If an experiment cannot be complete then it is always possible that a generic inference from an experiment may be totally wrong because some crucial factor has been missed out. Santos also does not deal with the issue of whether an analogy to field study is necessary to establish generic inference. One assumes that this is not the case but Santos does not explain why.
A final criticism is related to Santos’ division between ‘technological’ experiments and ‘behavioural’ experiments. Technological experiments are experiments designed to try out mechanisms for use in the outside world (the classic case is that of auction experiments used to design real-world auctions for selling electromagnetic bandwidth). Behavioural experiments are experiments designed to elicit human behaviour. The main difference between the two, according to Santos, is the level of control. Technological experiments basically find out how much control is necessary to force participants down a particular path. Behavioural experiments are designed to allow subjects to go down multiple pathways.
I am not sure as to the utility of this distinction and I feel that Santos spends far too much time on it. Bluntly, most experimentalists are not interested in technological experiments and the vast majority of experimental papers are on behavioural experiments. I cannot help thinking that the ‘control’ line would have more fruitfully have been drawn somewhere else – between ‘over-controlled’ and ‘well-controlled’ experiments for example. Furthermore, while the technological experiments had the ultimate aim of controlling behaviour most of the actual experiments were behavioural in outcome – there was always the possibility that subjects could behave in the ‘wrong’ way. The experiments produced behavioural data even if the aim of the research process was to design institutions. This suggests that there is no hard line between the two types of experiment.
In spite of these criticisms, I think that this is an original and thoughtful book that needs to be read by practicing experimental economists as well as methodologists. Santos’ central conclusion about the necessity for controls that do not close down relevant actions is of crucial importance in experimental design. In addition, the book uses an up-to-date framework that emphasizes experiments rather than theory testing (see Jones Reference Jones2008 for another example). This is a framework that I think should be used when thinking about the methodology of experimental economics rather than the old theory testing paradigm. In conclusion this is a book that is well worth reading and deserves a wide audience.
