According to its author Error in Economics is ‘a monograph on methodological problems in applied economics’ which relies on several case studies to:

1. (i) call into question the ‘pre-eminence of theory’ perspective that currently dominates economic modelling, as well as

2. (ii) put forward suggestions on how ‘to supplement the theory-based orthodoxy with an evidence-based alternative’.

As stated in chapter 1, the author believes that ‘after studying a range of specific cases, it is possible to draw some more general lessons’ bearing on three fundamental questions:

(1) ‘How do we base economic claims on evidence?’; (2) ‘what types of evidence do we need?’; and (3) ‘what ought the relationship between evidence and hypothesis be?’

The author's modus operandi in addressing these questions is the distinction between:

1. (a) prima facie evidence e for a claim h which ‘gives a license to investigate h’,

2. (b) valid evidence e for h which ‘gives a license to believe h’, and

3. (c) sound evidence e for h which ‘gives a license to act on the basis of h’. (p. 11)

This distinction permeates the rest of the book in the sense that it is employed to evaluate the various case studies discussed, and provides the threads that loosely unify the various chapters.

There is a lot to recommend this monograph: first and foremost, the author's criticisms of the pre-eminence of theory perspective, which has dominated economic modelling since Ricardo (1817), are both insightful and pertinent, making a convincing case for the need to redress the balance between theoretical and empirical concerns.

Second, this is a refreshingly provocative book because the discussion is bold and daring, with no qualms about questioning the conventional wisdom in either economics or philosophy of science. I only wished that the author's constructive proposals matched the boldness of his criticisms and the daring of his assertions!

Third, a lot of ideas and arguments associated with his evidence-based alternative are right-headed. Examples of such far-reaching and bold statements include:

Fourth, the use of case studies makes the discussion a lot more focused and relevant by helping the author avoid the platitudes that often plague books on economic methodology.

Having praised some of its virtues, a crucial omission renders the book's proposals ‘on how to ground economic claims more firmly in evidence’ unpersuasive and vulnerable to criticism.

Notwithstanding the author's apparent broad familiarity with the relevant literature in both philosophy of science and economics/econometrics, it is clear to me that he missed the very parts which could have provided him with the broader methodological framework he is striving for.

I found it astounding that the book's primary theme of ‘probing for errors and controlling them’ did not naturally lead him to the error statistical perspective; see Mayo (1996). Unfortunately for the author, his omission has dire consequences for his proposals ‘on how to ground economic claims more firmly in evidence’. His case, although right-headed, suffers from a fatal flaw:

This renders the book's arguments concerning having evidence for a claim problematic.

Prime facie evidence. The author defines an observational or experimental outcome e as constituting prime facie evidence for h ‘if and only if knowing e is relevant for believing or acting on h.’ (p. 7). I am sure that Fleischmann and Pons believed that they produced a cold fusion reaction at the University of Utah in 1989, but did they?

Valid evidence. His notion of valid evidence adds to the prime facie evidence the condition that ‘all known sources of error in inferring from e to h have been controlled for’ (p. 8). However, instead of explaining how the ‘controlling of errors’ is achieved, he proceeds to claim that: ‘A method is reliable when it is likely to yield results that help at achieving a given purpose’ (p. xi).

I will resist the temptation to give jesting examples that fit this definition.

A moment's reflection on what the author seems to have in mind in ‘controlling errors’ suggests that the error statistical account of reliability appears considerably more appropriate:

The ‘controlling’ for establishing reliability is achieved via the relevant error probabilities associated with the inference procedure stemming from a modification/extension of the Fisher–Neyman–Pearson (F-N-P) approach to statistical inference; see Mayo and Spanos (2006). Unfortunately, the author seems completely unaware, not only of error probabilities, but more broadly on how the error statistical perspective can help him address the pivotal questions (1)–(3) mentioned above.

The error statistical perspective acknowledges that objective reliable evidence e for a claim h is almost never readily available. Instead, e is fashioned by the modeller using reliable inference procedures in analysing raw data x₀. Hence, the answers proposed are based on using the modified/extended F-N-P framework:

1. (i) to transform a finite and incomplete set of raw data x₀ – containing impurities and noise – into reliable ‘evidence’ e pertinent for the appraisal of a claim or a hypothesis h, and then

2. (ii) to relate h and e, in a statistically reliable way via error probabilities, in order to assess whether e provides evidence for or against h.

Regrettably, in this book F-N-P statistics are not just disregarded (there is not a single reference to any statistics book or paper), they are also misunderstood. The author's criticism of randomized clinical trials (RCT) (pp. 9–10, 136–138), reveals a lack of appreciation of crucial statistical issues pertaining to securing valid evidence e for a claim h based on data x₀. The author seems unaware of the fact that what ensures the reliability of inference in practice is the statistical adequacy of the premises of inference: the probabilistic assumptions comprising the statistical model in question are valid vis-à-vis data x₀. The RCT is not a magic wand, but an example of a fallible experimental design procedure which, in principle, aims to secure the validity of certain probabilistic assumptions, implicit in statistical techniques like ANOVA (the relevant statistical model). In practice, these assumptions need to be assessed using thorough Mis-Specification testing (M-S) vis-à-vis data x₀, and their validity established before any reliable inferences can be drawn; see Mayo and Spanos (2004). Any departures from these assumptions will vitiate inferences based on such data; see Student (1931) on what went wrong with the famous ‘Lancashire milk experiment’. Erroneous probabilistic assumptions will give rise to discrepancies between nominal and actual error probabilities associated with the relevant inference procedures (e.g. ANOVA), rendering any control over error probabilities hopeless, and thus undermining the reliability of the ensuing inferences.

Granted, not all evidence-based arguments rely on statistical data and inference, but the latter provides the quintessential example which can be used to assess what is needed in less structured cases. Having said that, the overwhelming majority of cases discussed in this book revolve around economic data. Moreover, cases which involve armchair empiricism, such as the ‘market for lemons’ (chapter 6), often lead to questionable conceptions of what constitutes empirical evidence for a claim h.

Sound evidence. The author's definition of sound evidence adds to the valid evidence conditions the restriction that ‘h is the relevant hypothesis for attaining purpose p’, and qualifies it with a right-headed reference to ‘potential errors that could invalidate the evidence’ (pp. 10–11). A closer reading at what the author is worried about, especially his alluding to external validity, reveals that the generic problem he is raising is how one can extend valid evidence for a statistical hypothesis to a substantive claim pertaining to the phenomenon of interest.

Again, the error statistical perspective can be of great help in addressing this issue by using a post data evaluation of inference based on ‘severe testing’ reasoning; see Mayo and Spanos (2006). Indeed, I will go a step further and argue that this perspective can accommodate most the author's concerns motivating his distinctions between prime facie, valid and sound evidence, by viewing these concerns in the context of a sequence of models, theory (primary), structural (experimental) and statistical (data) model, aiming to bridge the gap between theory and data; see Mayo (1996: chapter 5), Spanos (1986: 20–21). Space limitations prevent me from fleshing out this claim, but I strongly believe this will provide a much richer perspective for the issues discussed in all three parts of the book. It will also tie together several loose ends in the book using the notion of statistical adequacy.

The notion of a statistically adequate model – a probabilistic description that accounts for the statistical regularities in the data – provides data with ‘a voice of its own’, separate from the theory in question; see Spanos (2006). This error statistical notion can help the author in a multitude of ways to carry out his primary objective of ‘grounding economic claims more firmly in evidence’, including the following:

1. (a) Substantiate his bold claims in [4] cited above (p. 183).

2. (b) Weed out unwarranted ‘empirical adequacy’ claims made by many authors of the theory-based explanatory mechanisms he discusses in chapters 6–9, instead taking them at face value (see Spanos 2006).

3. (c) Untangle the Instrumental Variables method (chapter 7) from the clutches of the ‘pre-eminence of theory’ perspective by bringing out the role of the probabilistic structure of the data in selecting the relevant instruments (see Spanos 1986: 637–43).

4. (d) Render largely irrelevant the discussion in chapter 10 concerning causally based vs. counterfactual reasoning for any evidence-based evaluation, since none of the structural models used in that discussion are likely to be statistically adequate vis-à-vis any available observational or even experimental data; see Spanos (1995).

In conclusion, I recommend this monograph highly to anybody interested in applied economics/econometrics for its insightful and pertinent criticisms of the pre-eminence of theory perspective. However, its glaring omission to relate the notion of empirical evidence to data – via statistical inference reasoning – limits considerably the value of its proposals ‘on how to ground economic claims more firmly in evidence’.