Economics is awash in data, and the vast majority of published articles are empirical. Yet the question that animates Julian Reiss's new book, Error in Economics, is, can economics can be made into an evidence-based (or at least a more evidence-based) science? In Reiss's view there is something wrong with the evidential base of economics, and his purpose is to set it right. His is not a systematic account, which is both a plus and a minus. It is a plus in that, in lieu of a systematic account, he treats a number of cases in rich detail – especially, the proposed reform of the US consumer price index (the Boskin Commission), Card and Krueger's famous minimum-wage study, and Angrist's natural experiment aimed at teasing out the effects of military service on lifetime income. While an economist may complain about particular details, he cannot task Reiss with wilful caricature. It is a minus in that much of the rest of the book is a critical examination of particular methods that tends to focus on the limitations without providing much of a roadmap for the investigator. This is a philosopher's approach; and, though Reiss professes himself optimistic about the prospects for empirical economics, it would be hard for an economist to come away from these discussions feeling optimistic.
One key question is, what constitutes economic evidence? On this and on method, Reiss declares himself to be a pluralist. He begins with a tripartite distinction among prima facie evidence, which is relevant to, but which may be defeasible and hence not decisive for, the question at hand; valid evidence, which controls for all known error; and sound evidence, which is both valid and relevant to particular purposes.
The focus on the relativity of evidence to purposes is usefully illustrated with respect to the consumer price index (CPI). Reiss is quite correct that the CPI embeds certain values in its design. It aims to support distributional questions (e.g. the indexing of pensions), and it may be misleading in other contexts and for other purposes (e.g. as a measure of the general price level in a money-demand function). Yet it is a weak form of value-ladenness that infects the CPI; and, despite Reiss's implication, it is little threat to the positive/normative distinction as that has been understood by economists at least since John Neville Keynes. Two economists may disagree on the weights to be given different costs or even whether compensation for changing prices is a good idea; yet, conditional on their principal's purpose, have no trouble in agreeing on whether the CPI successfully meets that purpose.
A second key question is, how does evidence modify our claims to knowledge? In the contest between the priests of deductivism and the sinners of inductivism, Reiss is a standard-bearer for Satan's party. In contrast, economists have typically worshipped in Mill's church, stressing the complexity of the economy and the role of a priori theory in making sense of it. Their deductive orientation explains the gut appeal of Popperian methodology to many economists. In principle, if not always in practice, they reject induction but approve of empirical testing.
A weakness of Reiss's argument is that he really does not answer their concerns – nor those of Hume or Mill or Popper, except to assert that Mill is unduly pessimistic, since methods have improved since the 1840s. Rather than offering a systematic account of induction, Reiss focuses on some prominent – and, apparently, inductive – methods. Reiss cites Nancy Cartwright's distinction between clinchers and vouchers. Vouchers add inductive weight to a conclusion without making it certain. Reiss barely mentions examples (case studies are one such), and he gives no account of how vouchers meet the traditional problems of induction. Instead, he focuses on clinchers, which under strong assumptions convert an inductive problem into a deductive problem. Examples include instrumental variables, Bayes nets, and random controlled experiments. His account is critical, stressing just how strong prior assumptions must be in order for such methods to clinch their conclusion. For example, in his treatment of instrumental variables, Reiss considers a series of progressively weaker interpretations. Interpreted as an entirely data-based method, instrumental variables are, he argues, not operational, as there is no basis on which to establish that an instrument is correlated with a putative cause and uncorrelated with the unobservable error term in a regression equation. Ultimately, the economist must appeal to causal background knowledge to assert the appropriateness of the instrument. Of course, this is exactly what economists were taught from the beginning. The earlier stronger interpretations are straw men. What is more, the instrumental-variable method under this interpretation is not a clincher. It has the form of a clincher, but is used as a voucher; for its clinching depends on the strength of the causal background knowledge. Sometimes Reiss talks as if his analysis of these methods discovers the bounds beyond which they simply do not apply – they are clinchers or they are nothing. What is really needed is an account of vouching: what vouches for the causal background assumptions? How do such conditional clinching methods vouch for conclusions when their presuppositions are themselves uncertain? Reiss is aware of the distinction between internal and external validity – indeed, it is implicated in his Chapter 10 (joint with Cartwright) on the use of counterfactuals in policy analysis. What is needed is a more searching account of external validity – how do vouchers vouch?
In his consideration of each method, Reiss's principal concern is causal. His pluralism leads him to reject the primacy of mechanistic accounts of causal explanation, such as Woodward's invariance account, on the grounds that we can learn about causes or engage in other scientifically respectable activities without having mechanistic or structural accounts. For example, he cites David Hendry and Michael Clements's advocacy of non-causal-mechanistic forecasting models. The example, however, does not persuade. While Hendry and Clements's forecasting models are not direct representations of a mechanism, they are completely based in a causal-mechanical understanding of the economy. Their strategy is to identify typical changes or instabilities in economic mechanisms and to ask what sorts of forecasting rules can adapt to them. The general point is also true with respect to Bayes nets and invariance methods for causal inference, which Reiss sees as not presupposing a mechanistic account. While it is true that they do not presuppose a particular mechanistic account, their logic is grounded in the existence of an unobserved mechanism and their goal is to characterize it.
In a related vein, Reiss challenges Friedman and Schwartz's argument for mechanistic accounts. In a famous example Friedman and Schwartz, who had bolstered claims for the causal priority of money over nominal income with statistical evidence of a type that Reiss regards as broadly relevant, go on to assert that, if the same type and strength of evidence were offered in favour of the causal priority of the production of dressmakers’ pins over nominal income, we would rightly reject it on the ground that statistical evidence is supportive of particular causal claims only if we can also provide a mechanistic account. Reiss goes on to dismiss various arguments for the necessity of mechanism, including the most fanciful – namely, that merely giving a theoretical description of a mechanism is an argument for its reality.
These analyses – in the end – seem to miss Friedman and Schwartz's point. Friedman was not an apriorist à la Mill, and he was strongly opposed to the Cowles Commission's Millian econometrics. Rather he was an empiricist, just as Reiss thinks we ought to be. So why prefer statistics supported by causal-mechanical accounts? Friedman explicitly claims that data do not speak for themselves in that we cannot even begin to think what data might be relevant to a question or interpret how their statistical properties bear on a question without at least a broad, though possibly tentative or conjectural, theoretical account.
One way of interpreting Friedman and Schwartz is to take the mechanistic-causal account as providing a context in which clinchers vouch for causal conclusions. No particular investigation is sui generis. Each – as Reiss has ably clarified – rests on various background assumptions, which we maintain on a variety of grounds. The causal-mechanical (or theoretical) account is the arena in which we can judge the consistency of the new empirical information with accepted, if tentative, understandings. Friedman and Schwartz reject the pin theory of the business cycle, because it is not consistent with the much wider set of information and commitments summarized in the economic theory. The statistics which are taken to support it appear to have the form of clinchers, but only with respect to background assumptions that the causal-mechanical account does not warrant. The pin-income data is inconsistent with the whole set of our understandings. The pin-income data does not vouch. That is not, however, the end of the matter. If we really have the statistics relating pins and income that Friedman and Schwartz presume, one alternative is to take the inconsistency as pointing out its artefactual nature, and the relationships should dissolve on further investigation. If this turns out not to be true, we are left with a puzzle. One resolution would be to construct a pin theory that is just as consistent with the wider information set as Friedman and Schwartz believe the money theory to be. An alternative would be to find an account consistent with the money theory that would explain why pins and income display such statistical relations.
Error in Economics is a difficult book to review in a short space. It touches on many important issues. And the fine details of the arguments matter. Its great strength is to take empirical economics much more seriously than economic methodologists often did in the past and to apply the lessons of the recent philosophy of science much more concretely than philosophers typically do. Reiss's arguments deserve to be discussed and debated far more thoroughly than is possible here.
