Nobody in this debate questions the point that neuroeconomics remains full of potential, and little else as yet. If so, that really is progress of sorts. I was getting afraid that we would have to open nominations for the Captain Ahab Award for obsessive work on the promotion of neuroeconomics.Footnote 1
1. GENERAL REACTIONS
I am particularly heartened that those commenting on the debate appreciated that certain concerns, while serious, were put to one side. The embarrassing state of statistical inference underlying the “data” that are used in neuroeconomics has been accepted by all, but one can imagine that being fixed in time and in a methodologically predictable manner. The unwillingness to allow raw data to be shared or evaluated is appalling, but the economists at this table just turn their eyes away and smile. That will have to change in time, and so be it.Footnote 2
Two general problems demand more attention, however: the heavy use of localization assumptions, and the reverse inference problem.
1.1. Localization assumptions
Neuroeconomists truly are the New Phrenologists, since they rely heavily on localization assumptions about the modularity of the cognitive processes operating within the brain. Apart from the auxiliary assumptions about the correlates of craniology with brain activity, the latent assumptions about modularity are essentially the same as in the old Phrenology.Footnote 3 These concerns with modularity are old ones in psychology, and fundamental to the inferential engine being used; I recommend Shallice (Reference Shallice1988) and Uttal (Reference Uttal2001) for detailed challenges to the orthodoxy. They are also old issues throughout much of linguistics and philosophy, but I honestly cannot see how those wandering, nuanced debates will perturb the rushing crowd of neuroeconomists.
There are many issues in understanding the way in which neuroscientists apply the localization assumption, but Tallis (Reference Tallis2008) provides a useful overview:
Simply listing the fallacies that have led to some of the less cautious neuroscientists’ conclusions (especially when they talk to the general public) would take many pages. It is, however, worth noting that apparent localization of human feelings in bits of the brain is a kind of artefact. First, when it is asserted that such-and-such a part of the brain lights up in relation to a particular stimulus, this conclusion is arrived at by subtraction. Much more of the brain is already busy or lit up; all the scientist can observe is the additional activity associated with the stimulus. Minor changes noted diffusely are overlooked. Secondly, the additional activity can be identified only by a process of averaging the results of subtractions after the stimulus has been given repeatedly: variations in the response to successive stimuli are ironed out. Finally, and most importantly, the experiments look at the response to very simple stimuli – for example, a picture of the face of a loved one compared with that of the face of one who is not loved. But love is not like a response to a stimulus. It is not even a single enduring state, like being cold. It encompasses many things, including: not feeling in love at that moment; longing, indifference, delight; wanting to be kind, wanting to impress; worrying over the logistics of meetings; lust, awe, surprise, jealousy, anger; imagining conversations, events; imagining what the loved one is doing when one is not there; and so on. (The most sophisticated neural imaging, by the way, cannot distinguish between physical pain and the pain of social rejection: they seem to “light up” the same areas.)
One reason that this problem is so severe for neuroeconomists is that they have uncritically adopted the labels tossed around by behaviourists, encouraging them to “see” their favourite label in brain scans as surely as some “see” erotica in clouds and others “see” grammaticality in sentences such as “Colorless green ideas sleep furiously” (Schütze Reference Schütze1996).
1.2. Reverse inference
The other general problem is related, and is known in the literature as the reverse inference problem. This arises when activations in certain physical regions of the brain are presumed to identify the activation of a (labelled) cognitive process. Poldrack (Reference Poldrack2006; p. 59) explains the process well:
The usual kind of inference that is drawn from neuroimaging data is of the form ‘if cognitive process X is engaged, then brain area Z is active.’ Perusal of the discussion sections of a few fMRI articles will quickly reveal, however, an epidemic of reasoning taking the following form:
(1) In the present study, when task comparison A was presented, brain area Z was active.
(2) In other studies, when cognitive process X was putatively engaged, then brain area Z was active.
(3) Thus, the activity of area Z in the present study demonstrates engagement of cognitive process X by task comparison A.
This is a ‘reverse inference’, in that it reasons backwards from the presence of brain activation to the engagement of a particular cognitive function.
This “epidemic of reasoning” is, of course, the sine qua non of inference in neuroeconomics.
It is self-evidently not a matter of deductive logic, and indeed in introductory philosophy classes it is presented as the formal fallacy of “affirming the consequent”. For example, “if neuroeconomics research is good quality, it is published in top journals like Nature and Science; some neuroeconomics articles are published in Nature or Science; therefore, neuroeconomics is good research”. Clearly this is false as stated, since sloppy work can get published in these journals if refereeing standards are lax and incestuous.
If we restate the epistemological claim in the intended form, it can be seen as a potentially useful heuristic, known as “inference to the best explanation” or “abduction” in philosophy. So we would simply conclude instead that “therefore, neuroeconomics is probably good research”. Some philosophers of science take this heuristic seriously, and that is appropriate since it is very widely used. In fact, one finds discussion of such things as abductive validation, which is the notion that there are criteria that can be used to decide if one explanation is the best possible explanation. For example, elegance and simplicity are two such criteria in economics, as are ex post explanatory power and ex ante predictive power. My only point is to be clear about the epistemological basis for these knowledge claims. No serious philosopher argues that abduction will always, or eventually, lead to valid claims.
Even so, the nature of this reverse inference should be examined. Poldrack (Reference Poldrack2006) offers a useful formalization using Bayes Theorem, so that we can think of the strength of the final inference as being determined by the precision of the data as well as the priors that a certain cognitive process has been engaged. The “precision of the data” here refers to the extent to which the cognitive process in question is known to activate certain areas in relation to the overall chance of activation of that area. In other words, to what extent does the cognitive process “select” one or more regions in a way that can be discerned from other processes that activate these regions?
There is much debate in neuroscience about the selectivity of cognitive processes. For example, one early discussion of the reverse inference problem in this context was by D'Esposito et al. (Reference D'Esposito, Ballard, Aguirre and Zarahn1998). They “conclude that human lateral prefontal cortex supports processes in addition to working memory. Thus, reverse inference of the form ‘if prefontal cortex is active, working memory is engaged’ is not supported” (p. 274). This claim is meant to illustrate the type of “normal science” in the field that neuroeconomists need to be aware of, and this claim is subject to debate and, of course, refinement with different designs, instrumentation and/or methods of statistical inference. Indeed, it is not hard to find these debates in the neuroscience literature, but they tend to be glossed in the neuroeconomics articles that should be reporting them.Footnote 4
One constructive suggestion would be to encourage more systematic reporting of the selectivity of activity in certain regions, to allow more nuanced judgments to be made about overall statistical power. One approach is to conduct meta-analyses of pooled neuroimaging data, as illustrated by Poldrack (Reference Poldrack2006), and then to incorporate that uncertainty into the overall inference in a full-information manner.
Another implication of concern with the reverse inference problem is to generate auxiliary data that can change the prior that the cognitive process in question is indeed being engaged. This is where experimental methods have a role to play in complementing neuroscientific methods. As noted by Christoff and Owen (Reference Christoff and Owen2006), it is common in neuroscience to vary the cognitive domain of the task (e.g. linguistic versus non-linguistic stimuli). But one might also obtain variation in the prior by varying the cognitive complexity of the task (e.g. lotteries with 1 or 2 outcomes versus lotteries with more than 2 outcomes). Just as there are many ways to define the domain of a cognitive task, there are many ways to define the complexity of a cognitive task, but these are subjects that have been investigated. For an excellent example in economics, looking at choice under uncertainty, see Wilcox (Reference Wilcox1993).
Several commentators suggested that it was critical to look at the range of inferences that are made, rather than any one study in isolation. Although it is tempting to be reminded of the fisherman that lost a penny on every sale, but expected to make it up in volume, there is a grain of truth in these claims. If we are honest about the noise in the data we claim come from neuroimaging studies, and we properly pool inferences across multiple tasks, overall inference can be improved. But for all the talk, I see no real examples of this in the neuroeconomics literature. One well-cited study in neuroscience that purports to do this is Greene et al. (Reference Greene, Sommerville, Nystrom, Darley and Cohen2001). But their small fMRI sample of N = 9 is supplemented with “behavioral data concerning participants’ judgments and reaction times” (p. 2107) to hypothetical moral dilemma questions. The reaction times move in certain directions, but no attempt is made to pool these inferences in the form of a joint likelihood. It is, quite literally, two small samples more or less pointing in the same direction, but with no formal sense that the joint distribution of responses is any more consistent with the theory.
The most important implication of the reverse inference problem is to recognize the limitations of the knowledge claims we must expect from neuroeconomics unless it is complemented by other experiments. Poldrack and Wagner (Reference Poldrack and Wagner2004: 180) draw the conclusion well from a study of the extent to which phonological and semantic processing are consistently associated with topographically distinct patterns of activity in the left interior prefontal cortex (LIPC):
Finally, it is important to note that the reverse-inference approach requires a strong caveat, because it is not a logically valid form of deductive reasoning. For example, in the present case, reverse inference would hold true logically only if activity in aLIPC or pLIPC occurred only because of semantic or phonological processing, respectively (i.e., it requires an “if and only if” statement to be logically true), and few researchers would support such a claim. Accordingly, the reverse-inference approach is strongest when it is used not as an ad hoc means to explain the occurrence of particular activations in a single study, but rather as a tool to drive hypotheses that are then tested in subsequent experiments. Indeed, well-designed neuroimaging studies intended to adjudicate between competing psychological hypotheses that, themselves, have been formally described are precisely the kind of studies in which the reverse-inference approach may be justified. Through such an approach, neural markers of cognition hold promise for advancing understanding of the mechanisms of mind.
The same caution should be understood in neuroeconomics studies.
2. SPECIFIC REACTIONS
Camerer (Reference Camerer2008) presents a generally balancedFootnote 5 statement of the potential of neuroconomics. I am surprised that he does not challenge my claim that neuroeconomists are “poor” at sharing data, and can only infer that he cannot defend that practice either. Similarly, in response to my technical concerns with the manner in which statistical inferences are made in this area, all he offers is the weak reassurance that
Neuroscientists worry about methods and data quality frequently and are constantly making improvements (and neuroeconomists even more so). Journals are filled with methodology articles and neuroscientists are constantly innovating in how different tools are combined, improving experimental designs and statistical efficiency.
So we should just stop asking the hard questions about the quality of statistical inference in this area, and just continue to use methods that are riddled with known problems? That is not how I want us to practice economics.
Maybe we need to spell out why data sharing is such an issue. Trust in the results of a field arises through replication of inferences, and without sharing of data there can simply be no replication. And here we mean “replication of inferences under alternative structural and stochastic statistical assumptions”, not replication in the sense of one neuroeconomist thinking he gets the same finding as another. This is, after all, the problem with data privacy: the rest of us are unable to critically examine the importance of statistical assumptions in driving results. Camerer (Reference Camerer2008) does not deny the data privacy problem, but still asks us to trust that results and interpretations wouldn't change if we looked at the data ourselves. Unacceptable.
Camerer (Reference Camerer2008) makes the case for doing neuroeconomics even when we have conceptually muddled hypotheses about the expected behaviour. This is a remarkable admission of one of the problems with neuroeconomics:
Another potential use of neural data is to speed up the process of resolving fundamental debates. For example, Harrison (2008) has suggested that “we have conceptual work to do before we fire up the scanner”, in the context of the debate about whether ultimatum-game rejections are due to a taste for reciprocity built into utilities, or reflect “field-hardened heuristics for playing repeated games”. His view seems to be that we shouldn't do any brain scanning until the “conceptual work” is done. However, these two interpretations about ultimatum rejections (as well as other prosocial behaviour like public goods giving) have been around for decades. There is no resolution in sight. Therefore, it is possible that the resolution could come much faster if we “fire up the scanner” soon rather than waiting for the conceptual work project to be finished. For example, if there was some agreement on what brain activity would result when a person who has developed “field−hardened heuristics” adapted to one environment is placed in an environment where those are no longer adaptive, or what it would mean neurally to have a utility for reciprocity in one-shot games, we could do the experiments and see what happens. [emphasis added]
This idea of just doing experiments and seeing what happens is, to me, incoherent, whether the experiments involve neuroscience measurements or not. It is also positively dangerous, given the ability some scholars have for story-telling. Or maybe we should say, given the proclivity of the profession for paying attention to story-telling rather than substance, the fault is with the broader profession for sitting back and taking such efforts uncritically. In addition, note well how the above line of reasoning relies on the subjunctive, per my emphasis.
Ross (Reference Ross2008) generously recasts my commentary in the language of philosophy, pointing to arguments developed at more length in Ross (Reference Ross2005). In this respect, he makes me feel a bit like an idiot savant (no doubt my critics will agree that I have that half right). On the other hand, the insights from Ross (Reference Ross2005, Reference Ross2008) and Wilcox (Reference Wilcox2008) show why it is a pity that philosophers do not speak more plainly to economists, and with a more aggressive, direct tone. Using Star Wars metaphors, Ross (Reference Ross2008) correctly presents me as cursing the House of Vadar as well as the shiny rebels who portray themselves as attacking the alleged Death Star of orthodox economics. He draws a valuable distinction between behavioural economics in the scanner (BES) and neurocellullar economics (NE), and argues that most of my critique of BES, which he accepts and amplifies, does not apply to NE. He defines NE as “the program of using the mathematics of economic equilibrium analysis to write down models of brain cell activity for the sake of refining and comparatively testing hypotheses about neural learning that originate from computational neuroscience”. Derived explicitly from the general approach of Marr (Reference Marr1982), the exposition in Glimcher (Reference Glimcher2003: Part II) illustrates the approach well. Indeed, my aim was at BES, and only takes in NE to the extent that it uses some of the same inferential tools in the end to test hypotheses about brain evolution. I do have some concerns with the methods employed in NE, particularly the focus on simulations of models of reinforcement learning (e.g. Sutton and Barto Reference Sutton and Barto1998). These models can be notoriously difficult to refute, even when known a priori to be false, when taken to any data that exhibits unobserved heterogeneity correlated with the unit of observation (Wilcox Reference Wilcox2006). Related to this issue of problematic inference, I am concerned that optimization might not be the best way to view the evolution of the design of the brain. Linden (Reference Linden2007: 2) poses the contrasting view well, noting that many popular scientific books and documentaries on neuroscience
. . . perpetuate a fundamental misunderstanding about neural function. They present the brain as a beautifully engineered, optimized device, the absolute pinnacle of design. You've probably seen it before: a human brain lit dramatically from the side, with the camera circling it as if taking a helicopter shot of Stonehenge and a modulated baritone voice exalting the brain's elegant design in reverent tones. This is pure nonsense. The brain is not elegantly designed by any means: it is a cobbled-together mess, which, amazingly, and in spite of its shortcomings, manages to perform a number of very impressive functions. But while its overall function is impressive, its design is not.
So the problem is that if we write down a theory that purports to predict behaviour, using notions such as reinforcement learning defined over economic concepts, we will almost certainly find good predictions but no necessary understanding of mechanism. To use the correct terms from Marr (Reference Marr1982), if we want to model the function of the brain in some task, we can have a relatively simple computational theory, but we should not casually assume that the behaviour of the entity is unmodulated by the possibly ad hoc manner in which that theory is represented in an algorithm or by the possibly ad hoc manner in which the physical implementation applies that representation. If the brain is a cobbled-together mess, as Linden (Reference Linden2007) claims, then we probably need to know something about the physical implementation to understand behaviour (since behaviour is likely conditioned on it, rather than being unconditionally implemented on a tabula rasa). But, as stressed by Wilcox (Reference Wilcox2008), while this view gives neural processes a role to play in explaining behaviour, it is really a sideshow. In effect, the constraints on the optimization are being confused with the objective function and the manner in which the overall optimization problem is to be solved. Sometimes the constraints determine the solution, of course, but not always, and arguably quite rarely when we talk more broadly about the cognitive functions of interest to economists.
Ortmann (Reference Ortmann2008) provides a wonderful series of reading notes for those wanting to connect the inter-disciplinary threads between psychology, economics and neuroeconomics. Much of the jargon, and rhetorical style, of neuroeconomics comes from behavioural economics, and that field in turn pays lip service to insights from cognitive and social psychology. But there is considerable spillage ‘twixt lip and cup,’ as Ortmann (Reference Ortmann2008) stresses. By trusting the few tourists who claim to have read the psychology literature, and assuming that they have reported things accurately, economists have allowed some severe biases to flourish, in more ways than one.
Rubinstein (Reference Rubinstein2008) shares my primary concerns, but I am not so comfortable with the manner in which he dismisses the marketing hype. Some of it is just schlock, and can be immediately detected as such: by now the behavioural literature has given our olfactory sensory neurons a lot of practice. But I do not like hushed, unsaid, ex cathedra dismissals by attendees of a seminar who did not have the scientific spine to speak up during the seminar, and Rubinstein (Reference Rubinstein2008) reports on these as if they count for anything.Footnote 6 My concern is that we let claims by neuroeconomists pass unchallenged if we do not speak up, and I do not see many doing that in an articulate, explicit manner.
Wilcox (Reference Wilcox2008) argues against the methodological reductionism of neuroeconomics. Claims that neuroeconomics will help us explain economic growth and the social fabric, from McCabe (Reference McCabe2008), are just the veneer about “broader impact” one has to add for grant applications. But there is a methodological debate due to Marr (Reference Marr1982) that is worth paying attention to here, and my brief references to that debate are brilliantly extended by Wilcox (Reference Wilcox2008).Footnote 7 His commentary on the argument of Gul and Pesendorfer (Reference Gul, Pesendorfer, Caplin and Schotter2008), using the three-level view from Marr, is particularly devastating. He explains that we have to be careful in explaining what Marr proposed, particularly his notion of a “computational theory” of an information processor, since that language can be confusing. And we have to be agnostic about what agent, or collection of agents, is the “processor” under study. This seemingly abstract language allows one to see the broader methodological significance of these ideas and debates, as Wilcox (Reference Wilcox2008) shows with references to the work of Hutchins (Reference Hutchins1995) and Clark (Reference Clark1997) in social cognition. I would add, to emphasize the general importance of this debate, the work of the philosopher Grice (Reference Grice1989) and its applications to linguistics by Clark (Reference Clark1996). In effect, Wilcox (Reference Wilcox2008) is arguing that even if neuroeconomics provides a convincing account of individual cognition, it will only have shown itself to be the world champion of the one-yard sprint.
3. SUMMARY
What is disquieting about the literature in neuroeconomics is that careful readers can sense that this is not the historical accumulation of knowledge that makes some steadily increasing, and therefore hopefully irreversible, sense. It is rhetoric. There is a place for rhetoric, but not in the manner we have seen. Of course, this is now common in behavioural economics as well: the debaters do not miss a trick, but they do miss the point. Maybe this debate over neuroeconomics can structure the methodological questions in a way that allows them to be aired productively.
It is also important to tease apart complaints with the neuroeconomics literature and the potential contribution of the field itself. I am heartened that this debate seems to have allowed that distinction to be drawn. My conclusion from the commentators, particularly those that do the stuff, is that there is a case for allowing neuroeconomics to contribute hypotheses, but that the case for it contributing evidence is a long way off. My own sense, shared by the critics of neuroeconomics, is that it is more efficient for us to derive hypotheses from more traditional sources.
