2. EXPERIMENTAL PHILOSOPHY OF SCIENCE

Experimental philosophers, or ‘X-philes’ for short, use experimental and survey methods and statistical analysis of collected data that are standard in psychological and sociological research, to investigate people's intuitions about philosophically important notions such as knowledge, causality, free will and moral responsibility. X-philes have made surprising discoveries about the variance of those intuitions across populations, and contributed to some of the major debates in philosophy of language, philosophy of mind, epistemology and ethics (Knobe Reference Knobe2007).

Although X-philes differ in their substantial interests and exact experimental techniques (for a useful typology see Knobe Reference Knobe2007), they all share a naturalistic conviction that philosophy is continuous with science as far as methodology is concerned. Although methodological naturalism is challenged in the debate on the nature of conceptual analysis and thought experiments (e.g. Sosa Reference Sosa, Knobe and Nichols2008), it is uncontroversial at least for the majority of philosophers of science who consider scientific practice as essential empirical data for their theorizing. X-phi, however, has not had much impact on the philosophy of science at the methodological level. When studying scientific practice, the empirical method philosophers of science adopt is primarily the case study of historical and contemporary materials written by scientists, such as textbooks, research articles and personal memoirs. Philosophy of economics is not an exception in this regard, and many debates in the field have been inspired by ‘rich’ – methodologically self-conscious but often philosophically perplexing – texts by economists, such as Friedman (Reference Friedman1953) and more recently Gul and Pesendorfer (Reference Gul, Pesendorfer, Caplin and Schotter2008).

The proliferation of case studies in the philosophy of science is welcome because those studies can reveal subtle but epistemically important aspects of scientific practice. There is, however, a trade-off between the level of intimacy with a particular domain of science gained by case studies on the one hand, and the scope of philosophical analysis based on such intimate knowledge of practice on the other. In general, case studies tend to create competing philosophical analyses of scientific theories, models and concepts derived from various fields and sub-fields, making it difficult to obtain a full view of a given field (Stotz Reference Stotz2009), let alone science in general. One way in which philosophers can balance the requirement of generality and care for relevant details in their analysis is to collect quantitative data regarding the map of scientific practice in question, so that existing case studies can be compared and contrasted in a coherent framework (Weinberg and Crowley Reference Weinberg and Crowley2009). Karola Stotz and Paul Griffiths pioneered such an approach called ‘experimental philosophy of science’ (Griffiths and Stotz Reference Griffiths and Stotz2008). Their Representing Genes Project has used a survey method to reveal competing and complementary gene concepts and locate them in different fields of biological research (Stotz Reference Stotz2009). Their project is partly motivated to test Moss's (Reference Moss2003) thesis that there are two fundamentally different biological concepts of genes, Gene-D (for developmental) and Gene-P (for preformationism) in modern biology. Stotz et al. (Reference Stotz, Griffiths and Knight2004) partially succeeded in operationalizing Moss's distinction between the two gene concepts, and further developed their own three-part typology (instrumental, nominal and post-genomic genes) (Griffiths and Stotz Reference Griffiths and Stotz2006; Stotz et al. Reference Stotz, Bostanci and Griffiths2006). In this case, an interesting hypothesis was first proposed based on a detailed case study, and then it has been tested and developed using the X-phi approach.

A second example of the X-phi of science is a series of studies on the notion of innateness (Griffiths et al. Reference Griffiths, Machery and Linquist2009; Linquist et al. Reference Linquist, Machery, Griffiths and Stotz2011). The distinction between innate and acquired characteristics plays an important role in contemporary scientific debates in behavioural ecology and cognitive psychology (Griffiths Reference Griffiths and Zalta2009), but philosophers have not been able to reach a consensus regarding the exact definition of innateness (Griffiths Reference Griffiths2002). Griffiths and his colleagues are therefore extending their X-phi study of the vernacular concept of innateness to systematically investigate scientists’ concepts of innateness (see also Knobe and Samuels Reference Knobe and Samuels2013).

So far, only two sets of X-phi studies of science have been conducted, but they already suggest a couple of advantages of X-phi in studying scientific practice. First, it can systematically reveal a variance of a certain theoretical construct (e.g. genes) among different scientific communities, in an analogous way as X-philes reveal a demographic variance of intuitions across different populations. Second, X-phi of science can uncover a deeper theoretical structure underlying a certain concept (e.g. innateness), just like some X-phi studies unveil deeper psychological mechanisms underlying moral reasoning. Unlike non-experimental conceptual analysis, which tends to fix on one aspect of a certain concept, experimental methods enable philosophers to simultaneously operationalize multiple dimensions of a concept, and investigate (i) which dimension(s) weigh more and (ii) whether and how different dimensions are interacting (Griffiths and Stotz Reference Griffiths and Stotz2008). How these findings inform analysis differs from one case to another, and in particular between X-phi and X-phi of science, but a general point remains the same: the empirical data make it possible to do better philosophical analysis. I thus suggest that the same experimental approach will benefit philosophical discussions on the preference concept in economics as well.

3. PREFERENCE, PSYCHOLOGICAL AND ECOLOGICAL

I have noted above that the concept of preference has long been disputed among economists with different orientations despite its central role in economic theory. This tension manifests itself in the recent neuroeconomics controversy (see the special issue of this journal 2008). While Camerer et al. (Reference Camerer, Loewenstein and Prelec2005) claim that unpacking – modelling and measuring – neurological processes underlying people's choice behaviour will help refine, revise or even eliminate the preference as a theoretical construct, Gul and Pesendorfer (Reference Gul, Pesendorfer, Caplin and Schotter2008) assert that the preference concept in revealed preference theory (the standard formal economic tool to calibrate people's preference ordering from the data regarding choice behaviour) refers only to observable choice behaviour, and thus neurological (and psychological) findings have no bearing on the concept.

A typical reaction of philosophers of science to this debate is to criticize Gul and Pesendorfer as instrumentalists. Craver and Alexandrova (Reference Craver and Alexandrova2008), for example, diagnoses that Gul and Pesendorfer's insistence on the autonomy of economics from psychology and neuroscience stems from philosophically untenable and practically ill-suited ideas of instrumentalism and interdisciplinary incommensurability. But notice that there is an alternative, perhaps better way of making sense of Gul and Pesendorfer's separationist stance, while maintaining with Craver and Alexandrova (Reference Craver and Alexandrova2008) that the sciences of mind and brain can and do bear on economics. That is, instead of assuming the central issue of the debate as a metaphysical one of reductionism vs. anti-reductionism or instrumentalism vs. realism, one can interpret Gul and Pesendorfer's extreme position as reflecting their reasonable concern that psychologists and neuroscientists do not understand the distinctly economic notion of preference, partly because they do not share underlying epistemic and practical goals central to economics.

This interpretation consists in two hypotheses; following Griffiths and Stotz (Reference Griffiths and Stotz2008), I call them conceptual variance and conceptual ecology, respectively. In general, conceptual variance is the thesis that different scientific communities use a particular concept differently (e.g. molecular and evolutionary biologists use ‘genes’ in different ways). Conceptual ecology is the thesis that such conceptual differences reflect scientific requirements of different communities. Don Ross is the philosopher who comes closest to explicitly stating these hypotheses in relation to the preference concept. In discussing why many economists are sympathetic to Gul and Pesendorfer (Reference Gul, Pesendorfer, Caplin and Schotter2008), Ross (Reference Ross2011) highlights a dissociation between economists' and psychologists’ concepts of choice:

These facts, Ross suggests, are reflected in economists' methodological practice of constructing choice models from (and testing them against) those choice data that are systematically responsive to changes in exogenous variables. According to Ross, the concept of choice, and therefore that of preference in economics,Footnote ¹ is distinct from psychological and folk counterparts. That is, Ross is arguing to the effect that preference, as economists understand and use the concept, cannot be causally equivalent to the aggregate of individual psychological or neurological processes. Rather, he claims, they are a more selective theoretical construct measurable only up to the scale of discernible statistical responses to those variables.

A related point has been made by both theorists (Becker Reference Becker1962; Satz and Ferejohn Reference Satz and Ferejohn1994) and experimentalists (Gode and Sunder Reference Gode and Sunder1993), who show that what produces the allocative efficiency of certain markets is not the rationality of individual participants (e.g. that they have consistent preference), but rather the structure of the markets in which choice takes place (e.g. budget constraints): ‘In the case of any given regularity [in choice data], factors ‘outside the head’ [. . .] may be carrying more or less of the load’ (Ross Reference Ross2011: 224) in predicting and explaining market phenomena.

The analyses by Ross and others suggest that the preference concept in economics is not purely psychological but ecological, in the sense that it captures causal mechanisms through which environmental factors (i.e. those outside the head) mould choice. My way of paraphrasing Ross's insightful observations is to say that different economists understand preference differently, and accordingly use them differently in their practice (conceptual variance), and that these differences reflect scientific requirements of different communities of economists, rather than conceptual confusions and idiosyncrasies of individual economists (conceptual ecology).Footnote ² More specifically, at least psychological and ecological concepts of preference/choice can be distinguished in economics. They are not necessarily mutually incompatible but distinct, stemming from different explanatory and practical purposes of different sub-fields in economics. For example, influenced by cognitive (neuro)science, behavioural economists and neuroeconomists are interested in unpacking psychological and neurological processes (e.g. framing and emotions) underlying individual decision making. In contrast, other economists, including some experimental economists, are primarily interested in the effects of various institutional and social arrangements to market efficiency.

To make clearer what I take the differences between psychological and ecological notions of preference to be, I mention another typology of the nature of preference in the literature, and discuss how this typology is related to mine. Some behavioural decision researchers (e.g. Slovic Reference Slovic1995) and behavioural economists (Hoeffler and Ariely Reference Hoeffler and Ariely1999) make a similar typology of different notions of preference, namely intrinsic and constructed. The former is the view that economic theory presupposes causally basic, underlying preferences (‘needs and wants’) behind consumer choices, while the latter is the position that often people construct their preferences in a given context based on cues available at the time of preference elicitation (Hoeffler and Ariely Reference Hoeffler and Ariely1999: 114–115).

Although Hoeffler and Ariely (Reference Hoeffler and Ariely1999: 113) note that ‘most researchers believe in a middle ground,’ they hint that economists are inclined to believe in the existence of preference as psychological dispositions, while psychologists tend to see them as constructed by the ways in which options are described and preferences elicited.Footnote ³ Notice that Hoeffler and Ariely's (Reference Hoeffler and Ariely1999) typology is alternative to and inconsistent with my conceptual variance hypothesis. My hypothesis implies that the mainstream economists believe that preferences are not much about individual psychology, whereas behavioural economists and neuroeconomists inspired by cognitive psychology and neuroscience study preferences as stable psychological dispositions measurable by experimentation. So according to my framework, the traditional economists are more likely to hold the constructed-preference perspective, while the heterodox behavioural economists might tacitly share the intrinsic view of preference that Hoeffler and Ariely (Reference Hoeffler and Ariely1999) attribute to the mainstream economists.

Why is it important to test these hypotheses? Because, first of all, it provides scientists with empirically reliable data regarding conceptual differences that can work as a basis of good communication across different scientific communities. Currently we have only casual comparisons between economics and psychology from one-sided disciplinary perspectives. Second, an overview of the conceptual variance will provide a framework within which philosophers and methodologists can evaluate and diagnose interdisciplinary research practices. For example, what is the cause of the moderate empirical success of the so-called ‘social preference’ models – models that try to explain apparently non-selfish behaviour as a result of individuals’ preferences for fairness etc.? One can diagnose social preference models as too economic or psychological, but whether these diagnoses are accurate and useful crucially depends on what are the distinctively economic and psychological uses of the preference concept. An X-phi of economics study can provide such information.

4. PREFERENCE, FOLK AND SCIENTIFIC

A second example of the debates that can be informed by the evidence from the Xphi of science approach concerns the ontology of preferences. Hausman (Reference Hausman1998) and Mäki (Reference Mäki2000) maintain that preferences are, like beliefs, a venerable, unquestionable folk-psychological concept whose ontology belongs to our everyday ontic furniture of tables and chairs, unlike other posited scientific unobservables such as electrons and gravity. Hausman and Mäki arrive at rather different conclusions from this observation, but both agree on the common-sensical status of most theoretical constructs in economics. This view, ‘commonsensible realism’, has been criticized as not cogent for some macroeconomic aggregates such as ‘real GDP’ and ‘general price level’ (Hoover Reference Hoover1995), and lately its truth in microeconomics is also questioned. Guala (Reference Guala, Kuorikoski, Lehtinen and Ylikoski2011), for example, argues that, although the preference concept is continuous with our folk psychology, it has also been used as a scientific or investigative kind term precisely defined by decision theory, and thus its ontological status can be and in fact has been empirically challenged. The accumulating evidence in behavioural research, Guala argues, strongly suggests that ‘preference’ is not a natural kind term that cuts the mind at its joints. That is, the concept refers to a set of heterogeneous psychological dispositions underlying choice behaviour (just like ‘jade’ refers to two distinct minerals, jadeite and nephriteas), and therefore its usefulness for scientific prediction and explanation of behaviour is rather limited. If this is true, commonsensible realists’ strategy to defend the preference concept as a sound scientific kind term by associating it with our folk psychology must be given up. Preceding case studies of the development of decision theory (e.g. Guala Reference Guala2000; Nagatsu Reference Nagatsu2010) support the view that there is no clear line between folk and scientific notions of preference, but rather that researchers from different scientific communities may hold different scientific concepts of preference that depart from the folk concept in different respects.Footnote ⁴ My conceptual variance hypothesis regarding preference explicates what this difference might be. And whether or not this specific hypothesis is true, the empirical results will inform the commonsensible realism debate, which at this stage is framed as, rather simplistically, a choice between the ‘folk’ and ‘scientific’ ways of understanding the preference concept.

While Guala's (Reference Guala, Kuorikoski, Lehtinen and Ylikoski2011) argument is based on his observation of the development of behavioural decision theory, Hands (Reference Hands, Ylikoski, Lehtinen and Kuorikoski2011) builds his critique of commonsensible realism on his examination of contemporary revealed preference theory (CRPT), an approach to the theory of consumer choice in contemporary microeconomics. CRPT:

Hands points out that this exercise cannot be interpreted as consistent with our folkpsychological understanding of preference (together with beliefs) as determinants of choice. That is, the exercise is not an attempt to measure underlying forces (preferences) that cause the aggregate consumer choice in question. This interpretation is excluded because, Hands explains, practitioners of CRPT characterize their exercise explicitly in a way that makes this commonsense interpretation impossible. Hands (Reference Hands, Ylikoski, Lehtinen and Kuorikoski2011) is critical about CRPT, not necessarily because it radically departs from folk psychology, but because it goes against the realist methodology of tracking the causal mechanisms to predict and explain phenomena, to which common sense psychology appears more congenial. That is, Hands seems to think that CRPT is both metaphysically and methodologically problematic because it does not care about causes, which are exactly what science should be after. But again, is it justified to criticize the practitioners of CRPT as scientific anti-realists? An alternative interpretation, suggested by Ross (Reference Ross2011), is that those economists are looking at the causes operating on a ‘different scale of resolution’ on the overlapping physical domains, namely on the scale of statistically discernible behavioural reactions to ecological forces rather than internal, psychological forces.

Hands's criticism derives from his interpretation of economists' commentaries on their practice, which he takes to be good data for methodological analysis. But why should we assume that those outspoken economists are good at philosophically articulating or generalizing about their practice? Since they are (usually) not trained in philosophy, we should at least compare economists' methodological reflections with other, more reliable types of evidence. Published (non-self-analysis-type) works by economists are an obvious candidate, and they are in fact the primary source of historical and philosophical case studies. But as I pointed out earlier, case studies are not helpful when it comes to making a generalization about scientific practice.Footnote ⁵ Hence the need of X-phi of science.

5. SOME OBJECTIONS AND FUTURE DIRECTIONS

In this section, I will first address three likely objections to applying X-phi in the study of economics. The third one will lead to a discussion of some of the methodological advantages of the factorial survey approach that some X-philes adopt. I also briefly and roughly sketch how one could begin to design an X-phi study of the preference concept in economics.

A first objection is a common charge by some philosophers that X-philes commit a naturalistic fallacy. But this allegation doesn't apply to X-philes of science who are under no illusion that normative critiques of science can be derived solely from the facts about scientific practice. Their suggestion is rather that philosophers of science can better perform normative jobs – be it metaphysical, conceptual or practical – if they have a better picture of relevant scientific practice, just like doctors can better improve particular patients’ conditions if they have a wider empirical basis, including epidemiological findings as well as close knowledge of those particular patients. A second objection, expected from some philosophers of science, is that generalization about scientific practice can be made by accumulating case studies, instead of conducting X-phi studies. However, this suggestion is impractical because, as Chang (Reference Chang2004) rightly stresses, case studies often aim at an articulation of abstract philosophical ideas that will greatly vary across different case studies, even if these studies look at the same research articles. The X-phi method, in contrast, can be used to simultaneously study a much larger sample of scientists in a systematic manner.Footnote ⁶ Third, some economists might suspect that the notions of preference that an X-phi study elicits will be personal opinions and tastes of different economists that are irrelevant to the understanding of their professional practice. This is a legitimate methodological concern, but it can be fully dealt with only by actually designing and conducting an X-phi study, and ultimately by demonstrating relevant correlations between professional backgrounds and elicited notions of preference. Since fully satisfying the sceptics in this way is beyond the scope of the present paper, I will try to indirectly address their concern by discussing (i) several methodological advantages of the factorial survey approach, and (ii) possible ways of identifying relevant dimensions in the preference concept.

Although I have found no X-philes acknowledging this, some X-phi studies, in particular X-phi of science studies, can be best characterized as a version of factorial surveys. The factorial survey approach has been used in sociology for more than three decades to study the structure of people's attitudes and judgements about social objects such as norms and practices (for a review see Wallander Reference Wallander2009). This approach has respondents evaluate several vignettes (fictive descriptions), but differs from the conventional survey method both in generating vignettes by systematically varying the levels of factors of theoretical interest, and in systematically assigning vignette sets to groups of respondents so that the main and interaction effects of factors can be statistically estimated. Factorial surveys, characterized for this reason as quasi-experimental, or survey experiments, are increasingly popular in social research, as they facilitate a cost-effective sampling of a large number of subjects while allowing for manipulation and control of factors in vignettes (Nock and Guterbock Reference Nock, Guterbock, Marsden and Wright2010). Compared with more conventional surveys, factorial surveys have several advantages (Wallander Reference Wallander2009), such as being (i) more suitable to studying the contexts and conditions that actually affect judgements because respondents are presented with concrete and detailed descriptions in which hypothesized factors are systematically varied, and (ii) better at identifying the actual determinants – or combinations of determinants – of human judgements because it can isolate and measure (interactions of) certain factors that influence their judgements, but of which people are not always aware and unable to articulate when they introspect about them. These methodological advantages have already been partly exploited in e.g. Griffiths et al. (Reference Griffiths, Machery and Linquist2009).

So the X-phi of science using factorial surveys seems promising in general, but how can we design an X-phi of economics study to test the conceptual variance hypothesis? First of all, prior to the factorial survey, it will be useful to ask economists a few straightforward questions regarding the nature and methodological values of the preference concept in economics, following Stotz et al. (Reference Stotz, Griffiths and Knight2004). This will be done by constructing multiple choices from stylized views found in economics textbooks and manifesto-type articles. Such a survey will be useful since currently there is no such quantitative data available. Moreover, the data can be used as a benchmark against which responses to a factorial survey are compared.

A first step in designing a factorial survey is to identify key dimensions of the preference concept from the literature. I will sketch three examples below: (i) Although compulsive or addictive behaviour may not be seen as a result of one's preferences in the vernacular sense, many drug addicts’ consumption behaviour is amenable to preference-theoretic analysis as it is responsive to changes in incentives (Ross Reference Ross2011). Vignettes about addictive consumption and other non-standard economic behaviour can be used to measure the importance of this factor (voluntariness) in associating certain behaviour with decision makers’ preferences. (ii) Another relevant dimension concerns the stochastic character of choice data, which Harrison (Reference Harrison2008: 326n) discusses in criticizing Gul and Pesendorfer's theorist bias. When experimentally measuring people's risk attitudes in binary choice, for example, it is inevitable to have certain assumptions about the nature of errors in choice. Economists have competing models according to which those errors take place at different stages (at the final choice, on the comparison of preferences, or determination of expected utility). Simulating this model choice in a questionnaire will be an interesting way of operationalizing the ecological–psychological distinction of preferences, particularly because the use of different stochastic models may reflect economists' different epistemic goals such as prediction and explanation (Wilcox Reference Wilcox, Cox and Harrison2008). (iii) The literature suggests a tension in the interpretation of the invariance principle in preference measurement. Invariance is the assumption that, in extensionally identical choice problems, the relationship between preferences and choice behaviour is not affected by how choice situations are framed or how preferences are elicited. Although it has been identified as ‘[a]n elementary effect of rationality’ in economics (Arrow Reference Arrow1982: 6), there is much evidence against invariance. An interesting question is whether the invariance principle in economics drives from the concept of preference as underlying psychological dispositions, as Hoeffler and Ariely (Reference Hoeffler and Ariely1999) suggest, or from some other methodological or practical reasons. One way of addressing this question is to operationalize invariance and intrinsicness separately, and measure their interactions.

Although these examples are preliminary sketches, I hope that the sceptics now see that the preference concept is a promising candidate for an X-phi study because the existing literature already points towards several interesting hypotheses regarding the concept and possible ways to test them.