Motivational context
When researchers contrast paradigms across settings, stimuli, and/or conditions, the results sometimes fail to replicate. For example, Ceci and Bronfenbrenner (1991, cited in Ceci Reference Ceci1996) asked children to predict where on a monitor geometric shapes would migrate after children pressed the space bar. A curvilinear algorithm determined where each shape would migrateFootnote
1
:
Even after 750 trials, children were still unable to predict the shapes' migration. The implication is that multiplicative reasoning is beyond their capability.
As shown in our Figure 1, however, when the identical algorithm controlled a video game in which the object was to predict the destination at which vehicles would meet a roadblock, children reached ceiling by 450 trials (Ceci Reference Ceci1996). Thus, behavior in ecologically challenging contexts led to findings at odds with those from socially sanitized settings.
Figure 1.
Curvilinear Distance Estimation in and out of Laboratory.
Semantic context
Much research on memory, reasoning, and moral development is based on stimuli expunged of meaningful associations (e.g., nonsense syllables) in the belief this will reveal underlying principles. For example, Wason's deduction task has been used to illuminate qualities of thinking. However, even when participants were from other cultures, it was clear that deduction depended on the content. Consider: The logic underlying Wason's task is that any inference that has the form:
is valid independent of the propositional content. However, it has been repeatedly shown this is not true and that content matters. Suppose four cards are laid out
with the instructions that each card has a number on one side and a letter on the other. Participants are told: If a card has a vowel on one side, then it has an even number on the other side. Their job is to determine which cards must be turned over to validate the rule. Even smart undergraduates have success rates of only 12%. However, if the content is changed to a mode of transportation on one side of each card and a destination on the other
and the rule is: Every time I go to Manchester, I go by train. Nothing has changed logically, but performance skyrockets to 60% (Johnson-Laird Reference Johnson-Laird1983). Similarly, we have demonstrated that differences in the way meaning is represented mediate children's metacognition (Ceci et al. Reference Ceci, Fitneva and Williams2010).
Cultural-cognitive context
Two of us (Kahan, Braman) have demonstrated that Americans who vary in their attitudes toward hierarchy and equality vary in their perceptions of legally consequential facts (Kahan, in press; Kahan et al. Reference Kahan, Hoffman and Braman2009). These competing attitudes cohere with opposing sets of norms, and related scripts of acceptable behavior, which can trump the demographic variables emphasized by the WEIRD critique (Kahan et al. Reference Kahan, Braman, Gastil, Slovic and Mertz2007).
Social meaning context
Actions have meanings as well as consequences. They embody attitudes, the expression of which shapes actors' perceptions of what they are doing and hence the value of doing it. For example, under what circumstances will an actor prefer the certainty of one sum to the probability of another? “When the expected value of the former exceeds the latter,” the classical microeconomist says, “subject to one's degree of risk aversion,” which the behavioral economist notes will vary depending on whether one understands the exchange as realizing a “gain” as opposed to avoiding a “loss.” But experimental work shows the answer also depends on what such an exchange means. Most members of WEIRD societies will shun trades involving tax evasion that they might well have accepted in the form of casino wagers, because the former, even if equivalent to the latter along the dimensions specified by economists, manifests attitudes and values antithetical to the self-conception of (most) WEIRD people (Baldry Reference Baldry1986). Likewise, economists' considerations – dominant payoff strategies, signaling, reciprocity – don't tell us all we must know to predict whether individuals will contribute to public goods (Liberman et al. Reference Liberman, Samuels and Ross2004). Even the bedrock axiom of economics – “demand curves slope downward” – founders on the shoals of meaning. Offering a wage can cause persons who previously did community service for free to refrain from doing any: payment destroys the meaning, and thus the associated value, of public-spirited behavior (Gneezy & Rustichini Reference Gneezy and Rustichini2000).
In a nutshell, ignoring meaning – as behavioral economics and related approaches do – generates unreliable predictions for any collection of persons who experience a common life. Heinrich et al. rightly question whether the behavioral dynamics in the work they examine generalize to non-WEIRD samples. But it is important not to assume this critique identifies some distinctive problem hovering at the “margins” of that work. Indeed, what does generalize about the new behavioral science is the question about its external validity: its neglect of meaning and other elements of context gives us reason to be cautious about accepting extrapolations from its stylized experiments.
In sum, our argument is that although we agree with Henrich et al. on the need to diversify samples, this alone will not unearth generalizable principles because contexts, attitudes, and meanings contribute systematic variance that must be included to reveal lawful ecological contrasts.
You have
Access
We commend Henrich et al. for pointing out the (over)reliance on participants from WEIRD societies and the tenuousness of universal laws whose support is based on a single subpopulation of a single society. Notwithstanding the merit of studying participants from diverse races, social classes, and national cultures, this will not in itself lead to generalizability of findings because it leaves unaddressed other threats to generalization, including the restricted physical, ideological, and attitudinal parameters of most research, and the omission of social meanings that participants attach to their choices.
Even when researchers include non-WEIRD participants, they rarely include contextual variation. Few of our principles are based on data from diverse settings and conditions. On those occasions when researchers do insert contextual diversity into their designs, it becomes apparent that theorizing is paradigm-bound – confined to the specific physical, motivational, and psychological conditions under which the data were gathered. Below we argue for the power of manipulating the context and social meanings, independent of sample diversity.
Motivational context
When researchers contrast paradigms across settings, stimuli, and/or conditions, the results sometimes fail to replicate. For example, Ceci and Bronfenbrenner (1991, cited in Ceci Reference Ceci1996) asked children to predict where on a monitor geometric shapes would migrate after children pressed the space bar. A curvilinear algorithm determined where each shape would migrateFootnote 1 :
Even after 750 trials, children were still unable to predict the shapes' migration. The implication is that multiplicative reasoning is beyond their capability.
As shown in our Figure 1, however, when the identical algorithm controlled a video game in which the object was to predict the destination at which vehicles would meet a roadblock, children reached ceiling by 450 trials (Ceci Reference Ceci1996). Thus, behavior in ecologically challenging contexts led to findings at odds with those from socially sanitized settings.
Curvilinear Distance Estimation in and out of Laboratory.
Semantic context
Much research on memory, reasoning, and moral development is based on stimuli expunged of meaningful associations (e.g., nonsense syllables) in the belief this will reveal underlying principles. For example, Wason's deduction task has been used to illuminate qualities of thinking. However, even when participants were from other cultures, it was clear that deduction depended on the content. Consider: The logic underlying Wason's task is that any inference that has the form:
is valid independent of the propositional content. However, it has been repeatedly shown this is not true and that content matters. Suppose four cards are laid out
with the instructions that each card has a number on one side and a letter on the other. Participants are told: If a card has a vowel on one side, then it has an even number on the other side. Their job is to determine which cards must be turned over to validate the rule. Even smart undergraduates have success rates of only 12%. However, if the content is changed to a mode of transportation on one side of each card and a destination on the other
and the rule is: Every time I go to Manchester, I go by train. Nothing has changed logically, but performance skyrockets to 60% (Johnson-Laird Reference Johnson-Laird1983). Similarly, we have demonstrated that differences in the way meaning is represented mediate children's metacognition (Ceci et al. Reference Ceci, Fitneva and Williams2010).
Cultural-cognitive context
Two of us (Kahan, Braman) have demonstrated that Americans who vary in their attitudes toward hierarchy and equality vary in their perceptions of legally consequential facts (Kahan, in press; Kahan et al. Reference Kahan, Hoffman and Braman2009). These competing attitudes cohere with opposing sets of norms, and related scripts of acceptable behavior, which can trump the demographic variables emphasized by the WEIRD critique (Kahan et al. Reference Kahan, Braman, Gastil, Slovic and Mertz2007).
Social meaning context
Actions have meanings as well as consequences. They embody attitudes, the expression of which shapes actors' perceptions of what they are doing and hence the value of doing it. For example, under what circumstances will an actor prefer the certainty of one sum to the probability of another? “When the expected value of the former exceeds the latter,” the classical microeconomist says, “subject to one's degree of risk aversion,” which the behavioral economist notes will vary depending on whether one understands the exchange as realizing a “gain” as opposed to avoiding a “loss.” But experimental work shows the answer also depends on what such an exchange means. Most members of WEIRD societies will shun trades involving tax evasion that they might well have accepted in the form of casino wagers, because the former, even if equivalent to the latter along the dimensions specified by economists, manifests attitudes and values antithetical to the self-conception of (most) WEIRD people (Baldry Reference Baldry1986). Likewise, economists' considerations – dominant payoff strategies, signaling, reciprocity – don't tell us all we must know to predict whether individuals will contribute to public goods (Liberman et al. Reference Liberman, Samuels and Ross2004). Even the bedrock axiom of economics – “demand curves slope downward” – founders on the shoals of meaning. Offering a wage can cause persons who previously did community service for free to refrain from doing any: payment destroys the meaning, and thus the associated value, of public-spirited behavior (Gneezy & Rustichini Reference Gneezy and Rustichini2000).
In a nutshell, ignoring meaning – as behavioral economics and related approaches do – generates unreliable predictions for any collection of persons who experience a common life. Heinrich et al. rightly question whether the behavioral dynamics in the work they examine generalize to non-WEIRD samples. But it is important not to assume this critique identifies some distinctive problem hovering at the “margins” of that work. Indeed, what does generalize about the new behavioral science is the question about its external validity: its neglect of meaning and other elements of context gives us reason to be cautious about accepting extrapolations from its stylized experiments.
In sum, our argument is that although we agree with Henrich et al. on the need to diversify samples, this alone will not unearth generalizable principles because contexts, attitudes, and meanings contribute systematic variance that must be included to reveal lawful ecological contrasts.