In his précis (sect. 3.1) and book (May Reference May2018, p. 85), Joshua May schematizes moral debunking arguments like this:
1. Some of one's moral beliefs are mainly based on a certain factor.
2. That factor is morally irrelevant.
3. So: The beliefs are unjustified.
This logically invalid argument depends on a suppressed premise that moral beliefs are unjustified whenever they are mainly based on a morally irrelevant factor.
May accepts the normative premise (2) as “eminently plausible” (Reference May2018, p. 90) in cases of genuine “framing effects” that are “clearly morally irrelevant” (p. 89). May's examples of irrelevant framing include order of presentation, equivalent wording (Petrinovich & O'Neill Reference Petrinovich and O'Neill1996), and grammatical person (Nadelhoffer & Feltz Reference Nadelhoffer and Feltz2008).
After granting the normative premise in these examples along with the suppressed premise, the only remaining way for May to avoid the conclusion (3) is to deny the empirical premise (1). Against premise (1), May replies “meta-analyses suggest that the vast majority of moral decisions remain the same in the face of genuine framing effects” (Demaree-Cotton Reference Demaree-Cotton2016)” (sect. 5.2; see also May Reference May2018, pp. 91, 218). Thus, May admits that a minority of moral beliefs are mainly based on a morally irrelevant factor, so premise (1) is true of those beliefs. His primary reply concerns the number and kind of moral beliefs that are changed by genuine framing effects.
We will argue that this reply is inadequate for three reasons. First, Demaree-Cotton's meta-analysis does not show as much as she and May claim. Second, even if it did show that framing does not affect “the vast majority” moral judgments, the number of judgments affected would still be enough to cause trouble for popular views about how moral beliefs are justified. Third, this trouble applies even to obvious moral beliefs that are immune from framing effects. We discuss these three points in turn.
First, May relies crucially on one meta-analysis by Demaree-Cotton (Reference Demaree-Cotton2016). (May also mentions Kühberger [Reference Kühberger1998], but that analyzes studies of framing effects on risky choices rather than morality.) Demaree-Cotton's meta-analysis is taken by May to show, “Roughly 80 percent of people's moral intuitions subject to framing effects don't change, and that figure excludes studies that found no effect” (May Reference May2018, p. 91; cf. p. 218). Unfortunately, Demaree-Cotton's meta-analysis suffers from several flaws.
One technical problem is that Demaree-Cotton arrives at her conclusion that “80 percent […] don't change” simply by taking the difference between the proportion of moral judgments in distinct frames. She takes the difference between frame groups to show the proportion of people whose moral intuitions are changed by the frame. This statistical interpretation, however, obscures differences among subjects and among types of moral intuitions in susceptibility to framing effects. Moreover, simply taking the difference between two point estimates of group averages (i.e., means) ignores the uncertainty of the measurements; indeed, there are cases in which a large observed difference between two averages is still statistically not significant because of large measurement uncertainty. Furthermore, simply observing an effect in a sample does not provide a valid statistical basis for inferring an effect of this magnitude in the general population. For example, Demaree-Cotton assumes that a 70% difference between framing groups is statistically equivalent to a 70% chance that a randomly selected person's moral judgment is determined by the frame. This inference misconstrues how true effect sizes are estimated and generalized to new populations. To determine whether the effect of an independent variable generalizes to new subjects, one would need to run a random-effects, or mixed-effects, regression analysis that treats “subject” as a random factor.
Additional problems with Demaree-Cotton's meta-analysis concern not her statistical analysis but the studies she surveyed. Most framing effect studies included in Demaree-Cotton's meta-analysis are between-subjects rather than within-subjects experimental designs. This distinction is important, because any difference observed in a between-subjects design also carries differences between subjects, because each person sees only one experimental condition. Within-subjects designs, in contrast, expose each person to several treatment conditions, so individual differences among groups do not conflate the observed difference between conditions in within-subjects designs.
In addition, the studies that Demaree-Cotton reviewed cover only a very limited range of moral dilemmas. Six out of seven of them concern killing one person to save others, and they all specify that consequences will definitely occur without any mention of how likely these consequences are. There is little basis for generalizing from this small subset to other kinds of moral dilemmas, especially scenarios that involve risk and uncertainty. As May (Reference May2018) says, “we're just bad at reasoning with probabilities and risk generally” (p. 91), so it would not be surprising if framing effects were greater in risky moral dilemmas.
These problems with Demaree-Cotton's sample in addition to flaws in her statistical analysis undermine her and May's conclusion that roughly 80% of moral intuitions are reliable and not susceptible to framing effects. To estimate the extent of framing effects more accurately, we are preparing a larger meta-analysis of more than 80 studies of framing effects on a wider variety of moral judgments. Before completing our meta-analysis, we cannot be sure whether the 20% average rate of framing effects is too high or too low. In any case, it is premature for May (Reference May2018) to conclude that framing effects on moral judgments are “negligible” (p. 92) or that the “vast majority” (p. 90) of moral judgments are immune from framing effects.
Our second point concerns whether a 20% rate of each framing effect is enough to cause trouble. It seems so, especially because of the multiplicity of framing effects, understood broadly as effects of morally irrelevant factors on moral judgments. May discusses effects of order, wording, and grammatical person. Other studies find effects of morally irrelevant factors, such as videos (Valdesolo & DeSteno Reference Valdesolo and DeSteno2006), disgust inducers (Schnall et al. Reference Schnall, Haidt, Clore and Jordan2008), cleaners (Helzer & Pizarro Reference Helzer and Pizarro2011), sleep deprivation (Killgore et al. Reference Killgore, Kahn-Greene, Lipizzi, Newman, Kamimori and Balkin2007; Olsen et al. Reference Olsen, Pallesen and Eid2010), and social setting (Kelly et al. Reference Kelly, Stich, Haley, Eng and Fessler2007). These factors lie in the environment of the person judging rather than in the situation of the judged act, so they make people in different environments ascribe contrary moral properties to the same act in the same situation. It would be inconsistent to say that both of these contrary judgments are correct, so the environment of the person making the judgment cannot affect the moral status of the judged act. That ensures that such factors are morally irrelevant.
In response, May repeatedly points out that each of these effects is small and occurs in only some people. Good point! Nonetheless, multiple small effects each on a minority sometimes add up to large effects on the majority. Imagine that we find five framing effects that each affects 20 people out of 100. If the same 20 people are vulnerable to all five framing effects, then the remaining 80 are not vulnerable to any of those effects. However, if each framing effect occurs in a separate 20 with no overlap, then every one of the 100 is vulnerable to one framing effect. Moreover, if one framing effect makes a person's moral judgment a little less confident or a little less extreme in content, then another framing effect moves it further in the same direction, and another framing effect moves it even further in that direction, then all three framing effects together can amount to a large effect. Thus, to determine how many people's moral judgments are misled by framing and how much, we need to look not at each framing effect in isolation (as May does) but at many framing effects together.
Furthermore, let's grant May's assumptions that frames affect about 20% of moral judgments and that the same people are subject to all of the various framing effects, so 80% are immune. Even on these generous assumptions, 20% of us still make moral judgments distorted by framing effects. May (Reference May2018) denies that 20% is “substantial” (p. 91). We disagree. A gambler does not know that a six-sided die will not come up six, although the chance of a six is less than 20%. Analogously, we do not know that a particular moral judgment is not distorted by framing effects when there is a 20% chance of distortion. Admittedly, it can be rational to bet a little on one die not coming up six. However, such bets become irrational when there is little to gain and mistakes are costly. In important and controversial moral debates, mistakes are costly, and not much is lost by suspending belief about moral claims that lack confirmation.
The problem arises because we as individuals often do not know whether we are in the unreliable 20% or the reliable 80%. If we do somehow know that our own moral judgments are not distorted by framing effects, even though other people's moral judgments are so distorted, then we might be justified in trusting our moral judgments. But how would we know that we are so lucky? We would need an independent test of whether our moral judgments are correct. And even if we had an independent test, that test rather than the mere fact that a certain moral judgment seems right to us would be what makes us justified. Framing effects thus show that we are not justified in relying on moral intuitions by themselves without any independent confirmation that our moral intuitions are reliable.
To all of this, May (Reference May2018) and others are likely to object that some moral judgments seem obvious. May's example is “condemning someone for intentionally and successfully poisoning an innocent co-worker” (p. 92). We agree that this example and others are obvious. We also agree that such obvious cases are unlikely to be changed by framing effects (cf. Tanner & Medin Reference Tanner and Medin2004). However, such cases are not enough to show that moral judgments in controversial cases can be justified. Those controversial moral beliefs are both interesting and important, so framing effects can debunk many significant moral judgments, even if not all.
Moreover, framing effects on the other moral judgments can reveal how moral judgments are justified even in the obvious cases. When so many other moral judgments are based on framing effects, one cannot be justified in believing any particular moral judgment without some reason to believe that this particular judgment is not subject to framing effects. Just as one cannot know that one is a reliable moral judge, whereas others are not, unless one has some reason to believe that one is special in some relevant way; so one cannot know that any particular moral judgment is undistorted by framing effects, whereas others are distorted, unless one has some reason to believe that this judgment is special in some relevant way. We still might be justified in believing that judgment, but only if we have independent confirmation that it is somehow immune from framing effects. Then that confirming evidence is what makes the moral judgment justified, instead of the mere fact that it seems obvious to us.
In this way, scientific studies of framing effects shift the burden of proof and create a need for independent confirmation. That is enough to refute philosophical moral intuitionism, which claims that moral judgments do not need independent confirmation when they seem intuitively obvious to us. This refutation of moral intuitionism does not prove that moral judgments are not justified, but it does show that they are not justified by intuition alone. That is the primary challenge of framing effects in moral judgment. We would like to see May respond to it.
In his précis (sect. 3.1) and book (May Reference May2018, p. 85), Joshua May schematizes moral debunking arguments like this:
1. Some of one's moral beliefs are mainly based on a certain factor.
2. That factor is morally irrelevant.
3. So: The beliefs are unjustified.
This logically invalid argument depends on a suppressed premise that moral beliefs are unjustified whenever they are mainly based on a morally irrelevant factor.
May accepts the normative premise (2) as “eminently plausible” (Reference May2018, p. 90) in cases of genuine “framing effects” that are “clearly morally irrelevant” (p. 89). May's examples of irrelevant framing include order of presentation, equivalent wording (Petrinovich & O'Neill Reference Petrinovich and O'Neill1996), and grammatical person (Nadelhoffer & Feltz Reference Nadelhoffer and Feltz2008).
After granting the normative premise in these examples along with the suppressed premise, the only remaining way for May to avoid the conclusion (3) is to deny the empirical premise (1). Against premise (1), May replies “meta-analyses suggest that the vast majority of moral decisions remain the same in the face of genuine framing effects” (Demaree-Cotton Reference Demaree-Cotton2016)” (sect. 5.2; see also May Reference May2018, pp. 91, 218). Thus, May admits that a minority of moral beliefs are mainly based on a morally irrelevant factor, so premise (1) is true of those beliefs. His primary reply concerns the number and kind of moral beliefs that are changed by genuine framing effects.
We will argue that this reply is inadequate for three reasons. First, Demaree-Cotton's meta-analysis does not show as much as she and May claim. Second, even if it did show that framing does not affect “the vast majority” moral judgments, the number of judgments affected would still be enough to cause trouble for popular views about how moral beliefs are justified. Third, this trouble applies even to obvious moral beliefs that are immune from framing effects. We discuss these three points in turn.
First, May relies crucially on one meta-analysis by Demaree-Cotton (Reference Demaree-Cotton2016). (May also mentions Kühberger [Reference Kühberger1998], but that analyzes studies of framing effects on risky choices rather than morality.) Demaree-Cotton's meta-analysis is taken by May to show, “Roughly 80 percent of people's moral intuitions subject to framing effects don't change, and that figure excludes studies that found no effect” (May Reference May2018, p. 91; cf. p. 218). Unfortunately, Demaree-Cotton's meta-analysis suffers from several flaws.
One technical problem is that Demaree-Cotton arrives at her conclusion that “80 percent […] don't change” simply by taking the difference between the proportion of moral judgments in distinct frames. She takes the difference between frame groups to show the proportion of people whose moral intuitions are changed by the frame. This statistical interpretation, however, obscures differences among subjects and among types of moral intuitions in susceptibility to framing effects. Moreover, simply taking the difference between two point estimates of group averages (i.e., means) ignores the uncertainty of the measurements; indeed, there are cases in which a large observed difference between two averages is still statistically not significant because of large measurement uncertainty. Furthermore, simply observing an effect in a sample does not provide a valid statistical basis for inferring an effect of this magnitude in the general population. For example, Demaree-Cotton assumes that a 70% difference between framing groups is statistically equivalent to a 70% chance that a randomly selected person's moral judgment is determined by the frame. This inference misconstrues how true effect sizes are estimated and generalized to new populations. To determine whether the effect of an independent variable generalizes to new subjects, one would need to run a random-effects, or mixed-effects, regression analysis that treats “subject” as a random factor.
Additional problems with Demaree-Cotton's meta-analysis concern not her statistical analysis but the studies she surveyed. Most framing effect studies included in Demaree-Cotton's meta-analysis are between-subjects rather than within-subjects experimental designs. This distinction is important, because any difference observed in a between-subjects design also carries differences between subjects, because each person sees only one experimental condition. Within-subjects designs, in contrast, expose each person to several treatment conditions, so individual differences among groups do not conflate the observed difference between conditions in within-subjects designs.
In addition, the studies that Demaree-Cotton reviewed cover only a very limited range of moral dilemmas. Six out of seven of them concern killing one person to save others, and they all specify that consequences will definitely occur without any mention of how likely these consequences are. There is little basis for generalizing from this small subset to other kinds of moral dilemmas, especially scenarios that involve risk and uncertainty. As May (Reference May2018) says, “we're just bad at reasoning with probabilities and risk generally” (p. 91), so it would not be surprising if framing effects were greater in risky moral dilemmas.
These problems with Demaree-Cotton's sample in addition to flaws in her statistical analysis undermine her and May's conclusion that roughly 80% of moral intuitions are reliable and not susceptible to framing effects. To estimate the extent of framing effects more accurately, we are preparing a larger meta-analysis of more than 80 studies of framing effects on a wider variety of moral judgments. Before completing our meta-analysis, we cannot be sure whether the 20% average rate of framing effects is too high or too low. In any case, it is premature for May (Reference May2018) to conclude that framing effects on moral judgments are “negligible” (p. 92) or that the “vast majority” (p. 90) of moral judgments are immune from framing effects.
Our second point concerns whether a 20% rate of each framing effect is enough to cause trouble. It seems so, especially because of the multiplicity of framing effects, understood broadly as effects of morally irrelevant factors on moral judgments. May discusses effects of order, wording, and grammatical person. Other studies find effects of morally irrelevant factors, such as videos (Valdesolo & DeSteno Reference Valdesolo and DeSteno2006), disgust inducers (Schnall et al. Reference Schnall, Haidt, Clore and Jordan2008), cleaners (Helzer & Pizarro Reference Helzer and Pizarro2011), sleep deprivation (Killgore et al. Reference Killgore, Kahn-Greene, Lipizzi, Newman, Kamimori and Balkin2007; Olsen et al. Reference Olsen, Pallesen and Eid2010), and social setting (Kelly et al. Reference Kelly, Stich, Haley, Eng and Fessler2007). These factors lie in the environment of the person judging rather than in the situation of the judged act, so they make people in different environments ascribe contrary moral properties to the same act in the same situation. It would be inconsistent to say that both of these contrary judgments are correct, so the environment of the person making the judgment cannot affect the moral status of the judged act. That ensures that such factors are morally irrelevant.
In response, May repeatedly points out that each of these effects is small and occurs in only some people. Good point! Nonetheless, multiple small effects each on a minority sometimes add up to large effects on the majority. Imagine that we find five framing effects that each affects 20 people out of 100. If the same 20 people are vulnerable to all five framing effects, then the remaining 80 are not vulnerable to any of those effects. However, if each framing effect occurs in a separate 20 with no overlap, then every one of the 100 is vulnerable to one framing effect. Moreover, if one framing effect makes a person's moral judgment a little less confident or a little less extreme in content, then another framing effect moves it further in the same direction, and another framing effect moves it even further in that direction, then all three framing effects together can amount to a large effect. Thus, to determine how many people's moral judgments are misled by framing and how much, we need to look not at each framing effect in isolation (as May does) but at many framing effects together.
Furthermore, let's grant May's assumptions that frames affect about 20% of moral judgments and that the same people are subject to all of the various framing effects, so 80% are immune. Even on these generous assumptions, 20% of us still make moral judgments distorted by framing effects. May (Reference May2018) denies that 20% is “substantial” (p. 91). We disagree. A gambler does not know that a six-sided die will not come up six, although the chance of a six is less than 20%. Analogously, we do not know that a particular moral judgment is not distorted by framing effects when there is a 20% chance of distortion. Admittedly, it can be rational to bet a little on one die not coming up six. However, such bets become irrational when there is little to gain and mistakes are costly. In important and controversial moral debates, mistakes are costly, and not much is lost by suspending belief about moral claims that lack confirmation.
The problem arises because we as individuals often do not know whether we are in the unreliable 20% or the reliable 80%. If we do somehow know that our own moral judgments are not distorted by framing effects, even though other people's moral judgments are so distorted, then we might be justified in trusting our moral judgments. But how would we know that we are so lucky? We would need an independent test of whether our moral judgments are correct. And even if we had an independent test, that test rather than the mere fact that a certain moral judgment seems right to us would be what makes us justified. Framing effects thus show that we are not justified in relying on moral intuitions by themselves without any independent confirmation that our moral intuitions are reliable.
To all of this, May (Reference May2018) and others are likely to object that some moral judgments seem obvious. May's example is “condemning someone for intentionally and successfully poisoning an innocent co-worker” (p. 92). We agree that this example and others are obvious. We also agree that such obvious cases are unlikely to be changed by framing effects (cf. Tanner & Medin Reference Tanner and Medin2004). However, such cases are not enough to show that moral judgments in controversial cases can be justified. Those controversial moral beliefs are both interesting and important, so framing effects can debunk many significant moral judgments, even if not all.
Moreover, framing effects on the other moral judgments can reveal how moral judgments are justified even in the obvious cases. When so many other moral judgments are based on framing effects, one cannot be justified in believing any particular moral judgment without some reason to believe that this particular judgment is not subject to framing effects. Just as one cannot know that one is a reliable moral judge, whereas others are not, unless one has some reason to believe that one is special in some relevant way; so one cannot know that any particular moral judgment is undistorted by framing effects, whereas others are distorted, unless one has some reason to believe that this judgment is special in some relevant way. We still might be justified in believing that judgment, but only if we have independent confirmation that it is somehow immune from framing effects. Then that confirming evidence is what makes the moral judgment justified, instead of the mere fact that it seems obvious to us.
In this way, scientific studies of framing effects shift the burden of proof and create a need for independent confirmation. That is enough to refute philosophical moral intuitionism, which claims that moral judgments do not need independent confirmation when they seem intuitively obvious to us. This refutation of moral intuitionism does not prove that moral judgments are not justified, but it does show that they are not justified by intuition alone. That is the primary challenge of framing effects in moral judgment. We would like to see May respond to it.