Firestone & Scholl (F&S) discuss task demand in study design. Although we agree that researchers should take care to address demand, we fundamentally disagree with F&S's claim that findings of top-down effects on perception can be explained by task demand. They have asymmetrically applied their own standards for evaluating evidence of task demand, and in so doing have misrepresented the available evidence regarding demand on both sides of the debate.
When standards are applied equally to evaluating all research, the strength of the evidence against top-down effects is hardly as convincing as F&S suggest. For example, they rely on work by Durgin et al. (Reference Durgin, DeWald, Lechich, Li and Ontiveros2011a) to argue that task demand plagues studies of top-down effects. However, this research fails to meet the standards F&S establish (see arguments 4.1.3, 4.4.3). Specifically, Durgin et al. (Reference Durgin, DeWald, Lechich, Li and Ontiveros2011a) did not replicate the focal effect – that valuable objects appear closer than less-valuable objects. The researchers had participants toss a beanbag at a target. Study 1 found that task instructions that may manipulate task demand – to come closest to or to hit the target – produced different distributions of tossing behaviors. Study 2 found that financial incentives to come closest did not affect tosses to a neutral target. It is an erroneous inference to suggest these results undermine the conclusion that object value influences distance perception. Because they do not replicate the effect of object value, the conclusion that demand explains the effect of object value on distance perception collapses under the standards F&S put forth. Though informative, the findings are actually irrelevant to the study of object value on distance perception. The conclusion that demand effects can exert an influence on perceptual experience should not be confused for evidence that demand does exert an influence.
F&S's suggestions for how to mitigate task demand confuse rather than clarify the issue, because they introduce new possibilities for demand rather than serve as demand-free comparison conditions. They encourage researchers to mitigate demand by telling participants that external factors do not affect perception, which instead impacts compliance and honesty of reported perceptual experiences. Likewise, they encourage researchers to offer participants alternative explanations for experimental conditions that are not relevant to the predictions – for example, that props held during the task are meant to assist with balance rather than affect body size. F&S assert that alternative cover stories free participants of the influence of demand. We strongly contest this claim. Alternative cover stories do not remove the opportunity to guess hypotheses, nor do they eliminate the possibility that participants will amend their responses in accordance with their conjectured suppositions. They simply introduce new task demands. Researchers who attempt to overcome the pernicious effects of demand by manipulating expectations use a technique that is inherently flawed.
Instead, we offer five of our own published techniques as effective methods for overcoming demand. First, we use accuracy incentives to limit the likelihood of response bias (e.g., Balcetis & Dunning Reference Balcetis and Dunning2010; Balcetis et al. Reference Balcetis, Dunning and Granot2012). In a binocular rivalry paradigm where two different images were presented simultaneously, participants reported what they saw (Balcetis et al. Reference Balcetis, Dunning and Granot2012). We associated one image type (e.g., letters) with specific positive point values and another (e.g., numbers) with specific negative point values. We converted points into raffle tickets for monetary prizes. To ensure that participants truthfully reported their perceptual experience, we offered participants an accuracy incentive. If participants correctly reported all of the visual information they saw (e.g., both the letter and number that appeared), their score increased by some undefined amount; if they did not, their score decreased. Despite the incentive to respond accurately, participants primarily reported seeing only the images associated with reward and very infrequently reported experiencing both percepts. Moreover, if participants were strategically choosing to report percepts in ways that maximized payoff, we should have and could have found evidence for inhibition in addition to facilitation effects. However, participants were no less likely to report perceiving images associated with the loss of points relative to a baseline condition.
Second, we use counterintuitive behavioral responses as dependent measures, such that even if participants guessed the purpose of the study, they would not know how to respond to support the hypotheses (e.g., Balcetis & Dunning Reference Balcetis and Dunning2010; Stern et al. Reference Stern, Cole, Gollwitzer, Oettingen and Balcetis2013). For example, participants estimated the distance to a desirable (chocolates) or undesirable object (dog feces) by moving themselves to stand a set, referent distance away from it (Balcetis & Dunning Reference Balcetis and Dunning2010, Study 3b). If the chocolate appeared closer, paradoxically, participants would need to stand farther away from it to match the set distance. That our predictions required they stand farther away from chocolates and closer to dog feces is a counterintuitive response that participants do not expect, reducing the likelihood of a demand effect.
Third, we conduct studies using between-subjects designs (e.g., Balcetis & Dunning Reference Balcetis and Dunning2010; Cole & Balcetis Reference Cole and Balcetis2013). Participants cannot know that our hypotheses predict they will stand farther from chocolates if randomly assigned to the chocolate condition, and closer to feces if randomly assigned to that condition. Task demand is less likely to pertain when participants lack half of the information necessary to conjecture what the hypotheses expect.
Fourth, we use double-blind hypothesis testing in which both participants and experimenters are unaware of the assigned condition (e.g., Cole & Balcetis Reference Cole and Balcetis2013). For example, when participants drank juice sweetened with either sugar or Splenda before estimating the distance to a location, participants, experimenters, and the graduate student training experimenters and analyzing data were unaware of drink-type. This reduced if not negated the impact of task demand, response bias, and experimenter bias.
Finally, we dissociate participants' perceptual experiences from manipulations that could be influenced by demand (e.g., Cole & Balcetis Reference Cole and Balcetis2013). In a test of object construal on distance perception, participants tossed a beanbag with the intent to hit a picture frame that held a $100 bill or was empty. No financial reward or outcome was tied to the toss itself. Thus, the toss served solely as a behavioral measure of perceptual experience. By dissociating the perceptual measure from attainment of the reward, we reduced the likelihood that the measure reflected task demands.
These five techniques limit the opportunity for task demand and improve on F&S's suggestion to combat demand by manipulating expectations. We encourage researchers – including F&S – to apply the same rigorous standards not only to the analysis of studies that seek to provide confirmatory evidence for top-down effects, but also to studies that seek to provide disconfirmatory evidence.
You have
Access
Firestone & Scholl (F&S) discuss task demand in study design. Although we agree that researchers should take care to address demand, we fundamentally disagree with F&S's claim that findings of top-down effects on perception can be explained by task demand. They have asymmetrically applied their own standards for evaluating evidence of task demand, and in so doing have misrepresented the available evidence regarding demand on both sides of the debate.
When standards are applied equally to evaluating all research, the strength of the evidence against top-down effects is hardly as convincing as F&S suggest. For example, they rely on work by Durgin et al. (Reference Durgin, DeWald, Lechich, Li and Ontiveros2011a) to argue that task demand plagues studies of top-down effects. However, this research fails to meet the standards F&S establish (see arguments 4.1.3, 4.4.3). Specifically, Durgin et al. (Reference Durgin, DeWald, Lechich, Li and Ontiveros2011a) did not replicate the focal effect – that valuable objects appear closer than less-valuable objects. The researchers had participants toss a beanbag at a target. Study 1 found that task instructions that may manipulate task demand – to come closest to or to hit the target – produced different distributions of tossing behaviors. Study 2 found that financial incentives to come closest did not affect tosses to a neutral target. It is an erroneous inference to suggest these results undermine the conclusion that object value influences distance perception. Because they do not replicate the effect of object value, the conclusion that demand explains the effect of object value on distance perception collapses under the standards F&S put forth. Though informative, the findings are actually irrelevant to the study of object value on distance perception. The conclusion that demand effects can exert an influence on perceptual experience should not be confused for evidence that demand does exert an influence.
F&S's suggestions for how to mitigate task demand confuse rather than clarify the issue, because they introduce new possibilities for demand rather than serve as demand-free comparison conditions. They encourage researchers to mitigate demand by telling participants that external factors do not affect perception, which instead impacts compliance and honesty of reported perceptual experiences. Likewise, they encourage researchers to offer participants alternative explanations for experimental conditions that are not relevant to the predictions – for example, that props held during the task are meant to assist with balance rather than affect body size. F&S assert that alternative cover stories free participants of the influence of demand. We strongly contest this claim. Alternative cover stories do not remove the opportunity to guess hypotheses, nor do they eliminate the possibility that participants will amend their responses in accordance with their conjectured suppositions. They simply introduce new task demands. Researchers who attempt to overcome the pernicious effects of demand by manipulating expectations use a technique that is inherently flawed.
Instead, we offer five of our own published techniques as effective methods for overcoming demand. First, we use accuracy incentives to limit the likelihood of response bias (e.g., Balcetis & Dunning Reference Balcetis and Dunning2010; Balcetis et al. Reference Balcetis, Dunning and Granot2012). In a binocular rivalry paradigm where two different images were presented simultaneously, participants reported what they saw (Balcetis et al. Reference Balcetis, Dunning and Granot2012). We associated one image type (e.g., letters) with specific positive point values and another (e.g., numbers) with specific negative point values. We converted points into raffle tickets for monetary prizes. To ensure that participants truthfully reported their perceptual experience, we offered participants an accuracy incentive. If participants correctly reported all of the visual information they saw (e.g., both the letter and number that appeared), their score increased by some undefined amount; if they did not, their score decreased. Despite the incentive to respond accurately, participants primarily reported seeing only the images associated with reward and very infrequently reported experiencing both percepts. Moreover, if participants were strategically choosing to report percepts in ways that maximized payoff, we should have and could have found evidence for inhibition in addition to facilitation effects. However, participants were no less likely to report perceiving images associated with the loss of points relative to a baseline condition.
Second, we use counterintuitive behavioral responses as dependent measures, such that even if participants guessed the purpose of the study, they would not know how to respond to support the hypotheses (e.g., Balcetis & Dunning Reference Balcetis and Dunning2010; Stern et al. Reference Stern, Cole, Gollwitzer, Oettingen and Balcetis2013). For example, participants estimated the distance to a desirable (chocolates) or undesirable object (dog feces) by moving themselves to stand a set, referent distance away from it (Balcetis & Dunning Reference Balcetis and Dunning2010, Study 3b). If the chocolate appeared closer, paradoxically, participants would need to stand farther away from it to match the set distance. That our predictions required they stand farther away from chocolates and closer to dog feces is a counterintuitive response that participants do not expect, reducing the likelihood of a demand effect.
Third, we conduct studies using between-subjects designs (e.g., Balcetis & Dunning Reference Balcetis and Dunning2010; Cole & Balcetis Reference Cole and Balcetis2013). Participants cannot know that our hypotheses predict they will stand farther from chocolates if randomly assigned to the chocolate condition, and closer to feces if randomly assigned to that condition. Task demand is less likely to pertain when participants lack half of the information necessary to conjecture what the hypotheses expect.
Fourth, we use double-blind hypothesis testing in which both participants and experimenters are unaware of the assigned condition (e.g., Cole & Balcetis Reference Cole and Balcetis2013). For example, when participants drank juice sweetened with either sugar or Splenda before estimating the distance to a location, participants, experimenters, and the graduate student training experimenters and analyzing data were unaware of drink-type. This reduced if not negated the impact of task demand, response bias, and experimenter bias.
Finally, we dissociate participants' perceptual experiences from manipulations that could be influenced by demand (e.g., Cole & Balcetis Reference Cole and Balcetis2013). In a test of object construal on distance perception, participants tossed a beanbag with the intent to hit a picture frame that held a $100 bill or was empty. No financial reward or outcome was tied to the toss itself. Thus, the toss served solely as a behavioral measure of perceptual experience. By dissociating the perceptual measure from attainment of the reward, we reduced the likelihood that the measure reflected task demands.
These five techniques limit the opportunity for task demand and improve on F&S's suggestion to combat demand by manipulating expectations. We encourage researchers – including F&S – to apply the same rigorous standards not only to the analysis of studies that seek to provide confirmatory evidence for top-down effects, but also to studies that seek to provide disconfirmatory evidence.