Firestone & Scholl (F&S) give lip service to the claim that the real issue is that of a joint in nature between cognition and perception, but their focus in practice is on denying “true” top-down effects. For this reason they neglect the need to tweak the concepts of perception and cognition in order to home in on the joint—if there is a joint. I suggest a reorientation toward the question of whether direct, content-specific effects of cognition on perception can be eliminated by such tweaking.
I will explore this idea with regard to what is perhaps the most dramatic of the known top-down effects of cognition on perception. It has been shown that mental images can be superimposed on percepts, creating a combined quasi-perceptual state. Brockmole et al. (Reference Brockmole, Wang and Irwin2002) used a “locate the missing dot” task in which the subject's task was to move a cursor to a missing dot in a 5-by-5 array. If a partial grid of 12 dots appeared briefly followed in the same place in less than 50 ms by another partial grid of 12 different dots (that stays on the screen until the response), subjects were able to fuse the two partial grids and move the cursor to the missing dot, remembering nearly 100% of the dots on the first array. (If the subject erroneously clicked on a space in which there was a dot in the first array, that was counted as forgetting a dot in the first array.) However, if the second array was delayed to 100 ms, subjects' ability to remember the first array fell precipitously (from nearly 12 dots down to 4 dots).
Brockmole et al. explored extended delays – up to 5 seconds before the second array appeared. The amazing result they found was that if the second array of 12 dots came more than 1.5 seconds after the first array had disappeared, subjects became very accurate on the remembered dots. (See Fig. 1, following.) Instructions encouraged them to create a mental image of the first array and superimpose it on the array that remained on the screen. Brockmole et al. analyzed the subjects' errors, finding that subjects remembered more than 90% of the dots that they saw for up to 5 seconds. Further, the 1–1.5 seconds required to generate the mental image fits with independent estimates of the time required to generate a mental image (Kosslyn et al. Reference Kosslyn, Thompson and Ganis2006). Kosslyn and colleagues replicated this result with different methods (Lewis et al. Reference Lewis, Borst and Kosslyn2011).
Figure 1.
12-dot partial grid appears, then disappears. Another 12-dot partial grid appears – in the same place – up to 5 seconds later (i.e., after an interstimulus interval of up to 5 seconds). The second grid stays on the screen until the subject responds. The subject's task is to form a mental image of the first grid, superimposing it on the second grid so as to identify the dot missing from both grids. The dark line shows that subjects rarely click on an area that contains a dot on the screen. The dashed line shows that after a 1.5 second interval between the grids, subjects also rarely click on an area in which there was a dot in the remembered array. Thanks to James Brockmole for redrawing this figure.
Those results demonstrate a direct content-specific effect of cognition on perception. And effects of this sort are involved in many of the phenomena used to criticize the idea of a joint in nature between cognition and perception. Many of the effects of language and expectation on perception that opponents of a joint in nature (e.g., Lupyan Reference Lupyan2015a) cite seem very likely to involve superimposition of imagery on perception. For example, Lupyan notes that in a visual task of identifying the direction of moving dots, performance suffers when the subject hears verbs that suggest the opposite direction. Hearing a story about motion can cause motion aftereffects (Dils & Boroditsky Reference Dils and Boroditsky2010b), a point that enhances the plausibility that the result Lupyan notes involves some sort of combination of imagery and perception.
Another example: Delk and Fillenbaum (Reference Delk and Fillenbaum1965) showed that when subjects are presented with a heart shape and asked to adjust a background to match it, the background they choose is redder than if the shape is a circle or a square. As Macpherson (Reference Macpherson2012) points out, there is evidence that subjects are forming a mental image of a red heart and superimposing it on the cutout. Macpherson regards that as a case of “cognitive penetration.” But if this is cognitive penetration, why should we care about cognitive penetration? Mental imagery can be accommodated to a joint between cognition and perception by excluding these quasi-perceptual states, or alternatively, given that imagery is so slow, by excluding slow and effortful quasi-perceptual states.
In expert perception – for example, wine-tasting, or a doctor's expertise in finding tumors in x-rays – we find another illustration of the benefit of using direct, content-specific, top-down effects of cognition on perception to tweak the category of perception. Opponents of a joint (Lupyan Reference Lupyan2015a; Vetter & Newen Reference Vetter and Newen2014) often cite expert perception. But, as Fodor noted, the category of perception need not be taken to include perceptual learning (Reference Fodor1983). The concept of perception can be shrunk so as to allow diachronic cognitive penetration in perceptual learning while excluding synchronic cognitive penetration.
Moving to a new subject: F&S's treatment of attention is inadequate. Oddly, they focus on spatial (“peripheral”) attention and treat attention as akin to changing the input. However, Lupyan (Reference Lupyan2015a) is right that feature-based attention operates at all levels of the visual hierarchy (e.g., both orientations and faces) and so is not a matter of operation on input. Taking direction of motion as an example, here is how feature-based attention works: There is increased gain in neurons that prefer the attended direction; there is suppression in neurons that prefer other directions; and there is a shift in tuning curves toward the preferred direction (Carrasco Reference Carrasco2011; Sneve et al. Reference Sneve, Sreenivasan, Alaes, Endestad and Magnussen2015). Although feature-based attention is directed by cognition, it may in many cases be mediated by priming by a sample of the feature to be attended (Theeuwes Reference Theeuwes2013). These effects cannot be eliminated by tweaking the concepts. But contrary to Lupyan (Reference Lupyan2015a), feature-based attention works by well-understood perceptual mechanisms, and there is not the slightest evidence that it directly modulates conceptual cognitive representations. This is a direct content specific effect of cognition on perception but what we know of the mechanisms by which it operates gives no comfort to opponents of a joint in nature between cognition and perception.
Finally, although I have been critical of F&S, I applaud their article for its wonderful exposé of the pitfalls in experimental approaches to the joint in nature between cognition and perception.
Firestone & Scholl (F&S) give lip service to the claim that the real issue is that of a joint in nature between cognition and perception, but their focus in practice is on denying “true” top-down effects. For this reason they neglect the need to tweak the concepts of perception and cognition in order to home in on the joint—if there is a joint. I suggest a reorientation toward the question of whether direct, content-specific effects of cognition on perception can be eliminated by such tweaking.
I will explore this idea with regard to what is perhaps the most dramatic of the known top-down effects of cognition on perception. It has been shown that mental images can be superimposed on percepts, creating a combined quasi-perceptual state. Brockmole et al. (Reference Brockmole, Wang and Irwin2002) used a “locate the missing dot” task in which the subject's task was to move a cursor to a missing dot in a 5-by-5 array. If a partial grid of 12 dots appeared briefly followed in the same place in less than 50 ms by another partial grid of 12 different dots (that stays on the screen until the response), subjects were able to fuse the two partial grids and move the cursor to the missing dot, remembering nearly 100% of the dots on the first array. (If the subject erroneously clicked on a space in which there was a dot in the first array, that was counted as forgetting a dot in the first array.) However, if the second array was delayed to 100 ms, subjects' ability to remember the first array fell precipitously (from nearly 12 dots down to 4 dots).
Brockmole et al. explored extended delays – up to 5 seconds before the second array appeared. The amazing result they found was that if the second array of 12 dots came more than 1.5 seconds after the first array had disappeared, subjects became very accurate on the remembered dots. (See Fig. 1, following.) Instructions encouraged them to create a mental image of the first array and superimpose it on the array that remained on the screen. Brockmole et al. analyzed the subjects' errors, finding that subjects remembered more than 90% of the dots that they saw for up to 5 seconds. Further, the 1–1.5 seconds required to generate the mental image fits with independent estimates of the time required to generate a mental image (Kosslyn et al. Reference Kosslyn, Thompson and Ganis2006). Kosslyn and colleagues replicated this result with different methods (Lewis et al. Reference Lewis, Borst and Kosslyn2011).
12-dot partial grid appears, then disappears. Another 12-dot partial grid appears – in the same place – up to 5 seconds later (i.e., after an interstimulus interval of up to 5 seconds). The second grid stays on the screen until the subject responds. The subject's task is to form a mental image of the first grid, superimposing it on the second grid so as to identify the dot missing from both grids. The dark line shows that subjects rarely click on an area that contains a dot on the screen. The dashed line shows that after a 1.5 second interval between the grids, subjects also rarely click on an area in which there was a dot in the remembered array. Thanks to James Brockmole for redrawing this figure.
Those results demonstrate a direct content-specific effect of cognition on perception. And effects of this sort are involved in many of the phenomena used to criticize the idea of a joint in nature between cognition and perception. Many of the effects of language and expectation on perception that opponents of a joint in nature (e.g., Lupyan Reference Lupyan2015a) cite seem very likely to involve superimposition of imagery on perception. For example, Lupyan notes that in a visual task of identifying the direction of moving dots, performance suffers when the subject hears verbs that suggest the opposite direction. Hearing a story about motion can cause motion aftereffects (Dils & Boroditsky Reference Dils and Boroditsky2010b), a point that enhances the plausibility that the result Lupyan notes involves some sort of combination of imagery and perception.
Another example: Delk and Fillenbaum (Reference Delk and Fillenbaum1965) showed that when subjects are presented with a heart shape and asked to adjust a background to match it, the background they choose is redder than if the shape is a circle or a square. As Macpherson (Reference Macpherson2012) points out, there is evidence that subjects are forming a mental image of a red heart and superimposing it on the cutout. Macpherson regards that as a case of “cognitive penetration.” But if this is cognitive penetration, why should we care about cognitive penetration? Mental imagery can be accommodated to a joint between cognition and perception by excluding these quasi-perceptual states, or alternatively, given that imagery is so slow, by excluding slow and effortful quasi-perceptual states.
In expert perception – for example, wine-tasting, or a doctor's expertise in finding tumors in x-rays – we find another illustration of the benefit of using direct, content-specific, top-down effects of cognition on perception to tweak the category of perception. Opponents of a joint (Lupyan Reference Lupyan2015a; Vetter & Newen Reference Vetter and Newen2014) often cite expert perception. But, as Fodor noted, the category of perception need not be taken to include perceptual learning (Reference Fodor1983). The concept of perception can be shrunk so as to allow diachronic cognitive penetration in perceptual learning while excluding synchronic cognitive penetration.
Moving to a new subject: F&S's treatment of attention is inadequate. Oddly, they focus on spatial (“peripheral”) attention and treat attention as akin to changing the input. However, Lupyan (Reference Lupyan2015a) is right that feature-based attention operates at all levels of the visual hierarchy (e.g., both orientations and faces) and so is not a matter of operation on input. Taking direction of motion as an example, here is how feature-based attention works: There is increased gain in neurons that prefer the attended direction; there is suppression in neurons that prefer other directions; and there is a shift in tuning curves toward the preferred direction (Carrasco Reference Carrasco2011; Sneve et al. Reference Sneve, Sreenivasan, Alaes, Endestad and Magnussen2015). Although feature-based attention is directed by cognition, it may in many cases be mediated by priming by a sample of the feature to be attended (Theeuwes Reference Theeuwes2013). These effects cannot be eliminated by tweaking the concepts. But contrary to Lupyan (Reference Lupyan2015a), feature-based attention works by well-understood perceptual mechanisms, and there is not the slightest evidence that it directly modulates conceptual cognitive representations. This is a direct content specific effect of cognition on perception but what we know of the mechanisms by which it operates gives no comfort to opponents of a joint in nature between cognition and perception.
Finally, although I have been critical of F&S, I applaud their article for its wonderful exposé of the pitfalls in experimental approaches to the joint in nature between cognition and perception.