We welcome Hulleman & Olivers' (H&O's) invitation to abandon the display item as the fundamental unit of visual search. There is now considerable evidence – some of which we have contributed to (Enns & Kingstone Reference Enns and Kingstone1995; Fecteau et al. Reference Fecteau, Enns and Kingstone2000; Roggeveen et al. Reference Roggeveen, Kingstone and Enns2004; van Zoest et al. Reference van Zoest, Giesbrecht, Enns and Kingstone2006) – that display items cannot be considered in isolation from the items around them, nor from the limits of the observer's functional viewing field (FVF). However, H&O's call to revolution does not go far enough because they simply replace one operational unit (the experimenter-defined item in a search display) with another (the observer's FVF, as indexed by fixations). Both of these efforts to ground theories of search in easily observable third-person variables neglect the most important factor: the observer's mind. It is our view that what happens behind the observer's eyes is more important than what happens in front of them (the display items) or even in them (the FVF).
Fixations during visual search cannot be considered in isolation; they are always involved in a trading relationship with saccades. That is, at any given moment the observer is engaged in strategic decisions (albeit implicit ones) to keep their eyes still (allowing for seeing, the ability to distinguish targets from non-targets) or to move them (allowing for looking, the acquisition of new information from outside of the current fixation). Similar to agents in classical reinforcement learning models (Sutton & Barto Reference Sutton and Barto1998), who trade off between exploiting currently available resources and exploring for novel resources, visual searchers cycle between seeing the information in their current fixation and looking to a new location. In typical search tasks this cycle is repeated 3–4 times per second, which is consistent with H&O's decision to model fixations as lasting 250 ms. Yet setting this average time as a constant conceals important variability.
We recently manipulated the mental strategies of participants by randomly assigning them to either search passively, by giving them instructions to “let the target pop into your mind,” or to search actively, by telling them to “deliberately direct your attention.” We found a passive response time advantage, as in previous studies (Smilek et al. Reference Smilek, Enns, Eastwood and Merikle2006), and showed that this stemmed from the inherent trade-off between seeing and looking (Watson et al Reference Watson, Brennan, Kingstone and Enns2010). Passively instructed participants made fewer fixations of longer duration, and once they fixated the target region for the first time, they made fewer subsequent fixations, responding more quickly. This suggests that they placed a greater emphasis on seeing rather than looking, and so were better able to process the target. These differences may not reflect differences in FVF: passively instructed participants were no more likely to fixate closer to the center of the display, nor further away from individual items, either of which would have allowed them to take advantage of a larger FVF.
One of the most striking observations in Watson et al. (Reference Watson, Brennan, Kingstone and Enns2010) is that there is more than one way to succeed in visual search. Even after trimming participants from each group to equate overall speed and accuracy, passively instructed participants made fewer fixations separated by larger saccades. This means it is possible to trade the higher information resolution of seeing with the greater information acquisition of looking, without affecting overall success.
Another demonstration of the critical importance of mind comes from studies of rapid resumption of visual search, in which participants are able to accurately respond very rapidly (within 100–400 ms) to a display that has been re-presented following a brief interruption (Lleras et al. Reference Lleras, Rensink and Enns2005). To be as accurate, the same responses to the first look at a display take more than 500 ms. Successful rapid resumption of an interrupted search depends on participant's forming a mental prediction of the target's response-relevant features and location based on a first look at the display. When these features or location change after the first look, but all other aspects of the display remain constant, rapid resumption is eliminated (Lleras et al. Reference Lleras, Rensink and Enns2007). The prediction is more likely to be made when fixations are located close to the target, but it turns out that fixation location is not the determining factor either. When gaze-contingent displays are used, such that the target is always presented at fixation, rapid resumption is impossible (van Zoest et al. Reference van Zoest, Lleras, Kingstone and Enns2007). Once again, differences in the mind lead to differences in the processing that occurs during a fixation. If a correct prediction about the target has been made during the first glance, the second glance enables rapid responses; if this prediction has not been made, the search must be started over. This critical predictive aspect of visual search seems absent from H&O's account.
The role of fixations depends on mind-set, both between-subjects (as shown by our instructional study) and within-subjects (as shown by rapid resumption studies). H&O's account treats fixations as uniform, which is a serious oversimplification.
We conclude by reiterating three points Hochberg (Reference Hochberg and Haber1968) made long ago, recently updated in a series of studies in Peterson et al. (Reference Peterson, Gillam and Sedgwick2006). First, because every percept enters the mind through piecemeal views, a theory of perception must be about the mind's representation, not its trigger (the stimulus), nor its conduit (the eye). Second, as Hochberg liked to put it, “unlike objects themselves, our perception of objects is not everywhere dense” (Hochberg Reference Hochberg and Beck1982, p. 214). He contributed numerous demonstrations of the selectivity of perception, both in its overt actions (fixations) and in its covert processing (attention). Third, there is no perception in a glance that is divorced from prior mental representation. This recursive aspect of perception means that vision is as much influenced by what lies in the mind as what lies in the eye of the beholder.
We welcome Hulleman & Olivers' (H&O's) invitation to abandon the display item as the fundamental unit of visual search. There is now considerable evidence – some of which we have contributed to (Enns & Kingstone Reference Enns and Kingstone1995; Fecteau et al. Reference Fecteau, Enns and Kingstone2000; Roggeveen et al. Reference Roggeveen, Kingstone and Enns2004; van Zoest et al. Reference van Zoest, Giesbrecht, Enns and Kingstone2006) – that display items cannot be considered in isolation from the items around them, nor from the limits of the observer's functional viewing field (FVF). However, H&O's call to revolution does not go far enough because they simply replace one operational unit (the experimenter-defined item in a search display) with another (the observer's FVF, as indexed by fixations). Both of these efforts to ground theories of search in easily observable third-person variables neglect the most important factor: the observer's mind. It is our view that what happens behind the observer's eyes is more important than what happens in front of them (the display items) or even in them (the FVF).
Fixations during visual search cannot be considered in isolation; they are always involved in a trading relationship with saccades. That is, at any given moment the observer is engaged in strategic decisions (albeit implicit ones) to keep their eyes still (allowing for seeing, the ability to distinguish targets from non-targets) or to move them (allowing for looking, the acquisition of new information from outside of the current fixation). Similar to agents in classical reinforcement learning models (Sutton & Barto Reference Sutton and Barto1998), who trade off between exploiting currently available resources and exploring for novel resources, visual searchers cycle between seeing the information in their current fixation and looking to a new location. In typical search tasks this cycle is repeated 3–4 times per second, which is consistent with H&O's decision to model fixations as lasting 250 ms. Yet setting this average time as a constant conceals important variability.
We recently manipulated the mental strategies of participants by randomly assigning them to either search passively, by giving them instructions to “let the target pop into your mind,” or to search actively, by telling them to “deliberately direct your attention.” We found a passive response time advantage, as in previous studies (Smilek et al. Reference Smilek, Enns, Eastwood and Merikle2006), and showed that this stemmed from the inherent trade-off between seeing and looking (Watson et al Reference Watson, Brennan, Kingstone and Enns2010). Passively instructed participants made fewer fixations of longer duration, and once they fixated the target region for the first time, they made fewer subsequent fixations, responding more quickly. This suggests that they placed a greater emphasis on seeing rather than looking, and so were better able to process the target. These differences may not reflect differences in FVF: passively instructed participants were no more likely to fixate closer to the center of the display, nor further away from individual items, either of which would have allowed them to take advantage of a larger FVF.
One of the most striking observations in Watson et al. (Reference Watson, Brennan, Kingstone and Enns2010) is that there is more than one way to succeed in visual search. Even after trimming participants from each group to equate overall speed and accuracy, passively instructed participants made fewer fixations separated by larger saccades. This means it is possible to trade the higher information resolution of seeing with the greater information acquisition of looking, without affecting overall success.
Another demonstration of the critical importance of mind comes from studies of rapid resumption of visual search, in which participants are able to accurately respond very rapidly (within 100–400 ms) to a display that has been re-presented following a brief interruption (Lleras et al. Reference Lleras, Rensink and Enns2005). To be as accurate, the same responses to the first look at a display take more than 500 ms. Successful rapid resumption of an interrupted search depends on participant's forming a mental prediction of the target's response-relevant features and location based on a first look at the display. When these features or location change after the first look, but all other aspects of the display remain constant, rapid resumption is eliminated (Lleras et al. Reference Lleras, Rensink and Enns2007). The prediction is more likely to be made when fixations are located close to the target, but it turns out that fixation location is not the determining factor either. When gaze-contingent displays are used, such that the target is always presented at fixation, rapid resumption is impossible (van Zoest et al. Reference van Zoest, Lleras, Kingstone and Enns2007). Once again, differences in the mind lead to differences in the processing that occurs during a fixation. If a correct prediction about the target has been made during the first glance, the second glance enables rapid responses; if this prediction has not been made, the search must be started over. This critical predictive aspect of visual search seems absent from H&O's account.
The role of fixations depends on mind-set, both between-subjects (as shown by our instructional study) and within-subjects (as shown by rapid resumption studies). H&O's account treats fixations as uniform, which is a serious oversimplification.
We conclude by reiterating three points Hochberg (Reference Hochberg and Haber1968) made long ago, recently updated in a series of studies in Peterson et al. (Reference Peterson, Gillam and Sedgwick2006). First, because every percept enters the mind through piecemeal views, a theory of perception must be about the mind's representation, not its trigger (the stimulus), nor its conduit (the eye). Second, as Hochberg liked to put it, “unlike objects themselves, our perception of objects is not everywhere dense” (Hochberg Reference Hochberg and Beck1982, p. 214). He contributed numerous demonstrations of the selectivity of perception, both in its overt actions (fixations) and in its covert processing (attention). Third, there is no perception in a glance that is divorced from prior mental representation. This recursive aspect of perception means that vision is as much influenced by what lies in the mind as what lies in the eye of the beholder.