In the target article, Rosenholtz describes how limitations of peripheral vision can explain some visual phenomena that had previously been explained in terms of attention, including performance on visual search, as well as some change blindness illusions. She adds that her proposal “has as-yet-unrealized implications for all later visual processing, as vision scientists must re-envision how the visual system finds perceptually meaningful groups and computes properties such as 3D shape from the information available from summary statistics.”

Indeed, a complementary line of work claims that human vision does not construct a single coherent 3D mental representation, but is instead fragmentary (Linton et al., Reference Linton, Morgan, Read, Vishwanath, Creem-Regehr and Domini2023), as evidenced by many experiments in which viewers show behavior inconsistent with any single 3D mental world model. Rather, mental representations of 3D are fragmented in various ways, e.g., different mental representations used for different distance ranges (Vishwanath, Reference Vishwanath2023) or different tasks (Glennerster, Rogers, & Bradshaw, Reference Glennerster, Rogers and Bradshaw1996; Norman et al., Reference Norman, Crabtree, Clayton and Norman2005). A possible connection to the target article is suggested by Koenderink et al. (Reference Koenderink, van Doorn, Kappers, Doumen and Todd2008) argument that visual space varies with fixation, as well as the fact that some fragmentary 3D phenomena could broadly be described in terms of change blindness, e.g., (Glennerster et al., Reference Glennerster, Tcheang, Gilson, Fitzgibbon and Parker2006).

For example, consider redirected walking (Fan, Li, & Shi, Reference Fan, Li and Shi2022), a technique for giving virtual reality users the feeling of walking in a much larger virtual environment than they physically inhabit. In two notable redirected walking methods, the entire virtual world undergoes instantaneous, discrete rotations during eye blinks (Langbehn et al., Reference Langbehn, Steinicke, Lappe, Welch and Bruder2018) or saccades (Sun et al., Reference Sun, Patney, Wei, Shapira, Lu, Asente and Kaufman2018). If it were the case that users maintained detailed mental world models across fixations, then this world rotation would be very jarring. But users do not notice the rotations.

I have suggested combining Rosenholtz’s proposals with fragmentary 3D vision, into a hypothesis in which 3D vision is fragmentary across fixations (Hertzmann, Reference Hertzmann2024). At any instant, vision interprets 3D scene shape and appearance in a manner constrained by the limitations of peripheral vision. Then, rather than continually merging information into a dense 3D scene representation, the brain discards most information from each instant; only some abstracted spatial and scene awareness persists over time. Fragmentariness-across-fixations answers long-standing questions about perspective and shape perception in pictures: pictures that are inconsistent with any single fixed viewpoint can nonetheless give compelling illusions of 3D shape, so long as the illusion is effective for each individual eye fixation, and separate fixations share some limited consistency in peripheral vision and spatial memory. This explains the effectiveness of multiperspective techniques in art and photography, e.g. (Agrawala, Zorin, & Munzner, Reference Agrawala, Zorin and Munzner2000; Agarwala et al., Reference Agarwala, Agrawala, Cohen, Salesin and Szeliski2006; Perona, Reference Perona2013), over conventional linear perspective. Even impossible figures give some illusion of reality, so long as the inconsistencies are imperceptible within individual fixations (Rosenholtz, Reference Rosenholtz2011).

Together, the notions of summary statistics and fragmentary 3D vision suggest a view of visual processing as lossy: many stages of visual processing discard information. Information is discarded during summary statistics pooling, per-fixation scene understanding, visual memory storage and retrieval, and 3D scene representation. Of course, information must be discarded; we could not hope to store and access every retinal photoreceptor activation, and summary statistics are a sort of lossy image compression. But which information can be discarded, and when? A natural principle is that information loss can occur whenever information is redundant or inessential for ecologically valid vision. Indeed, several authors have argued that viewers need not maintain consistent mental world models, since viewers can rely on the consistency of the real world (Brooks, Reference Brooks1990; Koenderink, Reference Koenderink1998; O’Regan, Reference O’Regan1992). Our evolutionary history gives us no need to detect instantaneous world rotations that occur only during eye blinks.

Viewers do maintain some 3D awareness of the world outside of the current gaze. If I look at a houseplant and then look away, I can remember the houseplant and its general shape and characteristics – I have some notion of “what it looks like” – but I will not remember every leaf, or even how many leaves there were, unless I had deliberately studied these details and committed them to memory. Likewise, if I see a houseplant in peripheral vision, I get much less information about it than if I were to look right at it.

Illusions result from unaware information loss. We are often aware of having forgotten information we “ought” to have access to (“I forgot where I put my keys”), and we can be aware of having missed something for not having “paid attention.” But, most visual information loss happens without awareness, and so illusions can occur when the corresponding consistency assumptions are violated – such as in laboratory experiments that probe visual awareness. The two redirected walking methods that I mentioned above each exploit change blindness from viewers discarding visuospatial information.

In the spirit of the target article, attention represents one more way that information can be discarded, i.e., because it is not selected. Understanding attention as a distinct phenomenon requires identifying which information loss cannot be explained solely by passive, automatic information loss at some stage of processing.

Recent studies point out ways that 3D vision is fragmentary – and, often, depends on the task given to participants – but offer few explanations, and rarely take gaze or peripheral vision into account. Taking into account gaze and information loss in the stages of visual processing could help contribute to a unified understanding from the retina to 3D visual cognition.