We agree with Bowers et al. that accounting for results from behavioral experiments should serve as a North Star as we develop models of human vision. But what is a promising path to finding models that perform well on experimental benchmarks? In this commentary, we focus on the role of the task(s) on which models are trained. Zhang, Bengio, Hardt, Recht, and Vinyals (Reference Zhang, Bengio, Hardt, Recht and Vinyals2017) have shown that modern deep neural networks (DNNs) are more than expressive enough to overfit to any classification task on which they are trained. In particular, the authors show that DNNs can learn to classify ImageNet images (Deng et al., Reference Deng, Dong, Socher, Li, Li and Fei-Fei2009) with arbitrarily shuffled labels, demonstrating maximal flexibility with respect to this training set. To introduce a metaphor, our models are like sponges, capable of absorbing whatever information we teach them through the training tasks we present. Thus, when we ask about a model's behavior, we should ask, first, what it was trained to do. Although Bowers et al. take failures of ImageNet-trained models to behave in human-like ways as support for abandoning DNN architectures, we argue that we should instead consider alternative training tasks for DNNs.

Recent work has shown that pushing DNNs to perform well on ImageNet may not, in general, push them to be more human-like. Very high ImageNet performance becomes inversely related to primate neural predictivity (Schrimpf et al., Reference Schrimpf, Kubilius, Hong, Majaj, Rajalingham, Issa and DiCarlo2018; Schrimpf, Reference Schrimpf2022), and there is a tradeoff between perceptual scores from human judgments (Kumar, Houlsby, Kalchbrenner, & Cubuk, Reference Kumar, Houlsby, Kalchbrenner and Cubuk2022) and ImageNet performance, and between shape bias and ImageNet performance, when shape bias is modulated by data augmentation (Hermann, Chen, & Kornblith, Reference Hermann, Chen and Kornblith2020).

Certainly, humans can categorize the objects they see, but categorization is only a small part of how we process the visual world. Mostly, we use our visual systems to interact with the objects around us, in a closed loop comprising perception, inference, decision making, and action. There are several reasons to believe that training models on similarly embodied and active learning tasks may bring their behavior and representations closer to humans'. First, physically interacting with objects requires detailed perception of their global spatial properties (shape, position, motor affordances, etc.). Arguably, several of the most famous divergences between models and people stem from models' failures to weigh exactly this kind of information. For example, unlike people (Kucker et al., Reference Kucker, Samuelson, Perry, Yoshida, Colunga, Lorenz and Smith2019; Landau, Smith, & Jones, Reference Landau, Smith and Jones1988), many standard DNNs seem to rely on texture information more than shape (Baker, Lu, Erlikhman, & Kellman, Reference Baker, Lu, Erlikhman and Kellman2018; Geirhos et al., Reference Geirhos, Rubisch, Michaelis, Bethge, Wichmann and Brendel2019; Hermann et al., Reference Hermann, Chen and Kornblith2020). While, empirically, texture seems to be sufficient for good performance on ImageNet, it is unlikely to suffice for embodied navigation or manipulation tasks. In determining how to position oneself to sit in a chair, the shape and position of the chair are far more important than its color or upholstery texture. Similarly, adversarial examples (Nguyen, Yosinski, & Clune, Reference Nguyen, Yosinski and Clune2015; Szegedy et al., Reference Szegedy, Zaremba, Sutskever, Bruna, Erhan, Goodfellow and Fergus2013), another often-cited separator of humans and DNNs, arguably arise from models' over-reliance on local pixel patterns at the expense of the global configural information required for embodied interaction. Overall, we hypothesize that existing DNN architectures, if trained to navigate the world and interact with objects in the way that humans do, would be more likely to display human-like visual behavior and representations than they do under current training methods.

Another implication of the Zhang et al. (Reference Zhang, Bengio, Hardt, Recht and Vinyals2017) work is that modern networks are sufficiently large that training them on a 1,000-way classification task on a million images is insufficient to exhaust their capacity, leaving important degrees of freedom governing their generalization performance underconstrained, which allows for deviant phenomena such as adversarial examples of the kind and severity currently observed. As another example of flexibility in how DNNs can learn a classification task, models often learn spurious/shortcut features (Arjovsky, Bottou, Gulrajani, & Lopez-Paz, Reference Arjovsky, Bottou, Gulrajani and Lopez-Paz2019; Geirhos et al., Reference Geirhos, Jacobsen, Michaelis, Zemel, Brendel, Bethge and Wichmann2020; McCoy, Pavlick, & Linzen, Reference McCoy, Pavlick and Linzen2020), for example, using image backgrounds rather than foreground objects (Beery, Van Horn, & Perona, Reference Beery, Van Horn and Perona2018; Xiao, Engstrom, Ilyas, & Madry, Reference Xiao, Engstrom, Ilyas and Madry2021), or single diagnostic pixels rather than other image content (Malhotra, Evans, & Bowers, Reference Malhotra, Evans and Bowers2020). This brings us to a second argument in favor of embodied training tasks. A dataset of similar size to ImageNet but with a richer, more ecological output space – for example, choosing a physical action and its control parameters, or predicting subsequent frames – would contain a vastly larger amount of information, perhaps more fully constraining the model's behavior.

Existing work validates the impact of training tasks on model behavior and representations. Even when restricted to training on ImageNet images, the training objective and/or data augmentation can affect how well models match human similarity judgments of images (Muttenthaler, Dippel, Linhardt, Vandermeulen, & Kornblith, Reference Muttenthaler, Dippel, Linhardt, Vandermeulen and Kornblith2023), categorization patterns (Geirhos et al., Reference Geirhos, Narayanappa, Mitzkus, Thieringer, Bethge, Wichmann and Brendel2021), performance on real-time and life-long learning benchmarks (Zhuang et al., Reference Zhuang, Xiang, Bai, Jia, Turk-Browne, Norman and Yamins2022), and feature preferences (Hermann et al., Reference Hermann, Chen and Kornblith2020), and also how well they predict primate physiology (Zhuang et al., Reference Zhuang, Yan, Nayebi, Schrimpf, Frank, DiCarlo and Yamins2021) and human fMRI (Konkle & Alvarez, Reference Konkle and Alvarez2022) data. Still, it is possible to enrich DNN training tasks much further, even for object categorization (Sun, Shrivastava, Singh, & Gupta, Reference Sun, Shrivastava, Singh and Gupta2017).

We have discussed the promise of training embodied, interactive agents in rich, ethologically relevant environments. What efforts have already been made in this direction, and what might they look like in the future? Past work situating a vision system within a simulated agent navigating and interacting with its environment gives promising initial indications that human-like visual behaviors can emerge in this setting (Haber, Mrowca, Wang, Fei-Fei, & Yamins, Reference Haber, Mrowca, Wang, Fei-Fei and Yamins2018; Hill et al., Reference Hill, Lampinen, Schneider, Clark, Botvinick, McClelland and Santoro2020; Nayebi et al., Reference Nayebi, Kong, Zhuang, Gardner, Norcia and Yamins2021; Weihs et al., Reference Weihs, Kembhavi, Ehsani, Pratt, Han, Herrasti and Farhadi2021). The continued development of new, more naturalistic training environments (Gan et al., Reference Gan, Schwartz, Alter, Schrimpf, Traer, De Freitas and Yamins2021; Greff et al., Reference Greff, Belletti, Beyer, Doersch, Du, Duckworth and Tagliasacchi2022; Puig et al., Reference Puig, Ra, Boben, Li, Wang, Fidler and Torralba2018; Savva et al., Reference Savva, Kadian, Maksymets, Zhao, Wijmans, Jain and Batra2019; Xiang et al., Reference Xiang, Qin, Mo, Xia, Zhu, Liu and Su2020) should support pushing this research program still further toward human-like learning. In addition, state-of-the-art large language models provide a new means of communicating richer tasks to models (Chen et al., Reference Chen, Wang, Changpinyo, Piergiovanni, Padlewski, Salz and Soricut2023), and a new reservoir of human-like knowledge for models to draw on (Brohan et al., Reference Brohan, Chebotar, Finn, Hausman, Herzog, Ho and Fu2023). We predict further work in these directions will address shortcomings Bowers et al. identify and yield improved DNN accounts of human vision.