Language model training data are distinct not only in their quantity. In comparison to classic linguistic and cognitive models, language model training data are fundamentally richer, more diverse, and much noisier. Yet, they are still missing much of the diversity of human learning experiences. While Futrell & Mahowald discuss the role of data quantity in model training, they do not focus on the important role of data richness: that the diversity of variation captured in data can strongly affect the generalization of a learning system (cf. Raviv, Lupyan, & Green, Reference Raviv, Lupyan and Green2022). This principle that learning diversity drives generalization has been frequently observed in machine learning, within and beyond the language domain (e.g., del Rio et al., Reference del Rio, Raymond-Sáez, Florea, Icarte, Hurtado, Calderon and Soto2025; Goyal et al., Reference Goyal, Duval, Seessel, Caron, Misra, Sagun, Joulin and Bojanowski2022; Yu, Khadivi, & Xu, Reference Yu, Khadivi and Xu2022).

One illustrative example of this can be found in some of the target authors’ own work, though it is not emphasized as such in the target article: Misra and Mahowald (Reference Misra and Mahowald2024) show that under the same train-test syntactic generalization split, variability in the semantic fillers observed in training data affects the degree of syntactic generalization. That is, variation along non-syntactic axes can enhance generalization to novel syntactic structures. (Of course, seeing greater variability in syntactic structures will also typically improve generalization; e.g., Ahuja et al., Reference Ahuja, Balachandran, Panwar, He, Smith, Goyal and Tsvetkov2025; Qin, Saphra, & Alvarez-Melis, Reference Qin, Saphra and Alvarez-Melis2025).

Our prior work (Hill et al., Reference Hill, Lampinen, Schneider, Clark, Botvinick, McClelland and Santoro2020) on compositional generalization of language-learning agents shows a more dramatic effect: agents trained to perform classification through interaction in 3D environments show strong compositional language generalization, while agents trained to perform the same tasks on 2D images from the same environments generalize much more poorly. Thus, when comparing two language-learning systems – both of which are grounded, in the sense that their training tasks involve producing responses that depend on linking language to visual inputs and action outputs – the richness of the environmental grounding can significantly alter linguistic generalization. At a more basic level, various works have found that providing some type of semantic grounding can enhance syntactic generalization compared to systems that lack such grounding (e.g., Yedetore & Kim, Reference Yedetore and Kim2024).

These works showing the contribution of data diversity to generalization in controlled experimental settings complement the many works showing its importance in practical language model training. For example, targeting sampling of training data for diversity can improve out-of-distribution perplexity (Yu et al., Reference Yu, Khadivi and Xu2022). Conversely, language model training data collected from the internet often lacks diversity; for example, Lee et al. (Reference Lee, Ippolito, Nystrom, Zhang, Eck, Callison-Burch and Carlini2022) highlight a single sentence that appears over 60,000 times in a standard training corpus. What might naively seem to be 3.5 million tokens of unique training data (almost a year of a child’s experience) turns out to be only 61 words of unique content – an important caveat to remember when seeing naive numeric comparisons of learning efficiency. While this is a particularly egregious example, the problem of partially duplicated training data is prevalent, and Lee et al. (Reference Lee, Ippolito, Nystrom, Zhang, Eck, Callison-Burch and Carlini2022) show that reducing this duplication can substantially improve performance as well as training efficiency. This work provides another example of how learning diversity is important in practice as well as in theory: repeating data can actually harm performance.

These works illustrate how – both in controlled settings and in practice when training language models – richer variation in learning experiences can fundamentally change the resulting patterns of generalization. Importantly, this is true both for variation along the axes on which generalization will be assessed (e.g., the diversity of syntactic structures seen), but also along axes of variation that seem orthogonal to the syntactic or compositional language generalization in question (e.g., the visual richness of multimodal inputs).

These findings have important theoretical implications for the linguistic and cognitive issues at play in the target article. In particular, they suggest that at least some of the benefit of data quantity for model performance may really be a benefit of data diversity, which only happens to be achieved through scaling the datasets. In itself, I believe this lesson is important for linguistics and cognitive science: how many famous negative results about neural networks (or other models) failing to generalize were wild extrapolations from training models on toy data that lacked all the richness of human experiences? We need better theories of how diversity contributes to generalization, and we should discount models that neglect the diversity of human learning experiences without a compelling reason.

Yet if language models benefit from data diversity, why do they still need so much more data than humans? Of course, many factors may contribute; choices like data-deduplication strategies, tokenization, and optimizers can all drastically change learning efficiency; we probably have not saturated improvements along any of these dimensions. However, I suggest that differences in the richness of the input humans experience may play a crucial role. Human linguistic inputs begin as auditory signals rather than discrete tokens; these signals carry much more information to predict. Furthermore, human multimodal grounding is much richer. Indeed, while many models are now trained in part with multimodal inputs, for various reasons, language still tends to dominate (e.g., Sim et al., Reference Sim, Zhang, Dai and Fang2025) – whereas for humans, language develops more slowly than other systems for interaction. Indeed, there have been many arguments that features like prosody (e.g., Gervain, Christophe, & Mazuka, Reference Gervain, Christophe and Mazuka2020), gesture (e.g., Iverson & Goldin-Meadow, Reference Iverson and Goldin-Meadow2005), and joint attention (Tomasello & Farrar, Reference Tomasello and Farrar1986) play an important role in human language development. Current language model training paradigms lack these, and many other rich features of human learning. However, simulating richer learning experiences is increasingly achievable. Thus, I believe that the modern paradigm of data-driven machine learning opens up many promising directions for exploring the role of linguistic and nonlinguistic data features in language acquisition (cf. Carvalho & Lampinen, Reference Carvalho and Lampinen2025).