Although we agree with Futrell and Mahowald’s (F&M) overall stance as we understand it – that language models (LMs) can serve as tools for investigating features of human language and that theoretical and computational linguistics can inform each other going forward – we question the validity and generalizability of some of F&M’s conclusions cross-linguistically. Just as the language sciences (not least early work in generative grammar) have been criticized for basing assumptions primarily on examples from English (Blasi et al., Reference Blasi, Henrich, Adamou, Kemmerer and Majid2022; Evans & Levinson, Reference Evans and Levinson2009), some of F&M’s key takeaways (notably, that LMs’ success “strengthens the case for existing usage-based language theories”) are almost exclusively based on research on model capabilities in English, impacting the validity of LMs as a language-agnostic model of human language. This is (briefly) recognized as a “caveat” at the outset, but no effort is made to engage with the obvious argument that spending years building statistical models that optimize primarily over English text data can easily lead to overfitting to English linguistic structure (an argument made at least as early as Bender (Reference Bender2011) about machine-learning systems in general). This argument is not only theoretical; previous work has produced examples where LMs show a clear preference for English-like grammars over non-English-like ones (Davis and van Schijndel Reference Davis and van Schijndel2020; Davis Reference Davis2022; Dyer, Melis, & Blunsom Reference Dyer, Melis and Blunsom2019). Most egregiously, in section 4.4, the terms “real English text” and “human language” are practically treated as equivalent when discussing the learning of “impossible languages.”

This clearly impacts the discussion of whether LMs can be considered suitable models of human language. While F&M discuss how autoregressive LMs “show a bias towards information locality” obtained through their next-word prediction training setup and how this bias “helps language learning and is likely shared with humans,” it is by no means clear that all languages are equally well-suited to the next-word prediction objective. Indeed, recent work shows that certain very robust features of Icelandic are not effectively learned by LMs trained on more Icelandic data than a human is exposed to during the language acquisition phase, likely by orders of magnitude.

In Ármannsson et al. (Reference Ármannsson, Ingimundarson, Nowenstein and Sigurðsson2026), we find that state-of-the-art LMs (as of 2024) fall remarkably short of human accuracy for robust noun-phrase internal predicate-agreement patterns in Icelandic when rating clearly grammatical/ungrammatical sentences, often preferring ungrammatical gender agreement with a pre-adjectival subject in the presence of a post-adjectival predicate. While native speakers showed 99% acceptance for the grammatical sentences and 0.3% for ungrammatical ones, the best-performing model recorded 57% accuracy, and almost all fell below chance (Figure 1). In addition, assuming that children are exposed to roughly seven to ten million words per year (Gilkerson et al., Reference Gilkerson, Richards, Warren, Montgomery, Greenwood, Kimbrough Oller, Hansen and Paul2017), this agreement pattern seems robust in children “trained” on no more than twenty million words, presumably a fraction of the models’ Icelandic training data (the largest corpus of Icelandic available online, the Icelandic Gigaword Corpus (IGC; Barkarson, Steingrímsson, & Hafsteinsdóttir, Reference Barkarson, Steingrímsson and Hafsteinsdóttir2022), contains approximately 2.6 billion words).

Figure 1.

The mean acceptance rate of sentences in the gender agreement grammaticality judgment tasks in Ármannsson et al. (Reference Ármannsson, Ingimundarson, Nowenstein and Sigurðsson2026), shown for only those sentences that contained noun-phrase-internal predicate agreement (above) and those that did not (below). Native speakers categorically accept sentences containing noun-phrase internal predicate agreement and reject those without, while no model reliably matches the judgments of human participants.

While the suitability of LMs as models of human language obviously does not hinge on this or any other single experiment, it is worth contemplating that F&M, and much of the literature they refer to, focus on work that shows LMs successfully modelling – to various degrees – “nontrivial formal linguistic patterns” in English (often quite rare constructions, e.g., Misra & Mahowald, Reference Misra and Mahowald2024). Meanwhile, examples like ours show massive models failing to successfully generalize a ubiquitous pattern that appears to be both categorical in native speakers and unanimous in corpora of Icelandic (we found zero counterexamples in our search of the morphologically tagged IGC). This is in direct contrast with, e.g., the commonly cited work of Linzen, Dupoux, and Goldberg (Reference Linzen, Dupoux and Goldberg2016), who probe models through an English pattern that is known to be variable in speakers (or in F&M’s framing, “human accuracy in producing the right verb forms […] is 85%”) and dependent on various syntactic and semantic factors (den Dikken, Reference den Dikken2001).

Overall, F&M make an obvious and commendable effort to balance their arguments with the necessary caveats, which is why we find it surprising that they eventually arrive at the conclusion that “LMs are a proof of concept for gradient, usage-based theories of language.” Usage-based theories of language, as all theories of language, are meant to be language agnostic. (They are also meant to account for how language is learned with the time and input available to a child learner, which remains the most urgent obstacle to treating LMs as theories of language. In our view, this significant issue is not addressed well enough by F&M, but we leave this line of argument to other commentators.) There is a major leap between the well-argued claim that LMs “have learned nontrivial formal linguistic patterns better than many expected was possible” and the conclusion that their success “strengthens the case for existing usage-based language theories based on gradient representations.” In order to make this leap, F&M would need to present considerably more cross-linguistic evidence than they do and extend their argument against modern computational and neurolinguistic work rooted in generative linguistics (e.g., Belth et al., Reference Belth, Payne, Beser, Kodner and Yang2021; Gwilliams et al., Reference Gwilliams, Marantz, Poeppel and King2025; Payne, Kodner, & Yang Reference Payne, Kodner and Yang2021). This work is cited to some extent, but not included under the umbrella of generative linguistics, perhaps because it does not neatly fit into a more simplistic binary characterization pitting a usage-based/statistical tradition against a generative/“anti-statistical” one.

LMs’ successful performance in processing and outputting English text can lead to overhyped claims about their approximation of the human language acquisition process in general, across any language. Similarly, it can lead to claims that LMs provide us with clear answers in debates opposing usage-based and generative theories of language, despite a growing body of work showing parallels between various aspects of linguistic theory, including generative ones, and LM performance (e.g., Caucheteux & King, Reference Caucheteux and King2022; Desbordes et al., Reference Desbordes, Lakretz, Chanoine, Oquab, Badier, Trébuchon, Carron, Bénar, Dehaene and King2023; Simon et al., Reference Simon, d’Ascoli, Chemla, Lakretz and King2024). Hopefully, future work will focus more on the nuances present in F&M’s target article and less on reductive claims about LM performance and its impact on modern linguistic theory.