The trouble with text

For language comprehension, LLMs operate on text that has already discretized the continuous auditory speech stream into things that look like words, morphemes, syllables, and/or phonological segments (vowels and consonants), depending on the language. Conversely, crucial phonological aspects of spoken language like prosody and intonation aren’t usually represented in text. As a result, ambiguities differ between text and speech: English text but not speech distinguishes between sun and son, while English speech but not text disambiguates the polar vs. alternative readings of Do you like coffee or tea? In general, text-based models end up solving fundamentally different problems than human spoken language users, as both cognitive scientists and language technologists have noted before (Chrupała, Reference Chrupała2023; Dupoux, Reference Dupoux2018).

The structures of natural languages reflect overwhelmingly the properties of speech (or signing), rather than those of text. Merely enriching text-based LLMs with speech capacities through acoustic tokens or separate speech recognition/synthesis components (Arora et al., Reference Arora, Chang, Chien, Peng, Wu, Adi, Dupoux, Lee, Livescu and Watanabe2025) cannot resolve the deep trouble caused by textual bottlenecks in linguistic modelling. Researchers interested in modelling linguistic structure should therefore not settle for LLMs but rather learn to love models designed to capture the more natural form of human spoken language: the speech signal itself.

Linguistic structure in models of speech

Many current speech-technological applications employ self-supervised speech foundation models, i.e., deep learning architectures trained to represent audio signals on the basis of unlabelled speech recordings. Parallel to work on the interpretability of text-based LLMs (as covered by F&M), linguistic interpretability studies in the speech domain investigate to what extent speech-based models learn to capture any higher-level patterns that make up spoken language. Method-wise, most of these studies use representational probes to examine what linguistic information is represented in speech models’ internal states, with insightful results obtained for the encoding of linguistic units like phonemes (Alishahi, Barking, & Chrupała, Reference Alishahi, Barking and Chrupała2017; Martin et al., Reference Martin, Gauthier, Breiss and Levy2023) and words (Pasad et al., Reference Pasad, Chien, Settle and Livescu2024), as well as morphophonological (Gauthier et al., Reference Gauthier, Breiss, Leonard and Chang2025) and suprasegmental patterns (Shen et al., Reference Shen, Watkins, Alishahi, Bisazza and Chrupała2024); other studies use behavioural (minimal pair) tests to ask, e.g., whether models can distinguish words from pseudowords (Lavechin et al., Reference Lavechin, Sy, Titeux, Blandón, Räsänen, Bredin, Dupoux and Cristia2023) or prosodically natural vs. unnatural pauses (de Seyssel et al., Reference de Seyssel, Lavechin, Titeux, Thomas, Virlet, Revilla, Wisniewski, Ludusan and Dupoux2023). With these types of methods, speech models can be used as “psycholinguistic participants,” mimicking human speech perception experiments and their results, e.g., about perceptual similarity (Millet & Dunbar, Reference Millet and Dunbar2022) and phonetic categorization (de Heer Kloots & Zuidema, Reference de Heer Kloots and Zuidema2024).

Linguists can already draw an important theoretical insight from these findings: speech-only models can indeed learn relevant patterns of linguistic structure, without the pre-existing symbolic categories assumed by earlier connectionist work (McClelland & Elman, Reference McClelland and Elman1986; Smolensky, Reference Smolensky1990, Reference Smolensky1999).

Inductive biases from domain-general perception

Further investigations of what deep learning models learn from audio can potentially inform what mechanisms drive language-relevant perceptual learning in humans. Here, we agree with F&M that deep learning models can helpfully serve as proof-of-concepts for ideas that were previously hard to formalize. At least part of the human perceptual system involved in speech processing is also involved in processing a wide variety of other auditory signals. Deep learning models pre-trained on music and/or environmental sounds show more human-like behaviour than models trained on speech sounds alone, in detecting algebraic patterns in tone or syllable sequences (Orhan, Boubenec, & King, Reference Orhan, Boubenec and King2025) and in displaying native language effects when processing foreign speech sounds (Poli et al., Reference Poli, Schatz, Dupoux and Lavechin2024). Hence, it seems that some inductive biases relevant to the encoding of language-like structures can result from (evolutionarily) optimizing the perceptual system for representing sounds other than speech, perhaps more so than considered before (Aslin & Pisoni, Reference Aslin, Pisoni, Yeni-Komshian, Kavanagh and Ferguson1980; Soderstrom, Mathis, & Smolensky, Reference Soderstrom, Mathis, Smolensky, Smolensky and Legendre2006).

Inductive biases from bidirectional processing

One property of much cognitively grounded neural modelling, that is not shared with current technological speech or text machines, is that connection weights are symmetric (Kohonen, Reference Kohonen1982; Hopfield, Reference Hopfield1982; McMurray et al., Reference McMurray, Horst, Toscano, Samuelson, Spencer, Thomas and McClelland2009; Salakhutdinov & Hinton, Reference Salakhutdinov and Hinton2009). In linguistics, this corresponds to a specific inductive bias, namely that language employs bidirectional processing, i.e., language comprehension and language production utilize the same knowledge.

In phonetics and phonology, neural models with bidirectional connection weights straightforwardly predict the diachronic evolution of auditory dispersion in phoneme inventories (Boersma, Benders, & Seinhorst, Reference Boersma, Benders and Seinhorst2020). Unpublished simulations show that such communicative-success-optimizing effects readily transfer to other linguistic subdomains: with bidirectional connection weights, the maxims of Grice (Reference Grice, Cole and Morgan1975) emerge in pragmatics, as do anti-synonymity effects in semantics, morphology, and syntax (Boersma, Reference Boersma2009).

A drawback of small neural models is that they work exclusively on toy problems, rather than whole languages. Hopefully, evidence from toy models can nevertheless inspire speech technologists in architectural design. Most findings cited above were obtained with speech models trained on developmentally realistic amounts of speech (< 1000 hours); beyond improving data efficiency, exploring a variety of inductive biases is primarily crucial in developing more human-like systems with relevance for linguistic theory.

Take-home message

We agree that neural network models developed for technological applications can provide insights into human language and cognition. The early connectionist work that F&M describe as preceding the LLM era included efforts to take the continuous speech signal seriously (Elman & Zipser, Reference Elman and Zipser1988; Norris, Reference Norris1994), and we argue that current linguistic investigations with deep learning models can and should do the same. Once the bottleneck of text can be replaced, we stand a better chance of modelling more human-like linguistic processes, as well as handling the vast majority of local and regional language varieties that are rarely or never written down.