2.1. Virtual reality for language processing

Scholars have tested language processing theories in VR, simulating daily oral communication. In a real-life conversation, people rarely hear a sentence with no visual cues from interlocutors and/or surrounding environments. VR allows researchers to simulate these conditions while maintaining strict experimental control (Dijkstra & Peeters, Reference Dijkstra and Peeters2023; Huizeling et al., Reference Huizeling, Peeters and Hagoort2022; Tromp et al., Reference Tromp, Peeters, Meyer and Hagoort2018). Given that this field is still expanding, our discussion includes studies on both native (L1) and L2 processing, focusing on how experiments were conducted with VR and their implications for theories of language processing.

The studies using VR-based paradigms have examined if theories of L2 processing still hold in naturalistic environments. For example, Titus and Peeters (Reference Titus and Peeters2024) investigated how L2-English speakers with L1-Dutch code-switched between the two languages in a naturalistic setting, i.e., a virtual market in a CAVE system. While the authors found that the overall theoretical claims on inhibitory control still hold (Green & Abutalebi, Reference Green and Abutalebi2013; Meuter & Allport, Reference Meuter and Allport1999), they revealed that, only in the naturalistic virtual environment, switching from L1-Dutch to L2-English was harder than that from L2-English to L1-Dutch. The authors attributed this asymmetry to the naturalistic VR experience, which required more fluent and error-free speech production, when compared with lab-based experiments conducted on 2D screens. As a result, participants needed to exert increased inhibition on their L1 due to their lower proficiency in L2.

In addition, this line of research has examined the effects of non-linguistic sensorimotor information in L1 processing through involving virtual interlocutors to elicit naturalistic responses from the participants (e.g., Huizeling et al., Reference Huizeling, Peeters and Hagoort2022; Misersky et al., Reference Misersky, Peeters and Flecken2022; Tromp et al., Reference Tromp, Peeters, Meyer and Hagoort2018). For example, using the eye-tracking technique, Huizeling et al. (Reference Huizeling, Peeters and Hagoort2022) investigated how native Dutch speakers processed disfluent spoken sentences produced by avatars simulating native Dutch speakers in the CAVE system. As an indicator for anticipating speech in face-to-face conversations in the real world, listeners tend to look at their interlocutor’s face when disfluencies occur. The study found that the same strategy was used in the virtual environment, where participants looked at the avatar’s face upon hearing the disfluencies while waiting for the forthcoming speech. Likewise, Tromp et al. (Reference Tromp, Peeters, Meyer and Hagoort2018) employed EEG and observed consistent N400 effects in native Dutch speakers when they listened to sentences that inaccurately described the food on a table in a virtual restaurant. However, it is unclear if L2 learners adopt similar strategies when interacting with avatars in virtual environments.

Although VR as an experimental tool is promising for studying language processing, the linguistic constructs examined with this paradigm are still very limited. There are a few on speech (Huizeling et al., Reference Huizeling, Peeters and Hagoort2022; Titus & Peeters, Reference Titus and Peeters2024) and words (Huizeling et al., Reference Huizeling, Alday, Peeters and Hagoort2023), but very few studies have investigated the native processing of syntax in virtual environments (Misersky et al., Reference Misersky, Peeters and Flecken2022), and even fewer on L2 processing. To go beyond syntax, Taguchi (Reference Taguchi2022a, Reference Taguchi2022b) used VR headsets to examine pragmatics including how L1 and L2 speakers of English used intonation in dispreferred speech acts, such as complaining, in tasks presented via a VR headset or on a 2D screen. In addition, individual characteristics have been barely examined in the existing studies. The combination of the behavioral and neurocognitive measures has also been limited to eye-tracking and EEG.

To summarize, most research in this group suggests that VR can effectively elicit sensorimotor and social information of languages across participant groups even for the information that may be difficult to elicit and/or measure in conventional lab-based experiments. To reach this goal, participants in these studies usually interacted with 3D avatars in virtual environments. Broadening the field of language processing and learning, these studies demonstrate the potential of VR for examining how L2 learners process the multisensory and social information inherent in languages, contexts, and environments. The VR-based paradigms also show potential as pre- and post-test tasks for language learning studies.

2.2. Virtual reality for language learning

While studies on language processing reveal the real-time snapshots of an acquired language, language learning is a developmental process over time. One essential goal for studies on L2 learning is to understand the cognitive and neural bases of this process. Embodied cognition theories have proposed that human cognition, including L2 learning, develops when the body, mind and environments interact with each other and integrate information for learning (Atkinson, Reference Atkinson2010; Barsalou, Reference Barsalou2003, Reference Barsalou2008; Glenberg, Reference Glenberg1997). Under this framework, social L2 learning theory posits that social cues such as joint attention embedded in social interaction promote learning (Li & Jeong, Reference Li and Jeong2020; Zappa et al., Reference Zappa, Slater and Rodriguez-Fornells2025). However, it is challenging for traditional lab-based experiments on 2D screens to capture the dynamic interactions. With the help of VR, we can investigate to what extent L2 learners can encode the sensorimotor and social information of language into their mental representations during learning, and to what extent the encoded information enhances language learning.

Existing studies have primarily examined L2 vocabulary learning, with cognitive and neural measures conducted in pre- and post-tests. One line of the studies incorporated EEG during the (pre- and) post-tests to assess learning outcomes and the quality of mental representations of the learned words. In a study by Zappa et al. (Reference Zappa, Bolger, Pergandi, Fargier, Mestre and Frenck-Mestre2024), novice learners of Serbian with L1-French learned Serbian action verbs through either imitating the actions or pointing to the target verbs on a VR platform. Indicated by the increased N400 effect for semantic processing from the pre-test to the post-test, the authors found that both action methods in VR facilitated semantic encoding of the newly learned Serbian verbs. Further evidence was found by Jiao et al. (Reference Jiao, Zhu, Xu, Zhou, Schwieter and Liu2024, Reference Jiao, Lin, Schwieter and Liu2026) examining if visuospatial cues in virtual environments facilitated the learning of German nouns by L1-Chinese speakers. Their semantic priming task (2025) showed that by eliciting increased N400, VR enhanced the encoding of the learned German nouns in semantic memory.

Another line of research adopted magnetic resonance imaging (MRI) in the pre- and post-tests. Using functional MRI, scholars have shown that vocabulary learned with pictures and gestures, rather than word-word association, elicits greater activation in the visual and motor cortices during retrieval, suggesting more enriched encoding of words (Bechtold et al., Reference Bechtold, Cosper, Malyshevskaya, Montefinese, Morucci, Niccolai, Repetto, Zappa and Shtyrov2023; Mayer et al., Reference Mayer, Yildiz, Macedonia and von Kriegstein2015). Moreover, using structural MRI, Legault et al. (Reference Legault, Fang, Lan and Li2019b) investigated structural changes in learners’ brains under two learning conditions: Second Life, a desktop VR platform and a conventional picture-word association task. Novice learners of Chinese with L1-English learned 90 Chinese words over seven sessions within 20 days in one of the two learning conditions. Learning outcomes were measured at three time points, (1) after each learning session, (2) 3 weeks later and (3) 6 weeks later. MRI scans were obtained after the first and last learning sessions. The authors found that in the VR-based learning condition, the learners’ performance was positively correlated with the cortical thickness in the right inferior parietal lobe, a region associated with sensorimotor integration and embodied cognition (Claeys et al., Reference Claeys, Lindsey, De Schutter and Orban2003; Macaluso et al., Reference Macaluso, Driver and Frith2003). In contrast, the picture-word association condition was associated with cortical thickness in the right inferior frontal gyrus, indicative of semantic learning (Jones et al., Reference Jones, Green, Grogan, Pliatsikas, Filippopolitis, Ali, Lee, Ramsden, Gazarian, Prejawa, Price and Seghier2012). The findings suggest that VR-based learning may lead to distinct brain structure changes compared to the picture-word association task.

Different from the research on language processing, recent studies have explored how individual characteristics influence L2 learning with VR-based platforms. Jiao et al. (Reference Jiao, Zhu, Xu, Zhou, Schwieter and Liu2024) found that L2 age of acquisition was negatively correlated with N200 amplitude, an indicator of early lexical selection and phonological processing, whereas inhibitory control showed a null correlation. Rudling et al. (Reference Rudling, Rumetshofer, Nirme, Hedenström, Karlsson and Rydberg2026) revealed that VR-based learning platforms might benefit child L2 learners who struggled with tasks requiring sustained attention. Both Legault et al. (Reference Legault, Zhao, Chi, Chen, Klippel and Li2019a) and Macedonia et al. (Reference Macedonia, Mathias, Lehner, Reiterer and Repetto2023) reported that less successful learners especially benefited from VR learning. Macedonia et al. (Reference Macedonia, Mathias, Lehner, Reiterer and Repetto2023) suggest that the impact of VR-based motor cues, such as grasping, depends on a learner’s language aptitude, i.e., the specific cognitive abilities for L2 learning, which vary significantly across individual learners (Dörnyei & Skehan, Reference Dörnyei, Skehan, Doughty and Long2003; Rogers et al., Reference Rogers, Meara, Barnett-Legh, Curry and Davie2017). For high-aptitude learners with efficient declarative memory, these procedural cues offer negligible benefits. Conversely, for low-aptitude learners, interactive procedural activities provide a vital compensatory pathway that supplements a less efficient declarative memory system. Although the study did not directly assess the learners’ declarative and procedural memory, it represents a crucial step toward understanding how these memory systems as individual characteristics influence L2 learning in virtual environments (Morgan-Short et al., Reference Morgan-Short, Faretta-Stutenberg, Brill-Schuetz, Carpenter and Wong2014; Saito et al., Reference Saito, Cui, Suzukida, Dardon, Suzuki, Jeong, Révész, Sugiura and Tierney2022; Ullman & Lovelett, Reference Ullman and Lovelett2018).

To summarize, sensorimotor and social cues from immersive and interactive environments may facilitate L2 learning and thus contribute to the theoretical frameworks of embodied cognition and social L2 learning (see Mathias & von Kriegstein, Reference Mathias and von Kriegstein2023; Yu et al., Reference Yu, Gao, Zhao, Yu, Li, Li, Feng and Yao2026 for detailed discussion). However, the conclusion is tentative given the following limitations. First, the behavioral and neurocognitive measures have been dominantly used in the post-learning testing sessions, rather than both the learning and the testing sessions (except for Gao et al., Reference Gao, Jiang, Hu, Chan and Liunder review.a; Hsiao et al., Reference Hsiao, Lan, Kao and Li2017; see Section 3.2). Consequently, the learning processes remain a black box. Second, the neurocognitive measures in the testing sessions were based on conventional, non-embodied 2D paradigms, instead of directly assessing how well the sensorimotor and social cues of the learned language are encoded and retained (Jiao et al., Reference Jiao, Zhu, Xu, Zhou, Schwieter and Liu2024, Reference Jiao, Lin, Schwieter and Liu2026; Lin et al., Reference Lin, Schwieter, Jiao and Liu2026). It is necessary to develop novel VR-suitable tasks for the pre- and post-tests to accurately test the learning outcomes and to conduct the whole experiment in VR. Third, it is still unexplored how sensorimotor and social cues can maximize learning outcomes, especially in the long term within the constraints of human cognition (Zou et al., Reference Zou, Zhang, Mavilidi, Chen, Herold, Ouwehand and Paas2025). Commercial VR learning platforms as well as other digital language learning tools often include attractive but complex interactive features, but such features are often not tested or validated empirically and hence may be unnecessary or ineffective (Li & Lan, Reference Li and Lan2022). Future research should identify both effective and distracting design features for learners and integrate evidence-based learning strategies into immersive experiences. Such insights can greatly improve the pedagogical value of VR-based language learning platforms in the long run (Fuhrman et al., Reference Fuhrman, Eckerling, Friedmann, Tarrasch and Raz2021; Gao et al., Reference Gao, Zhou, Xia and Chenunder review.a; Hartfill et al., Reference Hartfill, Gabel, Neves-Coelho, Vogel, Räthel, Tiede, Ariza, Steinicke, Preim, Nürnberger and Hansen2020).

2.3. Gaps and areas for future development

With the above analyses, we have identified four shared areas for future advancement. First, future research should include more diverse learner populations, especially speakers of non-European languages, examine a broader range of linguistic constructs, test more environmental and social contexts and examine the interactions with interlocutors. This will contribute to the discussion on how the sensorimotor and social cues embedded in human languages and virtual environments are processed and learned within the constraints of human cognition and experience, and thus broaden the theories on language processing and learning in naturalistic embodied environments (Barsalou, Reference Barsalou2003, Reference Barsalou2008; Hagoort, Reference Hagoort2023; Mahon & Caramazza, Reference Mahon and Caramazza2008; Wilson, Reference Wilson2002; Zou et al., Reference Zou, Zhang, Mavilidi, Chen, Herold, Ouwehand and Paas2025). For example, we are currently testing if motion and spatial cues facilitate L2 learning of spatial language in a VR environment.

Second, we need to develop richer dynamic virtual environments based upon recent developments in VR technology and its integration with other technologies including Large Language Models (LLMs), portable neuroimaging devices and biofeedback (Yu et al., Reference Yu, Gao, Zhao, Yu, Li, Li, Feng and Yao2026). While LLMs enable sophisticated, naturalistic human-machine interaction, VR provides an embodied 3D context that grounds these interactions in vivid scenarios. For example, it is now feasible to generate context-appropriate, interlocutor-specific conversations by including LLM-powered AI agents in VR. This will help create a new mode of communication, moving from one-sentence responses to naturalistic human-avatar dialogues (Wan et al., Reference Wan, Zhang, Suria, Yao, Wang, Coady and Prpa2024). Additionally, by integrating measures like eye movements (Bovo et al., Reference Bovo, Abreu, Ahuja, Gonzalez, Cheng and Gonzalez-Franco2025) and fNIRS (Wen et al., Reference Wen, Middleton, Ping, Chawla, Wu, Feest, Nadri, Liu, Kaber, Zahabi, McMahan, Castelo, McKendrick, Qian and Silva2025), AI agents and VR environments can be dynamically adjusted in accordance with human real-time responses.

Third, the impacts of individual characteristics warrant comprehensive investigation. On the one hand, we need to examine how the well-established individual cognitive abilities like language aptitude, working memory span (Gao et al., Reference Gao, Jiang, Hu, Chan and Liunder review.a; Legault et al., Reference Legault, Zhao, Chi, Chen, Klippel and Li2019a), attention span (Rudling et al., Reference Rudling, Rumetshofer, Nirme, Hedenström, Karlsson and Rydberg2026), declarative and procedural memory systems (Ullman & Lovelett, Reference Ullman and Lovelett2018) and linguistic experiences such as length of residence in an L2 environment (Gao et al., Reference Gao, Jiang, Hu, Chan and Liunder review.b; Saito, Reference Saito2015) influence L2 processing and learning in naturalistic virtual environments. On the other hand, we need to examine the abilities relevant to VR such as spatial abilities, motion sickness and prior VR experience (Li et al., Reference Li, Legault, Klippel and Zhao2020; Zhao et al., Reference Zhao, Simpson, Sajjadi, Wallgrün, Li, Bagher, Oprean, Padilla, Klippel, Marchal, Ventura, Olivier, Wang and Radkowski2021). This line of inquiry will identify which learner groups benefit most from immersive environments, enabling developers to tailor VR platforms to diverse needs. Furthermore, such research deepens our understanding of the neurocognitive mechanisms for language processing and learning. For instance, it allows us to investigate whether established structural brain differences between fast and slow learners persist within ecologically valid, naturalistic settings.

Lastly, in terms of methodology, both language processing and learning research have utilized limited behavioral and neurocognitive measures in VR, although it is crucial to understand the cognitive and neural mechanisms for language processing in virtual environments, and to uncover how neural changes occur during learning with VR and how they affect both short- and long-term learning outcomes. An important direction for the future will be approaches for combining behavioral and neurocognitive measures within VR and for analyzing the multimodal data collected from such measures, to which we now turn.

3.1. Head-mounted display (HMD) and cave automatic virtual environment (CAVE) system for cognitive studies

Both HMD and CAVE are VR systems that offer users a 360° field of regard in virtual environments (Zhao et al., Reference Zhao, Wallgrün, LaFemina, Normandeau and Klippel2019). The CAVE system is a cube-shaped room with multiple projection surfaces (e.g., walls and floor) capable of displaying computer-generated 3D imagery (Cruz-Neira et al., Reference Cruz-Neira, Sandin, DeFanti, Kenyon and Hart1992; DeFanti et al., Reference DeFanti, Acevedo, Ainsworth, Brown, Cutchin, Dawe, Doerr, Johnson, Knox, Kooima, Kuester, Leigh, Long, Otto, Petrovic, Ponto, Prudhomme, Rao, Renambot and Wickham2011). Users inside the CAVE system can simultaneously explore virtual content while seeing each other’s bodies and responses. Such a shared space facilitates natural communications and collaborative demonstrations but comes at the high cost of specialized infrastructure. In contrast, HMDs provide a highly immersive, portable and cost-effective solution, allowing individual users to experience virtual environments from anywhere and with minimal visual input from the physical surroundings (Slater & Wilbur, Reference Slater and Wilbur1997). Both CAVE and HMD systems support interactive exploration through real-time motion tracking, although HMDs typically afford more embodied and tangible experiences due to their seamless integration of a virtual body and hand-based interactions (Zhao et al., Reference Zhao, LaFemina, Carr, Sajjadi, Wallgrün and Klippel2020). When wearing an HMD, users are visually isolated from the real world around them, which enhances immersion but limits social contact with other users (Lantz, Reference Lantz1996). In this regard, recent advances in networked collaborative VR now bring users from different physical locations into the same virtual environment (e.g., VRChat). Combined with fully rigged and customizable avatars, these developments are helping address earlier concerns about isolation among HMD users. Figure 1 presents an illustration of the CAVE and HMD systems and their utility for language learning research.

3.2. Tracking learning processes in virtual reality with behavioral and neurocognitive measures

As VR promises to bridge the gap between controlled experiments and real-world language learning, it becomes equally important to capture the diverse cognitive and behavioral processes that unfold along with learners’ exploration and interaction in these immersive spaces (Li et al., Reference Li, Legault, Klippel and Zhao2020; Li & Lan, Reference Li and Lan2022; Peeters, Reference Peeters2019; Zappa & Frenck-Mestre, Reference Zappa, Frenck-Mestre, Morgan-Short and Van Hell2023). Combining VR with behavioral and neurocognitive measures presents an invaluable opportunity to investigate learners’ real-time cognitive states, brain activities and behavioral patterns during ongoing learning processes (i.e., eye-brain-behavior correspondence; see Gu et al., Reference Gu, Peng, Nastase, Mayer and Li2024), thus introducing a novel way of uncovering how learning processes influence learning outcomes, which remains rarely studied in experiments with a pretest-training-posttest design, as discussed above. Advanced multimodal data analytic methods such as intersubject correlation (ISC) analysis can be used to handle these complex large datasets (Freund et al., Reference Freund, Brandmaier, Lewejohann, Kirste, Kritzler, Krüger, Sachser, Lindenberger and Kempermann2013; Madsen & Parra, Reference Madsen and Parra2022). In this section, we will discuss possibilities for the use of HMD and CAVE systems to be combined with behavioral and neurological data recordings, limitations of each combination, ways in which they may complement one another and methods for analyzing and interpreting these data.