1. Introduction
The rapid advancement of technology – including, more recently, developments in generative AI (GenAI) – coupled with the widespread availability of portable digital devices and an abundance of engaging online English learning resources, has profoundly reshaped the way English is learned today (Lai & Sundqvist, Reference Lai and Sundqvist2026; Liu, Soyoof, et al., Reference Soyoof, Reynolds, Neumann and Vazquez-Calvo2025). Increasingly, English as a foreign language (EFL) learners are independently using these digital tools and resources to pursue informal, self-directed English learning beyond classroom boundaries (Guo & Lee, Reference Guo and Lee2023; Soyoof et al., Reference Soyoof, Reynolds, Vazquez-Calvo and McLay2023; Zou et al., Reference Zou, Reinders and Amjad2026). This distinct phenomenon is known as informal digital learning of English (IDLE).
To precisely define IDLE, we draw on Benson’s (Reference Benson, Benson and Reinders2011) framework for learning beyond the classroom, which considers dimensions of formality, location, pedagogy, and locus of control. Within this framework, IDLE refers to self-initiated, naturalistic English learning activities that occur in out-of-class settings. Learners employ digital tools (e.g., smartphones or computers) and resources (e.g., apps, games, videos, podcasts, social media, and AI-mediated tools) to learn English without direct teacher intervention or formal assessment (Lee, Reference Lee2022b).
To situate IDLE among related constructs, we conceptualize it as part of a nested hierarchy. At the broadest level, informal second language learning (ISLL) refers to unstructured, out-of-class learning of any second language (Kusyk et al., Reference Kusyk, Arndt, Schwarz, Yibokou, Dressman, Sockett and Toffoli2025). Within ISLL, extramural English (EE; Rød & Calafato, Reference Rød and Calafato2025; Sundqvist & Sylvén, Reference Sundqvist and Sylvén2016) narrows the focus to informal contact with English outside the classroom, spanning both digital (e.g., online gaming) and non-digital (e.g., reading paperback books) activities. IDLE, in turn, can be understood as the digital-specific subset of EE, focusing mainly on online and technology-mediated informal English practices (Zou et al., Reference Zou, Reinders and Amjad2026). However, as IDLE learners tend to self-select their activities, platforms, and intensity in digital environments, IDLE is difficult to standardize and often partly unobservable. This challenge applies across diverse forms of IDLE, including gaming, video viewing, podcast listening, social media participation, online chatting, app-based practice, and AI-mediated interaction. As a result, researchers often rely on exploratory, cross-sectional, and qualitative approaches to capture naturally occurring practices and generate initial explanatory hypotheses (Lee, Reference Lee2026; Sundqvist, Reference Sundqvist2024).
To date, researchers have identified over 40 factors – both antecedents and outcomes – associated with IDLE engagement. The antecedents can be broadly grouped into three domains: (a) demographic variables, such as gender and access to digital devices (Zhang & Liu, Reference Zhang and Liu2025); (b) internal attributes, including ideal L2 self, enjoyment, basic psychological needs, and perceived usefulness of IDLE (Gao et al., Reference Gao, Wang and Reynolds2025; Liu, Darvin, & Ma, Reference Liu, Darvin and Ma2024; Liu, Zhang, & Zhang, Reference Liu, Zhang and Zhang2024); and (c) external influences, such as parental support (Soyoof et al., Reference Soyoof, Reynolds, Neumann and Vazquez-Calvo2025). On the outcome side, participation in IDLE has been linked to a wide range of learning benefits. These include affective outcomes, such as reduced anxiety and boredom (Rezai et al., Reference Rezai, Ashkani and Moradian2025; Taherian et al., Reference Taherian, Shirvan, Yazdanmehr, Kruk and Pawlak2024) and increased enjoyment and motivation (Fu, Reference Fu2025; Rezai et al., Reference Rezai, Ashkani and Moradian2025), as well as psychological gains like improved well-being and a greater willingness to communicate (Fu, Reference Fu2025; Lee & Chiu, Reference Lee and Chiu2024; Taherian et al., Reference Taherian, Shirvan, Yazdanmehr, Kruk and Pawlak2024). Cognitive benefits include enhanced self-efficacy (Fang et al., Reference Fang, Meng, Tang and Cui2025), while linguistic improvements encompass richer vocabulary and more fluent speaking skills (Rezai et al., Reference Rezai, Soyoof and Reynolds2026; Tam & Reynolds, Reference Tam and Reynolds2023). Cultural benefits have also been noted, including greater intercultural competence (Liu, Ma, et al., Reference Liu, Ma, Bao and Liu2025). Finally, pedagogical outcomes, such as enhanced digital competence and technological pedagogical content knowledge, further demonstrate the transformative potential of IDLE (Rezai et al., Reference Rezai, Soyoof and Reynolds2024b). However, findings on IDLE remain inconsistent across studies, often because of differences in sampling, measurement – especially the use of frequency-only IDLE scales – and the predominance of cross-sectional designs. A methodological review is therefore needed to examine the design, measurement, and analytic choices underlying these claims.
As research on IDLE has grown rapidly, several comprehensive reviews have attempted to synthesize the growing body of work. Among the earliest, Zhang et al. (Reference Zhang, Zou, Cheng, Xie, Wang and Au2021) conducted a scoping review of extramural language learning. Their review demonstrated the positive influence of extramural language learning on both linguistic proficiency and affective outcomes, while highlighting the mediating variables such as gender and target language proficiency. However, the study’s scope was limited by its narrow keyword search, which produced a relatively small dataset (n = 33). Moreover, by incorporating both offline (non-digital) and online (internet-based) activities across multiple languages other than English (LOTE; e.g., Chinese), the review’s breadth diluted its capacity to capture the unique characteristics of IDLE.
Building on this work, Kusyk et al. (Reference Kusyk, Arndt, Schwarz, Yibokou, Dressman, Sockett and Toffoli2025) conducted a more methodologically rigorous scoping review of 206 papers published between 2000 and 2020, aiming to identify key trends in ISLL. Using targeted search terms such as “outside the classroom,” “extracurricular,” and “digital wild,” they observed a growth of ISLL research, especially in Europe and East Asia. Most studies employed cross-sectional designs, relying heavily on questionnaires and interviews. While the review confirmed a positive link between ISLL engagement and L2 proficiency, it noted that causal relationships remained unclear. Another important insight was the field’s terminological diversity and frequent definitional inconsistencies – what Kelley (Reference Kelley1927) called the “jingle-jangle” problem – which make cross-study comparisons more difficult. For instance, some studies use the terms IDLE and EE interchangeably, even though EE is a broader construct that also includes non-digital activities. This lack of clear operationalization creates a methodological challenge, as it becomes unclear whether findings are specific to technology-mediated learning or apply more broadly to informal learning. Although the review was broad in scope and analytically rigorous, its inclusion of both offline and online learning, as well as LOTE, reduced its focus on IDLE specifically. In addition, because the review only covered studies up to 2020, it did not capture more recent developments associated with the rise of GenAI, including tools such as ChatGPT, which emerged in 2022 (Zou et al., Reference Zou, Reinders and Amjad2026).
A more focused review was provided by Soyoof et al. (Reference Soyoof, Reynolds, Vazquez-Calvo and McLay2023), who examined 30 IDLE studies between 1980 and 2019. Their analysis identified three main research themes: (a) linguistic outcomes, (b) affective and cultural dimensions, and (c) digital literacy and agency. The review also showed that most studies were conducted in Europe and Asia, focused on young adult learners, and mainly used qualitative or mixed-methods designs. While the review offered a useful thematic synthesis (Chong & Reinders, Reference Chong and Reinders2021), its contribution was limited by the absence of a formal review protocol, which reduces the reproducibility and systematicity of knowledge synthesis in IDLE.
Liu, Soyoof, et al. (Reference Soyoof, Reynolds, Neumann and Vazquez-Calvo2025) sought to address these gaps by conducting a thematic review of IDLE research published between 2014 and 2024, with a focus on Asian EFL contexts where exposure to English remains relatively limited. They identified three broad themes: (a) IDLE as an individual difference variable shaping English development; (b) teachers’ perceptions of, and readiness to integrate, IDLE into instruction; and (c) the role of technology access and digital literacy in contributing to unequal participation. While the review offered useful insights, it – like Soyoof et al. (Reference Soyoof, Reynolds, Vazquez-Calvo and McLay2023) – did not follow a formal review protocol. In addition, its focus on Asia meant that perspectives from other important regions, such as Europe, were not included, which limits its global scope.
Addressing some of these limitations, Guo and Lee (Reference Guo and Lee2023) conducted a systematic review of IDLE based on Bronfenbrenner’s (Reference Bronfenbrenner1979) ecological systems theory. Their analysis showed that most IDLE research has focused on individual and microsystem factors, while broader contextual dimensions, such as the mesosystem, exosystem, macrosystem, and chronosystem, have received less attention. The authors therefore called for a more holistic ecological approach to IDLE research, highlighting the importance of examining how parental support, institutional settings, and cultural contexts shape learner engagement. They also encouraged the use of more varied methodologies and cross-cultural comparisons to better understand the role of context in IDLE. Although the study was systematic and theoretically grounded, its main focus was on influencing factors rather than methodological issues. As a result, it did not directly address questions of research design quality, instrument validity, or analytical rigor that affect the strength of the field’s cumulative evidence.
Taken together, these reviews show that IDLE research has grown and become more diverse in scope. At the same time, they point to a continuing gap: there is still no systematic account of the field’s methodological practices. Most previous reviews focus on themes, contexts, or influencing factors, while paying less attention to how evidence is actually produced. This matters because choices about research design and measurement, such as the heavy reliance on self-report and cross-sectional data, shape how confidently the field can speak about causality, underlying mechanisms, and ecological validity. Without a clearer picture of current practices, repeated calls for greater rigor and transparency remain difficult to put into practice (Chong et al., Reference Chong, Lin and Chen2022). A methodological review is therefore needed to map current approaches, identify key methodological bottlenecks, and support more robust and replicable future research.
To respond to this need, the present study uses the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) framework to conduct a methodological review of IDLE research published between 2015 and 2025. We recognize that PRISMA was originally developed for health-care intervention reviews and may not fully accommodate a heterogeneous and conceptually diverse field such as IDLE. Its structured inclusion and exclusion procedures may also leave out some nuanced work. Even so, we adopt its core principles for three main reasons (Page et al., Reference Page, McKenzie, Bossuyt, Boutron, Hoffmann, Mulrow, Shamseer, Tetzlaff, Akl, Brennan, Chou, Glanville, Grimshaw, Hróbjartsson, Lalu, Li, Loder, Mayo-Wilson, McDonald and Moher2021): first, to support a transparent and rigorous search and selection process; second, to provide a replicable audit trail that helps reduce selection bias in a fragmented body of literature; and third, to offer a common structure that improves the clarity and usefulness of the review. More broadly, this study aims to map the methodological landscape of IDLE research, identify prevailing research practices, and offer practical recommendations for improving rigor, innovation, and methodological transparency in future studies. Specifically, it addresses the following research questions:
RQ1. What research designs are most commonly used in IDLE research?
RQ2. What research instruments are most frequently employed in IDLE research?
RQ3. What data analysis methods are most frequently used in IDLE research?
2. Methodology
2.1 Data sources and search strategies
We searched Web of Science and Scopus for IDLE-related studies published between January 1, 2015, and February 6, 2025, with the search conducted in early February 2025. Given the field’s persistent “jingle-jangle” issue – where overlapping constructs are used interchangeably (Kelley, Reference Kelley1927; Kusyk et al., Reference Kusyk, Arndt, Schwarz, Yibokou, Dressman, Sockett and Toffoli2025) – we intentionally adopted a comprehensive and inclusive search strategy. Our keyword list encompassed a wide range of related terms, including “language learning beyond the classroom,” “LBC,” “informal language learning,” “autonomous language learning,” “self-directed language learning,” “extramural English,” “EE,” “naturalistic CALL,” “CALL in the digital wild,” “Language learning in digital wild,” “informal language learning with technology,” “online informal language learning,” “informal digital language learning,” “online informal learning of English,” “OILE,” “informal digital learning of English,” and “IDLE.”
2.2 Inclusion and exclusion criteria
The initial database search identified 1,412 articles, which was reduced to 1,079 after duplicates were removed. To maintain focus and relevance, we included studies that (a) explicitly examined informal digital English learning, (b) reported original empirical data, (c) were written in English, and (d) were published in journals indexed in the Arts & Humanities Citation Index (A&HCI) or Social Sciences Citation Index (SSCI). For example, studies referring to “informal language learning with technology” were excluded if they focused on LOTE, such as French. By contrast, studies framed as “extramural English” were included when they involved a clear digital component, even if they also considered offline activities (Guo & Lee, Reference Guo and Lee2023). We also excluded non-empirical publications, such as reviews, conceptual papers, and commentaries. The corpus was limited to A&HCI/SSCI-indexed journals to support transparency and comparability, as these indexes apply relatively consistent selection and peer-review standards and thus provide a more stable and auditable sampling frame for a methodological review (see Figure 1).
PRISMA flow diagram for data extraction.
2.3 Operationalizing the “jingle-jangle” problem in screening
Because terms such as IDLE, EE, and learning beyond the classroom are used inconsistently in the literature, we did not base inclusion solely on titles or keywords. Given the scope of this methodological review, we also do not attempt to resolve broader definitional debates, such as those concerning intentionality, frequency or duration thresholds, or passive versus active engagement, which have been discussed elsewhere (e.g., Arndt & Kusyk, Reference Arndt and Kusyk2026).
Instead, during abstract and full-text screening, we applied a content-based criterion aligned with our working definition. Studies were considered eligible if they examined informal English learning that involved a clearly identifiable digital or technology-mediated component taking place primarily outside the classroom, regardless of whether the authors referred to it as IDLE or EE. This meant that (a) studies on EE were included if they examined digital activities such as online gaming, social media, streaming, online chatting, app-based practice, or, where relevant, AI-mediated interaction, even when offline activities were also reported; (b) studies focusing only on offline exposure were excluded; and (c) studies in which digital activity was embedded within formal instruction, such as required or assessed platform use, were excluded as formal digital learning. Borderline cases were discussed among the authors and judged using a practical criterion: whether the study provided enough information about the digital component through its measures and/or findings to allow meaningful extraction and coding of its design, instruments, and analysis.
Applying these criteria, we excluded 831 irrelevant or non-English records at the abstract-screening stage. A full-text review of 242 papers led to further exclusions, including studies on LOTE, formal-only studies, and studies without a digital component, as well as non-empirical publications and papers published outside A&HCI/SSCI-indexed journals. The final corpus consisted of 104 studies for in-depth analysis (see Appendix A in the supplementary material).
2.4 Data extraction and analysis
Each study was coded across three main categories: design (quantitative, qualitative, or mixed methods, including subtypes), instruments (e.g., questionnaire, test, interview, diary/artifact, observation), and analysis (e.g., correlation, regression, structural equation modeling [SEM], thematic analysis, content analysis, discourse analysis). For studies using hybrid designs, we applied a clear coding rule: we identified the overall design logic (e.g., explanatory sequential or convergent) while also coding each instrument and analytic technique at the level at which it was reported (see Appendix B in the supplementary material). For example, a study that combined a survey with learner diaries, and used the diary data to help explain statistical results, was coded as explanatory sequential mixed methods. In that case, “questionnaire” and “diary/personal artifact” were coded as instruments, and the quantitative and qualitative analyses were coded separately.
Using the finalized scheme, the first author coded all included studies. To assess inter-coder reliability, the second author independently coded a randomly selected 10% of the dataset. Agreement between the two coders was high, with a Cohen’s kappa of 0.83 (McHugh, Reference McHugh2012). The full set of coded data was then reviewed and confirmed by a third author with expertise in IDLE and CALL. Finally, we calculated the frequencies of the different methodological approaches. This quantitative synthesis helped identify dominant trends and emerging patterns, providing an overview of methodological practices in IDLE research.
3. Findings
3.1 Research designs
The overwhelming majority of IDLE research relies on non-experimental designs, representing 94.23% (98 out of 104) of the studies reviewed (see Figure 2). Experimental research remains scarce, with only six studies – three randomized and three quasi-experimental – attempting causal investigation.
Research design types.
Among the 98 non-experimental studies, three primary approaches emerged (see Table 1): (a) quantitative designs, used in nearly half the studies (n = 48; 48.98%); (b) mixed-methods designs, combining qualitative and quantitative inquiry (n = 32; 32.65%); and (c) qualitative designs, employed in a smaller portion (n = 18; 18.37%).
Categorization of non-experimental research designs
Table 1. Long description
A table categorizing non-experimental research designs by frequency. The table has three main categories: Quantitative, Mixed methods, and Qualitative. Quantitative designs include Correlational design (39), Causal-comparative design (5), Repeated-measures design (2), and Longitudinal design (2). Mixed methods include Explanatory sequential mixed methods (22), Convergent mixed methods (8), and Exploratory sequential mixed methods (2). Qualitative designs include Case study (9), Descriptive method (6), Ethnography (2), and Phenomenology (1).
3.1.1 Quantitative designs
Within the quantitative research, four major subtypes were identified. First, correlational studies dominated (n = 39; 39.8%), exploring relationships between IDLE engagement and outcomes like vocabulary growth and intercultural competence (Liu, Ma, et al., Reference Liu, Ma, Bao and Liu2025; Rezai et al., Reference Rezai, Soyoof and Reynolds2026) or examining predictors of IDLE participation, such as the “ideal L2 self” or “enjoyment” (Liu, Darvin, & Ma, Reference Liu, Darvin and Ma2024). Second, causal-comparative designs (n = 5; 5.1%) examined whether differing IDLE experiences led to distinct language outcomes. For example, Calafato and Clausen (Reference Calafato and Clausen2025) examined digital gaming among Norwegian students and found that language references, inferencing, and note-taking behaviors correlated with productive vocabulary, while only inferencing predicted receptive vocabulary. Third, repeated-measures designs (n = 2; 2.04%) traced the same learners over time to monitor fluctuations in IDLE activity. For example, Lee and Chiu (Reference Lee and Chiu2024) collected experience-sampling data five times per day over two weeks, revealing that more frequent productive IDLE activities was associated with higher students’ willingness to communicate. Lastly, longitudinal designs (n = 2; 2.04%) were rare. Cadierno et al. (Reference Cadierno, Hansen, Lauridsen, Eskildsen, Fenyvesi and Hannibal Jensen2022) followed Danish children for a year and demonstrated that sustained exposure to English beyond the classroom was associated with gains in receptive language skills.
3.1.2 Mixed-methods designs
Explanatory sequential, convergent, and exploratory sequential designs differed not only in timing but also in how the two strands were brought together. In explanatory sequential studies (n = 22; 22.45%), quantitative findings typically informed the qualitative phase. For example, statistical patterns were used to shape interview protocols and/or participant selection, and the qualitative findings were then used to help explain or add context to those results (e.g., Gao et al., Reference Gao, Wang and Reynolds2025).
In convergent studies (n = 8; 8.16%), the two datasets were brought together during interpretation through direct comparison or triangulation. For instance, app-use statistics were considered alongside learner reflections to support or refine the same claims (e.g., Rosell-Aguilar, Reference Rosell-Aguilar2018). In exploratory sequential studies (n = 2; 2.04%), the qualitative phase came first and informed the later quantitative stage, such as when themes from diaries or interviews were used to develop constructs, items, or hypotheses for subsequent testing (e.g., Lai et al., Reference Lai, Liu, Hu, Benson and Lyu2022).
3.1.3 Qualitative research
Qualitative studies in the corpus fell into four broad approaches. Case studies were the most common (n = 9; 9.18%). These were coded when the research focused on a clearly bounded case, such as a learner, family, class, program, or community, and examined it in depth, often using multiple sources to build a context-sensitive understanding. For example, Soyoof et al. (Reference Soyoof, Reynolds, Neumann and Vazquez-Calvo2025) examined Iranian families to explore how cultural and familial dynamics shape maternal scaffolding of children’s IDLE engagement.
Qualitative descriptive studies (“descriptive method”; n = 6; 6.12%) were coded when the study aimed to provide a relatively close, low-inference account of learners’ IDLE experiences in their everyday terms, without being organized around a bounded case. In our coding, this emphasis on staying close to participants’ accounts was used as an indicator of analytic orientation or level of abstraction, rather than as a judgment of theoretical sophistication or study quality. For instance, Lee (Reference Lee2023) described how learners used emerging social media platforms for informal language learning, focusing on reported experiences rather than a case-based structure.
Less commonly, ethnography appeared in two studies (2.04%), with attention to community practices and interactional norms (e.g., Vazquez-Calvo, Reference Vazquez-Calvo2021), while phenomenology appeared in one study (1.02%), focusing on learners’ lived experiences with GenAI tools such as ChatGPT (Li et al., Reference Li, Bonk, Wang and Kou2024).
3.2 Research instruments
As shown in Figure 3, among the 193 research tools identified, quantitative instruments dominated, appearing in 118 cases (61.14%), while qualitative tools were used less often, in 75 instances (38.86%).
Research instrument types.
3.2.1 Quantitative instruments
Of the 118 quantitative tools, self-report measures were by far the most prevalent, used in 83 instances (70.34%). Objective measures followed with 33 occurrences (27.97%). Only two studies (1.69%) employed computer tracking (see Table 2).
Categorization of quantitative instruments
Table 2. Long description
A table categorizing quantitative instruments by frequency. The table has three columns: Category, Subcategory, and Frequency. The Category column lists three categories: Self-report, Objective measure, and Computer tracking. The Subcategory column lists specific types within each category. The Frequency column lists the number of occurrences and percentage for each subcategory. Self-report includes Questionnaire with 82 occurrences (69.49%) and Self-assessment with 1 occurrence (0.85%). Objective measure includes Researcher-constructed test with 27 occurrences (22.88%), Standardized test with 4 occurrences (3.39%), and Course-based test with 2 occurrences (1.69%). Computer tracking has 2 occurrences (1.69%).
Within the subcategories, questionnaires stood out as the most frequently used tool (n = 82; 69.49%) to collect three main types of data: demographic information, IDLE behaviors, and variables related to antecedents or outcomes of IDLE. Only one study utilized self-assessment, in which participants rated their English proficiency based on the Common European Framework of Reference for Languages (Jurkovič, Reference Jurkovič2019).
Researcher-constructed tests (n = 27; 22.88%) were typically tailored to specific contexts and designed to evaluate English skills (e.g., Tam & Reynolds, Reference Tam and Reynolds2023). Standardized tests, like the English section of a nationwide entrance exam (Ghorbani & Golparvar, Reference Ghorbani and Golparvar2020), were less common (n = 4; 3.39%). Course-based assessments, including university English exams measuring learning outcomes (Ghorbani & Golparvar, Reference Ghorbani and Golparvar2020), were rare (n = 2; 1.69%).
A small number of studies employed computer tracking (n = 2; 1.69%), leveraging digital systems to trace learners’ online English activities. For example, Zhao et al. (Reference Zhao, Zhuang, Xie and Yu2024) employed a web platform to capture participants’ task scores and behavioral metrics like revision frequency. Such indicators can provide useful evidence of interaction within a digital environment, although they should not be interpreted on their own as direct evidence of language development.
3.2.2 Qualitative instruments
Among the 75 qualitative tools, interviews were the most widely used (n = 46; 61.33%). Artifact analysis followed with 15 instances (20%), while open-ended questions appeared in nine studies (12%). Observation was the least common, used in only five studies (6.67%; see Table 3).
Categorization of qualitative instruments
Table 3. Long description
A table categorizing qualitative instruments by frequency. The table has four rows and three columns. Column headers are Category, Subcategory, and Frequency. Row labels include Interview, Artifact, Open-ended question, and Observation. The Interview category has subcategories Individual interview, Stimulated-recall interview, and Focus group interview with frequencies 40, 5, and 1 respectively. The Artifact category has subcategories Personal artifact, Digital artifact, and Organizational artifact with frequencies 9, 4, and 2 respectively. The Open-ended question category has a frequency of 9. The Observation category has a frequency of 5.
Within the interview category, individual interviews predominated (n = 40; 53.33%), serving as the primary means of gathering in-depth insights into learners’ IDLE experiences and perspectives. For example, Tajeddin et al. (Reference Tajeddin, Hassani and Bakhshandeh2024) used individual interviews to explore how various out-of-class online activities supported English learning among Iranian adolescents. Stimulated-recall interviews (n = 5; 6.67%) engaged participants with visual prompts to recall past experiences (Gass & Mackey, Reference Gass and Mackey2000). Soyoof et al. (Reference Soyoof, Reynolds, Neumann and Vazquez-Calvo2025) employed videos to stimulate children’s perceptions of past IDLE activities. Only one study (1.33%) employed focus group interview. Jensen (Reference Jensen2019) gathered students who engaged in informal learning with friends to research peer-based language learning.
Personal artifacts were employed in nine studies (12%), such as diaries (Lai et al., Reference Lai, Liu, Hu, Benson and Lyu2022), journal entries (Lee & Roger, Reference Lee and Roger2023), or demographic profiles (Soyoof et al., Reference Soyoof, Reynolds, Neumann and Vazquez-Calvo2025). Digital artifacts appeared in four studies (5.33%), involving social media content such as posts, comments, or messages (e.g., Lyu & Lai, Reference Lyu and Lai2024). Organizational artifacts, which refer to documents created by or for institutions such as teacher records (Brevik & Holm, Reference Brevik and Holm2023), were the least frequent (n = 2; 2.67%).
Open-ended questions (n = 9; 12%) were used to elicit participants’ narrative responses. For instance, Dizon (Reference Dizon2026) employed written open-ended responses to explore learners’ experiences and attitudes toward using ChatGPT for English learning, thereby treating AI use as one form of IDLE experience rather than as evidence of learning in itself. Lastly, observation appeared in only five studies (6.67%). Observations in these studies were conducted indirectly through video recordings or screenshots, enabling detailed post hoc analysis while minimizing disruption (Brevik & Holm, Reference Brevik and Holm2023).
3.3 Data analysis strategies
Across the 285 data analysis methods identified, quantitative methods overwhelmingly dominated (n = 225; 78.95%), while qualitative methods made up a smaller portion (n = 60; 21.05%; see Figure 4).
Data analysis strategies.
3.3.1 Quantitative data analysis
Five main quantitative techniques emerged (see Table 4). Correlational analysis was the most frequently used (n = 94; 41.78%). Next came descriptive statistics (n = 87; 38.67%), followed by group comparison (n = 36; 16%). By contrast, factor analysis (n = 6; 2.67%) and cluster analysis (n = 2; 0.89%) appeared only occasionally, suggesting their specific research use.
Categorization of quantitative data analysis strategies
Table 4. Long description
A table categorizing quantitative data analysis strategies by frequency. The table has 4 columns and 13 rows. Column headers are Category, Subcategory, and Frequency (N = 225). Row labels include Correlation, Descriptive statistics, Group comparison, Factor analysis, and Cluster analysis. Correlation has subcategories: Correlation analysis, Regression analysis, and Structural equation modeling. Descriptive statistics has no subcategories. Group comparison has subcategories: ANOVA, t tests, Mann-Whitney U test, Kruskal-Wallis test, and MANOVA. Factor analysis and Cluster analysis have no subcategories. Frequencies are provided for each category and subcategory.
Descriptive statistics offered a snapshot of central tendencies and variability, serving as the groundwork for more complex analyses. Correlation analysis (n = 28; 12.44%) explored the strength and direction of relationships between IDLE and related constructs. For example, Rezai et al. (Reference Rezai, Soyoof and Reynolds2026) found a strong positive correlation between IDLE and vocabulary knowledge among Iranian EFL learners. Regression analysis (n = 28; 12.44%) helped reveal directional effects. Tsang (Reference Tsang2026) demonstrated that English exposure beyond the classroom could significantly predict children’s English proficiency in Hong Kong. SEM, used in 38 studies (16.89%), allowed researchers to examine complex relationships and latent variables. For example, by modeling the complex relation between IDLE and online motivation for learning (OML), online foreign language enjoyment (OFLE), and telecollaborative foreign language anxiety (TFLA), Rezai et al. (Reference Rezai, Ashkani and Moradian2025) found that IDLE is significantly associated with OML, OFLE, and TFLA.
Group comparison techniques, such as ANOVA (n = 15; 6.67%) and t tests (n = 11; 4.89%), were frequently applied to compare learners with and without IDLE experiences (e.g., Guan et al., Reference Guan, Zhang and Gu2025) or among learners with differing IDLE exposure (e.g., Tsang & Lam, Reference Tsang and Lam2025). When normality assumptions were violated, researchers used nonparametric tests like the Mann–Whitney U (n = 5; 2.22%) and Kruskal–Wallis (n = 2; 0.89%). MANOVA (n = 3; 1.33%) enabled simultaneous comparison of multiple dependent variables, as in Leona et al. (Reference Leona, van Koert, van der Molen, Rispens, Tijms and Snellings2021), who compared motivation, vocabulary, and English exposure between learners with and without formal instruction.
Factor analysis (n = 6; 2.67%) was used to uncover underlying structures influencing observed variables, such as identifying factors of EE engagement types (e.g., gaming and social interaction) in Sundqvist and Uztosun (Reference Sundqvist and Uztosun2024). Cluster analysis (n = 2; 0.89%) classified learners into typologies, as in Lee and Xie (Reference Lee and Xie2023), who discovered five learner profiles: minimal, maximal, gaming-entertainment, entertainment, and entertainment-socializing.
3.3.2 Qualitative data analysis strategies
Four main approaches characterized qualitative data analysis (see Table 5). Thematic analysis was the most common (n = 25; 41.67%) and was used to identify recurring patterns in learners’ experiences (Braun & Clarke, Reference Braun and Clarke2006). For example, Liu, Darvin, and Ma (Reference Liu, Darvin and Ma2025) examined learners’ use of AI in IDLE and identified themes such as a greater willingness to use AI as a conversational partner. These findings illustrate learners’ perceptions and interactional practices with AI, rather than demonstrating language development directly. Content analysis (n = 11; 18.33%) was also used to organize data into categories and identify patterns (Krippendorff, Reference Krippendorff2019). For instance, Rezai et al. (Reference Rezai, Soyoof and Reynolds2024a) grouped Iranian teachers’ perspectives into ecological layers – personal, micro-, macro-, and chrono-systems – that shaped IDLE engagement. Linguistic analysis (n = 2; 3.33%) focused on structural and lexical features of language. Soyoof et al. (Reference Soyoof, Reynolds, Neumann and Vazquez-Calvo2025), for example, examined sentence complexity and content words to explore how IDLE relates to children’s language development. Discourse analysis (n = 2; 3.33%) was used to examine meaning-making through language and multimodal interaction. For example, Lee et al. (Reference Lee, Hampel and Kukulska-Hulme2019) used multimodal discourse analysis to study how learners employed technology and gestures to negotiate meaning in informal communication.
Categorization of qualitative data analysis strategies
Table 5. Long description
A table categorizing qualitative data analysis strategies and their frequencies. The table has 6 columns and 10 rows. Column headers are Category, Subcategory, and Frequency (N = 60). Row 1: Thematic analysis (n = 25; 41.67 percent), no subcategory, Frequency 25. Row 2: Content analysis (n = 11; 18.33 percent), no subcategory, Frequency 11. Row 3: Linguistic analysis (n = 2; 3.33 percent), Syntactic-level, Frequency 1. Row 4: Linguistic analysis (n = 2; 3.33 percent), Lexical-level, Frequency 1. Row 5: Discourse analysis (n = 2; 3.33 percent), no subcategory, Frequency 2. Row 6: Others (n = 20; 33.33 percent), no subcategory, Frequency 20.
A further point to note is that one-third of the studies were grouped into the “Others” category because they did not clearly specify a recognizable qualitative analytic approach (n = 20; 33.33%). In our corpus, these less clearly reported cases appeared more often in earlier publications (pre-2019) and were especially common in explanatory sequential mixed-methods studies, where interview or open-ended data were added after an initial survey phase. For example, one anonymized mixed-methods study noted that interview data were “transcribed, coded, and synthesized” to supplement the quantitative findings, but did not describe the analytic framework or key procedures, such as whether coding was inductive or deductive, how codes were developed, or how themes or categories were generated. We return to this pattern and its implications in the limitations and recommendations sections.
4. Discussion
The findings of this methodological review suggest a clear methodological tendency in IDLE research. The strong predominance of non-experimental designs (94.23%) shows that the field has been shaped mainly by exploratory and correlational approaches (Lee, Reference Lee, Reinders, Lai and Sundqvist2022a; Sundqvist, Reference Sundqvist2024). This pattern is understandable, given the nature of the phenomenon itself. Informal language learning is often nonlinear, dynamic, and highly individualized, taking place in unstructured, out-of-school settings (Lee, Reference Lee2026). Because of these features, controlled experiments or interventions can be difficult to implement. As a result, cross-sectional surveys and qualitative methods, such as interviews, have often been more suitable for mapping the phenomenon and developing an initial understanding of it (Sockett, Reference Sockett2013; Soyoof et al., Reference Soyoof, Reynolds, Neumann and Vazquez-Calvo2025).
Although these dominant approaches have been useful for identifying consistent associations between IDLE engagement and learning outcomes (Kusyk et al., Reference Kusyk, Arndt, Schwarz, Yibokou, Dressman, Sockett and Toffoli2025), their concentration also brings some limitations. With relatively few experimental or longitudinal studies, the field still has only limited capacity to make causal claims or further develop explanatory theory (Burston et al., Reference Burston, Athanasiou and Giannakou2024; Choubsaz et al., Reference Choubsaz, Jalilifar and Boulton2024; Lee, Reference Lee2026). As a result, it remains unclear whether IDLE itself contributes to development or whether it tends to occur alongside factors such as motivation and contextual advantage (Ballance, Reference Ballance2024; Lee, Reference Lee, Reinders, Lai and Sundqvist2022a).
This pattern also creates a practical challenge. Cross-sectional and qualitative designs are well suited to capturing the complexity of informal learning in natural settings (Sockett, Reference Sockett2013), but decision-makers, such as policymakers and school leaders, often need stronger forms of evidence when considering institutional investment or curricular integration (Lee, Reference Lee2026). For example, longitudinal or quasi-experimental evidence can provide a firmer basis than correlation alone for decisions about allocating curriculum time to guided out-of-class tasks, adopting school-supported digital resources including, where appropriate, AI speaking tools, or expanding a pilot beyond a single grade (Creswell & Creswell, Reference Creswell and Creswell2023; Taherian et al., Reference Taherian, Shirvan, Yazdanmehr, Kruk and Pawlak2024). In a similar way, combining self-reports with behavioral or trace data, such as platform analytics, learner-exported activity records, or carefully interpreted AI interaction logs, may help educators identify what should be monitored and supported in practice, including spacing, feedback uptake, and revision routines (Lee, Reference Lee2026; Sundqvist, Reference Sundqvist2024). However, AI logs should be understood as records of AI-mediated activity co-produced by learners and systems, not as transparent records of learner language development alone. Clearer reporting of sampling and subgroup differences may also help inform more equity-oriented policies, such as device or data subsidies and differentiated support (Lee, Reference Lee2026).
Given the review’s focus on methodological patterns, the strong reliance on self-reported instruments, especially questionnaires, is understandable, although it also raises some questions about validity (Creswell & Creswell, Reference Creswell and Creswell2023; Sundqvist, Reference Sundqvist2024). Questionnaires appear to be widely used not only because they are practical but also because they align closely with the field’s most common research aims (Arndt & Rose, Reference Arndt and Rose2023). In many studies, they are used to operationalize IDLE engagement, most often through learners’ self-reported frequency or duration of specific digital activities (Arndt & Kusyk, Reference Arndt and Kusyk2026). They are also commonly used to measure antecedents or individual difference factors, such as motivation, enjoyment, ideal L2 self, and perceived usefulness, as well as outcomes, often in the form of learners’ perceived gains in areas such as vocabulary, speaking confidence, or intercultural competence rather than performance-based measures (Lee, Reference Lee2026). This pattern is consistent with an evidence base that is largely correlational and often oriented toward SEM, where survey instruments are used to examine pathways linking engagement, motivational variables, and outcomes (Liu, Ma, et al., Reference Liu, Ma, Bao and Liu2025).
At the same time, this instrument profile may help explain why our findings can sometimes seem somewhat “catalog-like.” When most studies rely on cross-sectional survey data to examine associations, the resulting evidence is naturally strongest for perceived engagement and self-reported outcomes (Arndt & Rose, Reference Arndt and Rose2023; Lee, Reference Lee2026). As questionnaires are susceptible to recall error and social desirability bias, some widely cited relationships may reflect learners’ retrospective perceptions more than their actual behavior in digital environments (Creswell & Creswell, Reference Creswell and Creswell2023). Another related issue concerns construct coverage (Arndt & Kusyk, Reference Arndt and Kusyk2026). Many IDLE scales capture how often or how long learners engage, but provide less insight into how that engagement unfolds, such as in terms of interaction, attention, or feedback uptake, even though different activity types may place quite different cognitive and linguistic demands on learners (Sundqvist, Reference Sundqvist2024). These limitations do not diminish the value of survey-based research; rather, they suggest ways in which the evidence base can be strengthened (Lai & Sundqvist, Reference Lai and Sundqvist2026; Lee, Reference Lee2026). Interviews can provide added depth (Jensen, Reference Jensen2019), and self-reported data may become more convincing when combined with additional behavioral indicators, such as platform analytics, learner-exported logs, or screen-tracking data, which are still relatively uncommon in the corpus (Arndt & Rose, Reference Arndt and Rose2023; Lee, Reference Lee2026). These indicators can help document patterns of engagement, but they vary considerably across activity types and should not be assumed to carry the same meaning in gaming, video viewing, podcast listening, online chatting, app-based practice, or AI-mediated interaction. Additionally, trace data also bring ethical and practical challenges, including issues of privacy, ownership, platform policy, interpretability, and learner consent (O’Grady et al., Reference O’Grady, Gokturk and Rossi2026). In some IDLE contexts, such as gaming, streaming, podcast listening, or online chatting, collecting meaningful and ethically defensible trace data may be difficult or even inappropriate. They may also reduce ecological validity when they depend on intrusive monitoring or highly controlled conditions (Arndt & Rose, Reference Arndt and Rose2023; Jensen, Reference Jensen2019; Lee, Reference Lee2026). For this reason, consent-based and minimally intrusive approaches, such as participant-mediated exports and aggregated indicators, may offer a more realistic balance for IDLE research (Lee, Reference Lee, Reinders, Lai and Sundqvist2022a; Sundqvist, Reference Sundqvist2024).
Also, our corpus shows meaningful variation within the broad correlational category, ranging from simple bivariate correlations to regression-based modeling and more complex SEM applications. The growing use of techniques such as SEM may indicate increasing methodological development, as these approaches can model latent constructs and examine more complex theoretical relationships (Liu, Ma, et al., Reference Liu, Ma, Bao and Liu2025). Even so, techniques such as SEM cannot by themselves overcome the limitations of cross-sectional, perception-based designs, and their contribution to causal explanation therefore remains limited (Lee, Reference Lee2026). On the qualitative side, there is also room for greater methodological transparency, as one-third of the studies did not fully specify their analytical procedures, which may affect judgments about replicability (Creswell & Creswell, Reference Creswell and Creswell2023). Taken together, these patterns add nuance to the broader point about methodological concentration: despite differences in analytic sophistication, many studies still rely on similar cross-sectional survey logics, with less attention given to mechanisms, development over time, and contextual complexity (Sundqvist, Reference Sundqvist2024).
5. Implications
This review suggests that the field still shows methodological homogeneity, with a recurring reliance on cross-sectional, correlational, self-report designs. These approaches have helped identify broad patterns, but they are less suited to explaining how IDLE works, for whom, and under what conditions. To strengthen explanatory claims, future research should better align its questions with designs that capture (a) what learners actually do in digital spaces, (b) how engagement changes over time, and (c) which mechanisms plausibly connect engagement to development (Choubsaz et al., Reference Choubsaz, Jalilifar and Boulton2024).
A practical starting point is clearer construct reporting. Future studies could specify (a) boundary conditions (what counts as IDLE), (b) intentionality assumptions (incidental exposure vs. goal-directed practice), and (c) activity structure (unstructured use vs. app-based progression or feedback). A brief “IDLE eligibility checklist” aligned with Benson’s (Reference Benson, Benson and Reinders2011) dimensions could reduce definitional drift and improve comparability across studies. Because “time” is not equivalent across engagement modes, instruments could also move beyond frequency and duration by adding a small set of engagement-quality indicators, such as interaction, language density, and feedback uptake, while separating activity type from engagement mode (passive vs. active).
To make longitudinal work more feasible in informal settings, researchers could use short micro-longitudinal or burst designs that balance ecological validity with participant burden, for example, six to eight weeks of weekly check-ins or 14 days of brief daily diaries repeated at two or three points in a semester. A minimal protocol could combine a two- to three-minute end-of-day mobile diary (activity type, minutes, perceived challenge, and one “learning moment”) with a weekly short outcome task, such as a voice note or brief writing prompt. This keeps data collection light while allowing researchers to examine within-person change and spacing patterns over time.
Learning analytics and, where appropriate, AI-related data sources can also be integrated into IDLE studies in transparent, low-burden ways. One option is participant-mediated analytics, in which learners periodically export and donate selected traces they control rather than installing invasive trackers. For example, participants could share weekly exports of AI-chat transcripts or revision histories, simple dashboard indicators such as screen-time breakdowns, or, in AI-mediated IDLE contexts, carefully contextualized chat transcripts, together with brief periodic performance checks such as a monthly speaking sample. Importantly, AI transcripts and usage metrics should be treated as records of AI-mediated interaction, not as direct evidence that learners have acquired new language knowledge or communicative ability. AI-based analytics may then support theory-driven process coding, such as identifying feedback moves, uptake, and strategy use, but such coding should account for the fact that AI systems actively shape interaction through topic suggestions, feedback style, simplification, and response design. These data should therefore be interpreted alongside other evidence, including learner reflections, task performance, interactional context, and longitudinal change.
From an applied perspective, more rigorous designs can support clearer educational decisions. For educators, distinguishing “time on English” from quality of engagement can inform task design by emphasizing interaction, feedback uptake, and retrieval practice rather than passive exposure. For policymakers and program leaders, evidence combining trace-informed engagement indicators with repeated outcome measures can better justify investment in digital resources, including when and how AI-supported practice may contribute to measurable gains over a term, for which learners, and under what conditions, while clarifying what data can be collected ethically and with minimal intrusion. Finally, frameworks attuned to nonlinear development, such as complex dynamic systems theory, may help interpret variability and context sensitivity in these datasets and strengthen theory building in IDLE (Larsen-Freeman, Reference Larsen-Freeman, Hiver and Al-Hoorie2019).
6. Limitations
This review has several limitations. First, our inclusion criteria may have introduced sampling bias. By focusing on studies published in A&HCI/SSCI-indexed, English-language journals, we likely excluded relevant work published in regional or emerging venues, other languages, or book chapters. This may have overrepresented methodological norms in highly visible international outlets while underrepresenting localized or resource-sensitive innovations, especially in Global South contexts (Guo & Lee, Reference Guo and Lee2023). As a result, the methodological convergence reported here should be interpreted mainly as applying to this indexed corpus rather than to the full landscape of IDLE scholarship. Future reviews could broaden sampling frames and code contextual variables such as region, educational level, and age group. We did not code region or age because these details were often reported inconsistently or only implicitly, and doing so would have required inference that could reduce comparability and inter-coder reliability.
Second, definitional and operational variation in what counts as “IDLE” limits comparability across studies and the construct validity of instruments that assume equivalent “exposure” across activities. Although we applied an explicit content-based screening rule (Section 2.3), our coding still depended on how clearly primary studies reported boundaries, intentionality, and activity context.
Third, incomplete reporting in some studies limited our ability to resolve disagreements about intentionality (incidental vs. goal-directed) and borderline activity types such as structured app-based learning. Relatedly, because many questionnaires operationalize IDLE through frequency or duration, our review can show how often such measures are used, but not whether “time on task” is comparable across very different engagement modes, nor could we independently assess the ecological realism of all questionnaire items. This limitation is especially relevant to AI-mediated IDLE, where available logs, prompts, responses, and usage metrics may reflect both learner activity and system behavior, rather than learner development alone.
Fourth, although we used a systematic coding framework, categorizing complex or hybrid designs still required interpretive judgment (Creswell & Creswell, Reference Creswell and Creswell2023), and our categories may not fully reflect each author’s original methodological intent. Finally, although PRISMA provided a disciplined audit trail, its use required adaptation to the field’s terminological diversity and boundary judgments (Kusyk et al., Reference Kusyk, Arndt, Schwarz, Yibokou, Dressman, Sockett and Toffoli2025). Because the review covers literature up to early 2025, it may also miss the most recent methodological developments, including emerging debates about how GenAI data should be collected, interpreted, and ethically used in IDLE research (Zou et al., Reference Zou, Reinders and Amjad2026).
7. Conclusion
This methodological review has examined the research designs, instruments, and analytical strategies that characterize contemporary IDLE scholarship. Our findings suggest that the methodological trajectory of IDLE research reflects both its empirical strengths and areas for further development. The field shows a degree of methodological homogeneity, often placing more emphasis on correlation than causation, perception than behavior, and quantification than interpretation (Chong et al., Reference Chong, Lin and Chen2022; Lee, Reference Lee, Reinders, Lai and Sundqvist2022a). Although this pattern has been helpful in identifying broad trends, it may also narrow the epistemological range of inquiry and limit the field’s ability to inform pedagogy in more nuanced ways (Lee, Reference Lee2022b; Sundqvist, Reference Sundqvist2024). Future research may therefore benefit from greater methodological pluralism, including the use of mixed methods, longitudinal designs, and experimental approaches, to better connect self-perceptions with behavioral evidence and statistical patterns with narrative depth (Guan et al., Reference Guan, Zhang and Gu2025; Taherian et al., Reference Taherian, Shirvan, Yazdanmehr, Kruk and Pawlak2024). Such diversification may help IDLE research move beyond descriptive mapping toward a more explanatory and theoretically integrated understanding of informal language learning. At the same time, methodological diversification should not be reduced to the adoption of AI-generated data or easily available digital metrics. Future IDLE research should continue to recognize the diversity of informal digital practices and treat AI-related evidence as one complementary, carefully interpreted source within a broader methodological toolkit.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/S0958344026100603
Data availability statement
Data are available upon request.
Authorship contribution statement
Xian Liu: Data curation, Formal analysis, Visualization, Writing – original draft; XinWei Pan: Data curation, Formal analysis; Ju Seong Lee: Conceptualization, Methodology, Writing – original draft, Writing – review & editing.
Funding disclosure statement
This research did not receive any specific funding.
Competing interests statement
The authors declare no competing interests.
Ethical statement
Ethical approval was not required.
GenAI use disclosure statement
The authors used AI (https://www.deepseek.com) to improve readability. The authors take full responsibility for the content of the publication.
About the authors
Xian Liu is a doctoral student in the Department of English Language Education at the Education University of Hong Kong. Her work focuses on informal digital learning of English (IDLE) in less-resourced regions.
XinWei Pan is a doctoral student in the Department of English Language Education at the Education University of Hong Kong. Her work focuses on informal digital learning of English (IDLE) and informal digital learning of Chinese.
Ju Seong Lee is an associate professor in the Department of English Language Education at the Education University of Hong Kong. His work focuses on informal digital learning of English (IDLE) and its social impact.
Open access
