Statistical methods in L2 motivation research

In the early days, Gardner’s socioeducational model (Gardner, Reference Gardner1985b), with its robust scientific methodology, attracted a strong following. However, criticism of its relevance to classroom-based L2 learning (e.g., Dörnyei, Reference Dörnyei1990) led researchers to seek alternatives. The second theory to dominate was Dörnyei’s L2MSS, with its main tenet of the ideal L2 self as a driver of L2 learning motivation. Despite the theoretical shift, the majority of research still continued within the same methodological paradigm—that developed by Gardner. Thus, even recent studies (e.g., Danesh & Shahnazari, Reference Danesh and Shahnazari2020; Vlaeva & Dörnyei, Reference Vlaeva and Dörnyei2021; You, Dörnyei, & Csizér, Reference You, Dörnyei and Csizér2016) continue to use self-report surveys analyzed with statistical methods. Within this wider framework, Al-Hoorie et al. focus on a “crisis of validation” of measurement instruments used in the L2MSS tradition.

Gardner’s methodology included two main strategies. First, factor analysis was used to establish parsimony and validity in attitudinal variables, and second, regression was used to explore connections with outcomes such as L2 achievement (Gardner, Reference Gardner1985b, p. 62). Thus, for example, Gardner and Lambert (Reference Gardner and Lambert1972) identified the attitudinal variable “integrativeness” using exploratory factor analysis, which then formed the basis of a test battery (the Attitude/Motivation Test Battery [AMTB]; Gardner, Reference Gardner1985a) and subsequent investigations of connections to L2 learning outcomes. As Al-Hoorie et al. describe, a central claim of the Gardnerian tradition was the rigorous approach to construct validation maintained throughout (Gardner & Tremblay, Reference Gardner and Tremblay1994). For the current authors, it is the first step of using factor analysis to test the underlying constructs that constitutes a necessary step in introducing new theories and measures in quantitative research (see also, Guildford, Reference Guilford1946; Rönkkö & Cho, Reference Rönkkö and Cho2020). It is also our view that this step appears to have been consistently and systematically omitted in the L2MSS tradition.

Issues of construct validation

The current authors feel that Al-Hoorie et al. do a remarkable job of highlighting a depth and variety of problems that should lead to a moratorium on the use of measurement instruments from the L2MMS tradition. Put simply, they detail how, over nearly 20 years, researchers in the tradition simply assumed the validity of their instruments, without subjecting them to the rigorous testing usually expected of scientific inquiry. We argue that this represents a failure of validation rather than a crisis and note our surprise that the model retains credibility among researchers.

We further contend that underlying this failure is a common misunderstanding of the importance of reliability and validity in instrument design. Thus, Dörnyei and Taguchi (Reference Dörnyei and Taguchi2010, pp. 93–96) start by defining reliability as “the extent to which scores on the instrument are free of measurement errors” and validity as “the extent to which a psychometric instrument measures what it has been designed to measure” (p. 93). They then proceed to state that, in practice, it is often not feasible to investigate reliability and validity when developing new instruments and therefore enough to rely on the “internal consistency” of scales (p. 93, emphasis in the original). Their discussion then details how this can be achieved using Cronbach’s alpha alone—conflating reliability and validity under this third term, internal consistency.

From the perspective of QRPs, two critical issues emerge. First, Cronbach’s alpha measures reliability only—through a numeric estimate of correlations among items in a scale (Yang & Green, Reference Yang and Green2011). Conversely, it has little to say about validity, that is whether the instrument measures what it is supposed to measure. Second, Cronbach’s alpha is calculated for each scale separately, meaning its use assumes unidimensionality within scales (i.e., that each measures a single unified construct) while providing no check on discriminant validity between scales. As Thompson (Reference Thompson2004, p. 4) asserts, “reliability is only a necessary—not a sufficient—condition for validity.” This stands in contrast to Dörnyei & Taguchi’s (Reference Dörnyei and Taguchi2010) argument that internal consistency measured with Cronbach’s alpha is sufficient in practice.

We notice, however, that misunderstanding of issues of reliability and validity, particularly through misuse of Cronbach’s alpha, is prevalent in the literature. The second author reflects on a personal misstep: when using a grammaticality judgment test in a dissertation in the early 2000s, she claimed the instrument was reliable based on a high Cronbach’s alpha value from a previous administration. It was only later that she recognized that reliability metrics are administration-specific and cannot be “imported” across contexts. We feel this highlights how researchers, even with the best intentions, can sometimes adopt research practices that, while seemingly sound at the time, may not fully align with the most rigorous standards. To conclude, in response to the question implied by Dörnyei and Taguchi (Reference Dörnyei and Taguchi2010), whether every instrument in a test battery should be fully tested for validity and reliability, our answer is a resounding “yes”, especially during development and early use.

Uses of factor analysis for theory validation

At this point, it is worth looking at how exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) can be used differently in psychometrics to validate theoretical constructs and their associated measures. As a reviewer pointed out, the choice between EFA and CFA often depends on the presence or absence of a reasoned theoretical model. If theory is underdeveloped EFA is the default, while the presence of established theory makes CFA preferable (Thompson, Reference Thompson2004, pp. 5–6). It should be noted, however, that this is a simplification, as either method can contribute to validation through falsification. We agree with the reviewer, however, that it is relevant to our discussion of the L2MSS tradition to look at how the two methods can be used to compare competing theoretical models.

EFA is principally used for identifying the single, most parsimonious, solution from multiple scales in a survey. By allowing the simplest reorganization of items into factors, EFA provides a “best-fit” solution based on “most variance accounted for” (Nunnally & Bernstein, Reference Nunnally and Bernstein1967). This can be used to evaluate the match of hypothesized constructs, such as ideal L2 self or integrativeness. As Gorsuch (Reference Gorsuch, Schinka and Velicer2003) notes, the lack of specific expectations makes EFA a “multitailed test” of any theory (p. 148). When used for theory validation, any mismatch between the hypothesized scales and the observed factors highlights discrepancies in both theoretical constructs and item wordings (e.g., Tavakol & Wetzel, Reference Tavakol and Wetzel2020). Meanwhile, cross-loadings of items on factors identify problems of discriminant validity—especially in cases where measurement and criterion variables are combined in the same survey instrument (Al-Hoorie et al., Reference Al-Hoorie, Hiver and In’nami2024).

The use of EFA for theory validation is, however, not without its critics. One key limitation is that it generates only a single “best-fit” solution, which then becomes the only model subject to statistical testing (Thompson & Daniel, Reference Thompson and Daniel1996). EFA thus provides no way of rejecting theorized constructs in a statistically rigorous manner, while risking problems of replication and theoretical parsimony through varying factor interpretations. More importantly, EFA does not allow comparisons of competing theoretical models against a single data set, which would facilitate a better understanding of the relevance in specific contexts. For many, these issues make EFA less suitable for comparing competing theoretical models (e.g., Thompson & Daniel, Reference Thompson and Daniel1996).

In contrast, CFA offers a more structured approach that allows comparison of models using goodness-of-fit indices, thus providing a more rigorous method for theory falsification (see, e.g., Schoonen, Reference Schoonen and Plonsky2015; Tabachnick & Fidell, Reference Tabachnick and Fidell2007). A further advantage is that CFA can form the first step of a more sophisticated structural equation modeling (SEM) analysis. In an overview, Schoonen (Reference Schoonen and Plonsky2015) describes the three main uses for SEM. The first is purely confirmatory and accepts or rejects a single model according to goodness of fit criteria. The second is theory testing through comparison of competing models on the basis of fit on a single data set. The third is model generation, where an initial model is altered to produce better correspondence to fit criteria. As Schoonen (Reference Schoonen and Plonsky2015) makes clear, it is the second strategy that is best suited to theory falsification. He further cautions that model modifications based on fit statistics alone risk turning the analysis into a semiexploratory, hybrid, procedure (also, Thompson & Daniel, Reference Thompson and Daniel1996), thus losing the principal benefit of choosing the CFA approach.

A definition of best practice in scale validation

To summarize, in the view of the current authors, theory validation involves an active attempt at disconfirmation, including checks for convergent and discriminant validity in the measurement scales (Campbell & Fisk, 1959), in the manner provided by factor analysis. Conversely, a sole reliance on reliability measures, such as Cronbach’s alpha, risks misrepresenting the validity of scales in multifactor models. Factor analysis, whether EFA or CFA, provides better validation of hypothesized constructs prior to use in further analyses such as regression. For well-established theories, SEM allows direct comparison of competing models, with the CFA ensuring the validity of the constructs and item wordings in the first (measurement model) stage. However, the use of SEM also involves certain risks. For theory verification, SEM requires a comparison of competing models on a single data set, not simple confirmation of a single model against statistical fit criteria. Indeed, Thompson and Daniel (Reference Thompson and Daniel1996) point out that the latter case tests the model against unlikely null models—positing no factors—that underly the criteria (Bentler, Reference Bentler1990, Reference Bentler and Reynolds1994). Additionally, excessive model modifications may undermine the integrity of hypothesis testing by blurring the line between exploratory and confirmatory analysis (Schoonen, Reference Schoonen and Plonsky2015).

In conclusion, we contend that the takeaway for consumers of this type of research is to check (1) that construct validity is verified through a factor analytic procedure, (2) that alternative and equally plausible models are tested, and (3) that model modifications are based on theoretical considerations, not statistical criteria alone. This, in turn, provides a template for rigorous conduct of research (RCR) in practical applications such as validation of newly proposed theory and its associated measurement instruments.

Failure of validation in the L2MSS tradition

Turning back to the L2MSS tradition, we feel the extent of the validation failure is best understood by looking at the first presentation and early attempts at validating the core construct of ideal L2 self. Most importantly, we focus on the evidence presented to show that ideal L2 self is preferable to integrativeness—a pivotal claim of the L2MSS tradition.

Dörnyei first suggested the ideal L2 self as a replacement for integrativeness in two papers (Csizér and Dörnyei, Reference Csizér and Dörnyei2005a, Reference Csizér and Dörnyei2005b) reporting a survey of schoolchildren in post–Soviet Hungary (also, Dörnyei, Csizér, & Neméth, Reference Dörnyei, Csizér and Neméth2006). Both presented findings from the survey, before moving on to develop logical arguments that integrativeness should be relabelled ideal L2 self in that context. The current authors note, however, that as neither of the studies included L2MSS-based scales, their conclusion is unsupported by the empirical evidence presented and thus purely speculative. In the vocabulary of QRPs, these findings, especially from Csizér and Dörnyei (Reference Csizér and Dörnyei2005b) and Dörnyei et al. (Reference Dörnyei, Csizér and Neméth2006), strongly resemble HARKing (Hypothesizing After Results are Known; see e.g., Kerr, Reference Kerr1998; Rubin, Reference Rubin2017), an issue we will return to below.

A further study from Hungary (Csizér & Kormos, Reference Csizér, Kormos, Dörnyei and Ushioda2009) included L2MSS scales in an SEM investigation of attitudes in secondary and university students analyzed separately. Regarding construct validity, they state, “First, measurement models were drawn up in accordance with the earlier factor analytical results reported in Csizér and Dörnyei (Reference Dörnyei2005), Dörnyei et al. (Reference Dörnyei, Csizér and Neméth2006) and Kormos and Csizér (Reference Kormos and Csizér2008) on similar data sets” (p. 103, emphasis in the original). This gives the impression that factor analysis was previously conducted on the L2MSS-derived components. Close inspection, however, shows that L2MSS-derived scales were not included in the first two studies, while the factor analysis was not presented in the third (Kormos & Csizér, Reference Kormos and Csizér2008). The reader is thus left to take their word that checks on construct validity were adequate, with the measurement models from Csizér and Kormos (Reference Csizér, Kormos, Dörnyei and Ushioda2009) also not presented or discussed. Regarding their conclusion “that [ideal L2 self] might rightfully take over the place of integrativeness” they refer the reader to Ryan (Reference Ryan, Dörnyei and Ushioda2009) and Taguchi, Magid, and Papi (Reference Taguchi, Magid, Papi, Dörnyei and Ushioda2009) for supporting evidence—rather than presenting it themselves.

These studies, Ryan (Reference Ryan, Dörnyei and Ushioda2009) and Taguchi et al. (Reference Taguchi, Magid, Papi, Dörnyei and Ushioda2009), were the first in a series, mainly conducted by Dörnyei’s graduate students, that set about validating the L2MSS framework outside Hungary. Most influentially, Ryan (Reference Ryan2008, Reference Ryan, Dörnyei and Ushioda2009) was the first to develop a scale explicitly designed to measure “ideal L2 self” in a survey of attitudes in Japanese learners of English. He combined this with a range of scales from the literature and a newly developed criterion scale of “intended learning effort.” In addressing the stated aim of “validating” the ideal L2 self-construct and comparing it to integrativeness, he relied on internal reliability in the scales—measured using Cronbach’s alpha (as also described by Dörnyei and Taguchi, Reference Dörnyei and Taguchi2010, and discussed earlier). This was followed by a correlational analysis that compared both ideal L2 self and integrativeness scales to the criterion scale “intended learning effort.” He concluded that the ideal L2 self is a better predictor of “motivated learning behavior” based on the correlations in Table 1. In particular, he points to higher correlations between ideal L2 self and intended learning effort across all groups, most especially university English majors whom he took to be the more motivated learners.

Table 1.

Correlations of integrativeness and ideal L2 self with intended learning effort (adapted from Ryan, Reference Ryan, Dörnyei and Ushioda2009, p. 133)

While Ryan (Reference Ryan2008, Reference Ryan, Dörnyei and Ushioda2009) interpreted these figures as evidence of the superiority of ideal L2 self over integrativeness, close examination reveals methodological flaws and questionable interpretations. In the first instance, the use of multiple scales from diverse sources makes the study exploratory in nature, suggesting EFA as an obvious first step in evaluating both theoretical parsimony and construct validity in the scales. This step was omitted, despite considerable precedent (e.g., Dörnyei, Reference Dörnyei1990, Reference Dörnyei1994), precluding rigorous assessment of convergent and discriminant validity in the scales. Further, the use of Cronbach’s alpha as the sole metric for validating the newly proposed ideal L2 self-scale represents a major methodological flaw. Finally, in reaching his conclusions, Ryan focused narrowly on the comparisons across rows in Table 1 (i.e., between ideal L2 self and integrativeness for each group), but neglected to consider variation within columns (i.e., between groups for ideal L2 self). For the current authors, uniformity in the correlations for ideal L2 self across obviously different groups of respondents (ranging from .71 to .75) suggests problems with the effectiveness of the measure. In contrast, integrativeness shows greater variation (.54 to .71), suggesting better sensitivity to differences between respondents. It also seems likely that the higher correlations across the board for ideal L2 self are anyway attributable to overlapping wordings between items in that scale and the criterion scale, as illustrated by Al-Hoorie et al. (p. 312).

To summarize, we find that Ryan’s (Reference Ryan, Dörnyei and Ushioda2009) study includes a string of questionable decisions, including an inappropriate choice of analysis, use of misleading metrics, and overinterpretation of results. Thus, rather than demonstrating the superiority of ideal L2 self, as claimed, we feel the study raises serious doubts about its usefulness as a motivational measure. It further highlights systemic issues of methodological rigor in the L2MSS tradition.

Subsequent studies, including Magid (Reference Magid2011), Papi (Reference Papi2010), and Taguchi (Reference Taguchi2010), used SEM to investigate the validity of the constructs implied by the L2MSS model. Summarized in Taguchi et al. (Reference Taguchi, Magid, Papi, Dörnyei and Ushioda2009), these studies adapted a survey for use in China, Iran, and Japan with the stated aim of “validating” the L2MSS in the three contexts (p. 66). Similarly to Ryan (Reference Ryan, Dörnyei and Ushioda2009) and Csizér and Kormos (Reference Csizér, Kormos, Dörnyei and Ushioda2009), they included a range of attitudinal scales—such as attitudes to learning English and integrativeness—along with new scales based on the L2MSS model, and a criterion of “intended effort.” It should be noted, however, that their use of SEM deviates from the approach recommended for theory validation by Schoonen (Reference Schoonen and Plonsky2015) of comparing competing models against the same data. Rather, similarly to Csizér and Kormos (Reference Csizér, Kormos, Dörnyei and Ushioda2009), the studies compared a single model across multiple data sets, further modifying the models separately to improve correspondence with fit criteria (Taguchi et al., Reference Taguchi, Magid, Papi, Dörnyei and Ushioda2009, p. 82; Taguchi, Reference Taguchi2010, p. 185).

Careful inspection reveals a further critical omission: as integrativeness was excluded from the SEM model, the analysis makes no meaningful comparison between ideal L2 self and integrativeness. Rather, we see this comparison followed Ryan (Reference Ryan, Dörnyei and Ushioda2009) and Csizér and Kormos (Reference Csizér, Kormos, Dörnyei and Ushioda2009), in relying on simple correlations calculated between the unverified scales and the criterion scale—with similar findings and similar problems. In conclusion, Taguchi et al. (Reference Taguchi, Magid, Papi, Dörnyei and Ushioda2009) report, “The average variance in the criterion measures explained by integrativeness is 29% while the average variance in the criterion measures explained by the ideal L2 self is 34%, which is 17% higher (sic)” (p. 78). Even ignoring the obvious numerical exaggeration, calculated by dividing the difference into the lower figure, it seems likely that the higher correlation (5% higher) is again largely an artifact of overlapping wordings in the measurement and criterion scales (Al-Hoorie et al., Reference Al-Hoorie, Hiver and In’nami2024).

Thus, despite adopting a sophisticated statistical technique, Taguchi et al. (Reference Taguchi, Magid, Papi, Dörnyei and Ushioda2009) missed the opportunity to rigorously test the L2MSS. First, by omitting integrativeness from their SEM model, they failed to compare the L2MSS against an obvious alternative framework. Second, substantial post hoc model modifications—including the removal of items and linking error terms within scales—undermine the robustness of their findings. Finally, the unusually high threshold of (.85) adopted as a criterion for identifying problems of discriminant validity between scales, far exceeds conventional benchmarks for medium (.40) or large (.60) correlations (Al-Hoorie et al., Reference Al-Hoorie, Hiver and In’nami2024, p. 316; Plonsky & Oswald, Reference Plonsky and Oswald2014). This not only risks multicollinearity but also suggests the possibility that overlapping item wordings could have exaggerated the fit of the final SEM model.

In conclusion, we feel that the studies summarized in Taguchi et al. (Reference Taguchi, Magid, Papi, Dörnyei and Ushioda2009) mirror many of the same QRPs as Ryan (Reference Ryan2008, Reference Ryan, Dörnyei and Ushioda2009), including inappropriate use of statistical methods, misleading use of metrics, and overinterpretation of results. Unfortunately, the methods employed by Ryan (Reference Ryan, Dörnyei and Ushioda2009) and Taguchi et al. (Reference Taguchi, Magid, Papi, Dörnyei and Ushioda2009) set a precedent for a great deal of subsequent research (see, e.g., Al-Hoorie, Reference Al-Hoorie2018; Hiver & Al-Hoorie, Reference Hiver and Al-Hoorie2020 for discussion) perpetuating these methodological shortcomings.

As a footnote, we feel it worthwhile to highlight a selection of QRPs from Plonsky and Larsson et al.’s (Reference Plonsky, Larsson, Sterling, Kytö, Wood, De Costa, Rabie-Ahmed and Cinaglia2024) taxonomy that relate most directly to the problems we see in the research discussed above (see Table 2). They are addressed in the order of the original list.

Table 2.

Selected taxonomy of QRPs (adapted from Plonsky and Larsson et al., Reference Plonsky, Larsson, Sterling, Kytö, Wood, De Costa, Rabie-Ahmed and Cinaglia2024).

The fatal attraction of the L2MSS model

The question remains of how the L2MSS model achieved such widespread appeal, despite such an apparent failure to establish the validity of the concepts and instruments used. As Al-Hoorie, et al. put it, “The L2 motivational self system … has, for some, become synonymous with L2 motivation and its instruments and data-elicitation scales isomorphic with learners’ actual levels of motivation” (p. 308) while Dörnyei and Ryan (Reference McClelland2015, p. 90) gleefully point to a “surge” in research output on L2 motivation after the appearance of the L2MSS, as evidenced by a ten-fold increase in related publications. Clearly, the L2MSS struck a chord with teachers and researchers alike over this period.

At this point, the first author must admit to having felt a similar enthusiasm at the time. The simplicity of the terminology, the intuitive appeal of L2 learners striving to realize their ideal L2 selves, and the promise of practical classroom interventions gave the L2MSS a strong allure. However, attempts to validate the L2MSS model in a university context in regional Japan, using EFA on the same scales used by Ryan (Reference Ryan, Dörnyei and Ushioda2009) and Taguchi et al. (Reference Taguchi, Magid, Papi, Dörnyei and Ushioda2009), failed to support their findings (McClelland, Reference McClelland2010a, Reference McClelland2010b), raising questions about the appropriacy of the model to that context (see also, McClelland, Reference McClelland2015; Al-Hoorie, McClelland, et al. Reference Al-Hoorie, McClelland, Resnik, Hiver and Botes2024). Further insight is provided by personal communications with those involved in developing the model, which suggested some ambivalence about its foundational constructs. For example, Dörnyei himself expressed skepticism toward statistical analyses more complex than basic t-tests, while questioning the usefulness of the ideal L2 self-construct in retrospect. During this period, efforts were also underway to explore alternatives such as “Imagery” and “Visual style” (Dörnyei & Chan, Reference Dörnyei and Chan2013) although subsequent critiques also identified problems with these investigations (Al-Hoorie et al., Reference Al-Hoorie, Hiver and In’nami2024, pp. 311–312). Despite the uncertainties, the L2MSS continued to gain traction, likely due to its widespread adoption and perceived utility in L2 learning research.

For the current authors, the persistence of the L2MSS as the dominant framework in the field also reflects broader systemic issues. Many researchers, including one current author (Larson-Hall & Dewey, Reference Larson-Hall, Dewey and Hansen2012), uncritically adopted an ideal L2 self-scale as a measure of motivation. It is now evident that the poor validation of the instrument undermines the reliability of those findings. Reflecting on this, the current authors advocate for a more critical approach to the selection of psychometric instruments, suggesting that mainstream educational theories—such as self-determination theory, or the four-phase model of interest—may provide more robust conceptual frameworks (see, e.g., Oga-Baldwin, Fryer, & Larson-Hall, Reference Oga-Baldwin, Fryer and Larson-Hall2019).

In our view, uncritical acceptance of the L2MSS stemmed from four primary factors. First, Dörnyei’s personal reputation lent considerable credibility to the model. Second, there was a perceived need for a unified, accessible measure of L2 motivation. Third, the model’s theoretical underpinning appealed to teachers due to its simplicity and promise of actionable insights. Finally, the complex statistical methods used in the L2MSS tradition acted as a “black box” that obscured methodological flaws from all but expert readers. This lack of critique from within the field, combined with limited engagement from researchers in related disciplines, allowed the L2MSS to dominate the discourse largely unchallenged (see Oga-Baldwin et al., Reference Oga-Baldwin, Fryer and Larson-Hall2019, for an interdisciplinary critique).

The epigraph from Lewis Carroll (Reference Carroll1865), in which gardeners paint roses red to hide a mistake, serves as a poignant metaphor for this situation. Despite internal acknowledgment of its flaws, the L2MSS was consistently presented as a robust theoretical framework, delivering significant academic prestige to its proponents. However, the risks of relying on unvalidated constructs—both in terms of the credibility of findings and the development of the field—are apparent. To prevent similar issues in the future, we urge SLA researchers to rigorously validate testing instruments in their own contexts or practice. The case of the L2MSS underscores that without validated measures, even the most appealing theoretical constructs risk undermining progress in the field.