R. Ellis (2005), in a recent thematic issue of this journal, reported a psychometric study that attempted to establish operational definitions for the theoretical constructs of explicit and implicit knowledge. R. Ellis's article was in response to a problem with instrumentation within this field, and he employed factor-analytic methods to address this problem. He argued that the lack of validated measures for these two theoretical constructs has prevented empirical tests of the interface positions between these two kinds of knowledge. I dispute the particular case that R. Ellis made for the role of instrumentation in the deficit of experimental-type studies, but I endorse his general position along with Hulstijn's (2005) view that instrumentation is a critical issue for the future. Instrumentation has been neglected and is important whether the research in question is of correlational or experimental design. I go on to argue that R. Ellis's use of factor analysis in the study is methodologically flawed and that the data should be revisited using a confirmatory factor analysis (CFA) rather than exploratory factor analysis (EFA). The problems generated by an exploratory approach to a verificational problem are explicated. To this end, three potential first-order recursive CFA models are specified and elaborated—two of them using the path-diagram conventions of AMOS 5.0.1. One model is consistent with R. Ellis's theoretical assumptions and one is a rival model. I then close the critique of R. Ellis's factor-analytic study and move on to a discussion of issues that are involved in addressing the psychometric shortfalls within this research trajectory. I use R. Ellis's study as a departure point and frame of reference for some of this discussion, but it is important to note that this part of the article is not a critique of R. Ellis's study. It is hoped that the suggestions made and issues discussed in the latter part of this article will stimulate debate on the psychometrics of explicit and implicit knowledge and help to widen the appeal for new benchmarks in best psychometric practices within the domain.

R. Ellis's (2005) study began with a brief review of the intractable debate between the mentalist and connectionist schools of thought within theories of linguistic knowledge. Citing Gregg (1989), Chomsky (1976), and N. Ellis (1996), the article then moved on to successfully argue that both schools rely on notions of implicit and explicit knowledge and that the distinction between these two types of knowledge represents a point of agreement between them. R. Ellis then returned to issues of discord by explaining the three interface positions between these two types of knowledge: the noninterface position (Hulstijn, 2002; Krashen, 1981; Paradis, 1994), the strong interface position (DeKeyser, 1998; Sharwood Smith, 1981), and the weak interface position (R. Ellis, 1993). The article argued that these positions have not been the subject of empirical inquiry and that one reason for this is “the lack of agreed instruments designed to ascertain the nature of knowledge acquired by learners: implicit, explicit, or some amalgam of the two” (R. Ellis, 2005, p. 146). This claim is not well supported because the limitations of previous studies (Green & Hecht, 1992; Hu, 2002; Macrory & Stone, 2000) presented by R. Ellis reside with design issues—as he, in fact, argued. However, the case for this lack of experimental design being rooted in the absence of agreed instruments is not made. Given that the absence of such instruments would also undermine the results of correlational studies, the case does, indeed, need to be made. Instrumentation refers to the issue of the correspondence between the constructs one is measuring and the instruments one is using to do this. Design refers to whether variables are being manipulated, or not, under controlled conditions so that implications of causality can be drawn. These are separate issues, and there is no reason why instrumentation should affect or prevent one design type more than the other. Measuring what one claims to measure is a minimal condition for any study. R. Ellis did cover one study of experimental design (DeKeyser, 1995) and argued that lack of attention to the validity of the chosen instrument precludes this study from counting as a direct test. Despite the weakness of R. Ellis's particular case for the lack of agreed instruments being the obstacle to a direct test of the interface hypotheses, his citation of Douglas' (2001) grievance concerning the general failure of the SLA research community to consider construct validity and instrumentation is apposite.¹

R. Ellis's concerns with the issue of instrumentation were echoed in Hulstijn's introduction to the same issue. Hulstijn (2005) stated:

The present article takes its cue from this citation. It endorses the view taken by Hulstijn that R. Ellis's article recognized a genuine problem within the implicit/explicit knowledge research area and signaled a response to it. However, it disputes R. Ellis's approach to the factor analysis and, therefore, disputes the findings of his study. I will now address this objection to R. Ellis's factor analysis.

After locating the study and its goals, R. Ellis proceeded to examine different ways in which explicit and implicit knowledge have been distinguished, and he arrived at a set of seven criteria. He then used these to develop his own set of seven criteria (roughly informed by the previous seven identified in past literature). These included degree of awareness, time available, focus of attention, systematicity, certainty, metalanguage, and learnability. From these criteria, he developed a set of five measures that are claimed to operationalize explicit and implicit knowledge—three measures for implicit knowledge and two for explicit knowledge. What is important to make absolutely clear at this point is that R. Ellis approached his study with an a priori model and a verificational agenda (note the word agenda is used and not design—it is the verificational credentials of the design that I dispute). This is important from a methodological point of view because it goes to the heart of this critique of R. Ellis's study. In cases in which the researcher approaches factor analysis with an a priori model that requires testing, the analysis should take the form of a CFA rather than an EFA (Byrne, 2001; Kline, 1994; Thompson, 2004; Thompson & Daniel, 1996). As Thompson and Daniel explained:

The verificational agenda of R. Ellis's study is further elaborated in the specific predictions offered with regard to which tests (operationalizations) would measure the constructs of implicit and explicit knowledge. Specific predictions are made and, in this case, a CFA is necessary.

Having made this foundational criticism, it is worth going on to examine in more detail how the execution of the EFA, in the context of a verificational agenda, and the consequent results have led to questionable interpretations and inferences. Thompson (2004) explicated the proper execution of an EFA as a linear sequence of decisions that involves the following questions:

1. Which matrix of association coefficients should be analyzed?
2. How many factors should be extracted?
3. Which method should be used to extract the factors?
4. How should the factors be rotated?
5. How should factor scores be computed if factor scores are of interest? (p. 27)

This critique raises methodological objections to points 2 and 4 of Thompson's linear sequence with regard to R. Ellis's study.

Turning first to the decision on how many factors to extract (Thompson's [2004] point 2), R. Ellis reported that:

There are two issues here. First, the theory in question is being used to make decisions on what is presented as an empirical test of the same theory. There is a built-in circularity here that undermines the scientific credibility of the study. The impression one gets from the study is that R. Ellis would argue that the two constructs, explicit and implicit knowledge, are stipulated and the loadings onto these by the five operationalizations is the subject of the test. However, this would reinforce the point that the analysis should have been a CFA, in which the expectations of which operationalizations will load on which constructs are directly hypothesized as part of the method and not only as part of the write-up of the pursuant journal article. Second, the part of the decision made on the basis of inspecting eigenvalues is inconsistent with current best practices in psychometrics. In the first run, the eigenvalue for the second factor was reported as 0.822, which is, as R. Ellis conceded, below the threshold of 1.0 (see Guttman, 1954; Kaiser, 1961). In the second run, in a theory-driven decision (citing Bialystok, 1979; Hedgcock, 1993) to replace total untimed grammaticality judgment test (GJT) scores with only the scores for ungrammatical sentences,²

an eigenvalue of 0.982 was derived on the second factor. This is just short of the threshold of 1.0, and one is inclined to grant some latitude here. However, it should be noted that in a Monte Carlo study,³ Zwick and Velicer (1986) found that the eigenvalue-greater-than-one rule severely overestimates factors to retain in almost all cases. Henson and Roberts (2006) cited four studies with consistent findings (Cattell & Jaspers, 1967; Linn, 1968; Yeomans & Golder, 1982; Zwick & Velicer, 1982), one of which was also a Monte Carlo study (Linn), and two with contrary findings (Humphreys, 1964; Mote, 1970). Overall, there is considerable weight for the position that the eigenvalue-greater-than-one rule overestimates the number of factors, particularly in view of findings from the Monte Carlo-type studies, which are methodologically compelling as indicators of bias. In this context and given the fact that the value failed to reach the threshold of 1.0 (even after post hoc transformations of the troublesome data on the untimed GJT), it is clearly arguable that only one factor emerged in the EFA.

Turning to the criteria that R. Ellis chose to use, rather than their application, current practice also emphasizes that the eigenvalue-greater-than-one rule is overused (Thompson & Daniel, 1996), primarily because it is the default option on the SPSS statistics program. Other criteria, such as inspection of a scree plot (Cattell, 1966), statistical significance tests (Bartlett, 1950), a parallel analysis (Horn, 1965), and the interpretability of obtained factors are typically advised. With regard to the criterion of the interpretability of obtained factors, R. Ellis would probably counterargue that this is a method employed in the study due to the theoretically driven expectation of two factors (explicit and implicit), but it should be noted that this criterion assumes an exploratory agenda, not a verificational agenda.⁴

With regard to statistical significance tests, these are not widely used due to their sensitivity to sample size (see Thompson, 1996). Finally, the criterion most frequently advocated (Henson, Capraro, & Capraro, 2004; Henson & Roberts, 2006; Thompson & Daniel) over the last few years is the parallel analysis, which is not automated in most conventional statistics packages but is readily available as SPSS syntax (O'Connor, 2000).

With regard to the decision on how to rotate the factors (point 4 in Thompson's [2004] linear sequence of decisions), the first issue is that this critical information was not reported. One assumes that the factors were not rotated due to the fact that this is the default position of SPSS and that a decision to depart from the default position would have been reported by R. Ellis (2005). The possible omission to rotate the solution as part of the statistical analysis is significant because a nonrotated principal components analysis (PCA) will, in most cases, yield a first principal component with large positive loadings. As Kline (1994) pointed out, this is an artifact of the method, and it is for this reason that rotation is used to arrive at simple structure. Given that R. Ellis repudiated the possibility of pure tests of explicit and implicit knowledge, this author would recommend (assuming an exploratory agenda, of course) an oblique rather than orthogonal rotation, as this assumes factors are correlated. As Cattell and Scheuberger (1978, cited in Kline) argued, it is unlikely within psychological phenomena that factors will be orthogonal.⁵

If R. Ellis did indeed omit to rotate the factors, then his constructs (explicit and implicit knowledge) are orthogonal. A PCA that is not rotated always produces orthogonal constructs because the first construct is partialed out of the data matrix before the second is extracted, the second is partialed out before the third is extracted, and so on. A strong case needs to be made for this assumption, particularly because it does not sit comfortably with the claim that there are no pure measures for these constructs.

Turning to the issue of questionable inferences and interpretations, I will now address the claim made by R. Ellis (2005):

The problem here again is circularity. The first part of the claim states that the results lend support to a model or theory of grammatical proficiency underpinned by the explicit and implicit categories of knowledge, but the decision to force two factors was driven by the same theory.⁶

In factor-analytic terms, this resembles the logical fallacy of asserting the consequent. By R. Ellis's reasoning, and this comment is not to be read as facetious, if he had chosen to infringe the eigenvalue-greater-than-one rule even further to arrive at a hypothetically desirable three-factor model, he would have ended up with three factors that accounted for even more than 75% of the variance. One can keep on extracting factors that add to the variance accounted for until the matrix is exhausted. The point is to have a criterion in deciding when to stop extracting that is independent of one's theoretical expectations—that is, if one wants to use an EFA for a verificational purpose at all. The second part of the claim then goes on to state that the respective levels of variance accounted for by the first factor and second factor lend some support to the theoretical position that implicit knowledge is primary. This part of the claim is open to four charges. First, all PCAs will load heavily on the first factor, less heavily on the second, even less heavily on the third, and so on. As previously stated, this is an artifact of the method. Second, it is not reported whether the factors were rotated, but assuming they were not (see inferences made with regard to SPSS), one can expect the disparity between the first factor and second factor to be significantly higher, as this is the artifactual problem that rotation is designed to reduce. Third, assuming for a moment that R. Ellis's model was valid, there is still the problem that the implicit construct was operationalized three times, and the explicit construct only two times. In this design, one has a data matrix dominated by the implicit operationalization, and one would expect the implicit factor to load first and account for much more variance because it is overrepresented relative to the explicit factor. In other words, R. Ellis was interpreting an artifact of his design as empirically meaningful.⁷ Finally, the comment with regard to the low eigenvalue represents an abuse of the criterion because failure to meet it should have led to rejection of a second construct. Instead, it was used to support an argument for a weak second construct.

R. Ellis (2005) made a very powerful claim in the discussion of results:

I challenge this assessment of the study conducted. It might well be the case that the operationalizations did indeed provide relatively independent measures of the constructs in question (implicit and explicit), but the study did not demonstrate this. The design is flawed because of an exploratory statistical tool being selected for the purpose of verification. The stipulation of the parent constructs (implicit and explicit) as two factors, which would have been acceptable in a CFA (a procedure in which the loadings of the five measures would also be hypothesized a priori), constitutes a circular operation when employed within an EFA.

I now turn to the issue of a proposed model for a CFA of R. Ellis's (2005) data. Figure 1 presents a model that uses the path-diagram conventions of AMOS 5.0.1. The model has 11 free parameters and 15 sample moments meeting the criterion of overidentification with four degrees of freedom. The model hypothesizes a priori that observed variables can be explained by two factors (explicit and implicit). Each observed variable has a nonzero loading on the factor it was designed to measure and a zero loading on all other factors. The two factors are correlated, consistent with R. Ellis's repudiation of pure measures for these factors. Error associated with each measure is uncorrelated. One indicator in each factor is fixed to one as a metric for the factor, and this is consistent with normal practice.

It is worth remarking, however, that the model in Figure 1 has few degrees of freedom and is arguably overfitted. This is the consequence of relatively few observable variables per latent construct. It is not really possible to deal with this post hoc, as this is actually a design issue. What this, in truth, indicates is an imperative to operationalize a construct in as many diverse ways as possible. In this context, it is arguable that the problem is a field issue (i.e., SLA as a body of research) rather than a design issue relative to this study, because it is difficult to think of other ways in which these constructs have been operationalized to date. Against this background, I would argue that the major obstacle that SLA research faces in arbitrating this interface debate is the lack of variety in potential operationalizations of these constructs. Therefore, a highly relevant study in this regard is that of Tokowicz & MacWhinney (2005), from the same issue of Studies in Second Language Acquisition; Hulstijn (2005) also pointed to this relevance in the introduction to the issue. The use of neurophysiological data presents new possibilities for diversified operationalizations that could significantly strengthen and ground structural equation modeling (SEM) models, of which CFAs are a subset, within the explicit and implicit knowledge research domain.

Figure 2 presents a rival model for R. Ellis's (2005) data. It is important that rival models are tested because the fit of a particular model does not preclude the possibility that other untested models fit better. The logical foundations of each model should also be explained (Thompson & Daniel, 1996). The logic of this model is derived from an alternative item analysis. Specifications are identical to the first model, except for the shift of one observed variable. The imitation and oral narrative variables constitute production, whereas the other variables all involve some kind of judgment of a stimulus. As such, they involve reflection and reaction. Although the time suppression is designed to suppress reflection in the timed GJT, it is arguable that it does not—especially because the time limit was the average time for a native speaker (NS) plus 20%.⁸

Despite the fact that R. Ellis's EFA does not bode well for this model, it should still be tested.

Another potential model, suggested as a possibility by the EFA, would be a loading of the imitation, oral narrative, timed GJT, and untimed GJT on one factor. This is suggested by R. Ellis's (2005) EFA as a strong possibility. However, this would leave only one observed or indicator variable on the explicit factor, which is unorthodox because there would be no data reduction on the factor; data reduction is, after all, the whole purpose of factor analysis. Nonetheless, this model is possible, provided the error variance for the single indicator is fixed, preferably to a nonzero value that has some rationale behind it (e.g., from previous studies). The other possibility in dealing with the single-indicator unorthodoxy is that R. Ellis could deaggregate his three measures for metalinguistic knowledge and provide them as three separate indicators of an explicit knowledge factor, with the other four indicators (oral narrative, imitation, timed GJTs, and untimed GJTs) on the implicit factor. This solution would be preferable.

The view of the current author is that this last model, which involves the deaggregation of the metalinguistic scores on the explicit factor, is the strongest rival model and should be tested. It has important theoretical implications as well. If the rival model fits better, the logical implication is that a model with explicit knowledge measured purely by metalinguistic knowledge fits better than one that does not measure explicit knowledge purely by metalinguistic knowledge. This would lend support to a view of explicit knowledge being more coextensive with metalinguistic knowledge than might have been previously assumed. Objections to this implication would probably follow, and follow on the basis of what the GJT actually measures. It is clear from the more ambivalent results derived from the GJTs in R. Ellis's (2005) data that GJTs are central to a future debate on the psychometrics of implicit and explicit knowledge, especially because so much past research has been based on them. R. Ellis's data could still be informative in this future debate if reanalyzed. For future studies, I would argue that one promising line of research would involve psychometric examination of the behavior of the GJTs under different conditions of timing, using other more reliable measures of explicit and implicit knowledge (e.g., metalinguistic measures and measures of free production, respectively) as the grounding operationalizations. In this new design, one would seek to verify a hypothesis that the GJTs are increasingly likely to load with the implicit factor (operationalized as free production) as the time limitation is increased and increasingly likely to load with the explicit factor (operationalized as metalinguistic knowledge) as the time limit is decreased. This would be achieved by assessing the model fit under these two different specifications and under different conditions of time constraint.

Departing from the critique of R. Ellis's (2005) study and with a view to the future, I now turn to some suggestions that might be helpful in addressing the psychometric shortfalls to which R. Ellis (2005) and Douglas (2001) referred:

1. Validity is a function of scores or data, not instruments. This means that validations should be conducted routinely in all studies where constructs are operationalized with test instruments.
2. Constructs should be operationalized in as many ways as possible.
3. Aggregation of scores should occur only when necessary (e.g., when converting to interval scales). Aggregation results in a loss of valuable dimensionality in the data and increases the possibility of an overfitted model.
4. The possibility of higher order models should be considered.
5. Full SEM models, the parent domain for CFAs, should also be considered as a powerful method to deal with construct validity and instrumentation issues because they simultaneously address the demand for a research body that moves forward on the conceptual and empirical fronts—a demand emphasized by Hulstijn (2005).

Concerning the first point, Feldt and Brennan (1989) and Wilkinson and the American Psychological Association Task Force on Statistical Inference (1999) have drawn particular attention to the fact that reliability is a property of scores derived from an instrument for a population of examinees. The SLA research community has a proclivity to treat validity as a property of the instrument. This is not the case. Obviously, repeated studies that indicate that a certain instrument consistently produces reliable results across populations will support the usefulness of the instrument. However, it cannot be inferred that reliable scores in one study on the basis of a particular instrument demonstrate the validity of that instrument. On a side note, assessment of the reliability of scores is important for effect sizes, which researchers often neglect to report within SLA.

With regard to the second point, it was mentioned earlier, in discussing Figure 1, that a construct should be operationalized in as many ways as possible. This helps to ensure that a CFA meets the criterion of overidentification and, more generally, reduces the possibility of an overfitted model. Connected to this is the third point. Aggregation of scores can potentially and unnecessarily reduce valuable dimensionality in the data. This point can be illustrated using R. Ellis's (2005) study. The metalinguistic test comprised three separate parts for which the scores were aggregated. This means that three separate operationalizations of metalinguistic knowledge were collapsed into one. These could have been kept separate, also opening up the possibility of a second-order recursive model (assuming a different scoring regime on some of the other scales) in which metalinguistic knowledge is a subconstruct of explicit knowledge loaded by three separate indicator variables.

This raises the fourth point, that higher order models be considered in the future. For the purposes of illustration and further discussion of the prospects that second-order models offer, Figure 3 is presented. It should be noted that design features and scoring exigencies preclude the application of this model to R. Ellis's (2005) data. What distinguishes this model from those presented in Figures 1 and 2 is the presence of a second tier that would facilitate a theoretical taxonomy of the explicit construct. This tier is represented under the explicit factor and is also latent. As stated, the model is for illustrative purposes and does not require precise specification in the current article. Some features, however, are worth noting. First, the first-order factors are interesting because they appear to be both dependent and independent variables at the same time, which is methodologically unacceptable. As Byrne (2001) pointed out, however, this is not the case. The second-order factor is hypothesized to account for all variance and covariance associated with its subordinate factors—hence, the absence of the double-headed arrows on the first-order factors, which indicates that their factor variances and covariances are not to be estimated. The prediction of the second-order factors by the first-order factors is taken to include error, and this is represented by the residual error terms. I would recommend further reference to Byrne and Kline (2005) for discussions of these issues.

To elaborate on the potential of the type of model presented in Figure 3, I turn again to R. Ellis's (2005) study (as a frame of reference and not as the object of critique) and other previous studies. R. Ellis made the point that “learners' explicit knowledge exists independently of both the metalanguage they know and their ability to explain rules” (p. 147), citing his earlier study (R. Ellis, 2004). He also cited Bialystok's (1979) early recognition of this issue. The two operationalizations (untimed GJT and metalinguistic knowledge) for explicit knowledge do not clearly differentiate these aspects, as neither of them required verbalization of rules. The untimed GJT was a judgment test, and the metalinguistic knowledge test comprised three kinds of tests, none of which involved verbalization of rules. R. Ellis (2004) argued that the potentially verbalizable part of explicit knowledge is better measured receptively than productively, as receptive knowledge usually exceeds productive knowledge. Although there is a very strong case that productive knowledge is a subset of receptive knowledge, it is still arguable that the operationalizations in R. Ellis's study do not fully circumscribe the theoretical claim that explicit knowledge is potentially verbalizable. In the second-order model in Figure 3, the first-order construct of metalinguistic knowledge could hypothesize the loading of indicators that include both measures of receptive and productive metalinguistic knowledge, leading to a more elaborated operational expression of the construct. Furthermore, it can do this while keeping these productive and receptive indicator variables or operationalizations of metalinguistic knowledge conceptually separate from the GJT.

In continuing this elaboration of the potential of models like the one presented in Figure 3, and again turning to R. Ellis's (2005) study as a frame of reference, the metalinguistic test on the explicit factor was labeled metalinguistic in his study, which begs the question as to what the untimed GJT is if it is not metalinguistic. This is particularly relevant when viewed in terms of comments made by Hulstijn (2005) in the same issue that “definitions of learning—whether implicit or explicit—as a process (how) can easily become contaminated with the object of learning (what)” (p. 133). The subjects were, after all, asked whether the item was grammatical or not, and the what part of this has metalinguistic import. The part of explicit knowledge that is taken to be independent of metalanguage is often referred to as analyzed knowledge. This terminology rescues the notion that this part of explicit knowledge is not part of a subsymbolic architecture while avoiding the claim that it is coextensive with metalanguage, which, quite obviously, is part of a symbolic architecture. There is a troublesome feeling here that the notion of analyzed knowledge is negatively specified rather than positively specified. What is required is some kind of positive theoretical articulation of what the symbolic basis of the analysis is, if it is not metalingual.

Now it might seem at this point that this last argument is about to sink beneath the burden of a hair-splitting analysis, but there is a genuine gray area here that is arguably as bedeviling as the parent debate concerning the interface between explicit and implicit knowledge. If metalinguistic and explicit knowledge are related but not exactly the same thing, then what are they and what is the nature of the connection, or dare we say interface? If metalinguistic knowledge is that part of explicit knowledge that is potentially verbalizable, then what is the part that is not verbalizable and that is distinguishable from this metalinguistic knowledge? Also, what is its symbolic architecture if not metalingual? Once theoretically specified, how do we then operationalize it as such and introduce it as part of a fully elaborated and testable construct of explicit knowledge? It is beyond the scope of this discussion to try to tackle this problem fully in the context of the current critique. However, as a way of showing how a CFA in general (and a second-order model in particular) can begin to assist with the problem, four potential hypotheses or rival models of the relationship between the two postulated aspects of explicit knowledge are presented in Table 1. These are testable—with some careful thought on theoretical taxonomy, model specification, and operationalizations. The fourth hypothesis is enabled by a second-order model. I argue that the difficult part of this task will come in theoretically articulating and operationalizing the sibling constructs to metalinguistic knowledge in a second-order model (the fourth hypothesis in Table 1). The main charge to which the hypotheses in Table 1 are exposed concerns the use of correlation between constructs as an indicator of the degree of interface or coextensive boundary. There is a case to be made here. Statistical correlation is not necessarily isomorphic with interface; however, this author still maintains that the testing of these hypotheses would be a good start.

One other important comment is worth making with regard to what Figure 3 illustrates—one that is conceptually related to comments made previously about the positive theoretical specification of the nonmetalinguistic part of explicit knowledge. The first-order constructs are labeled untimed GJT and metalinguistic knowledge. Metalinguistic knowledge is a construct, not an operationalization, and it is appropriately represented at the first-order level of constructs. It is as much a construct as explicit knowledge is—only a subconstruct. As a construct, it can be operationalized in a number of different ways (e.g., receptively and productively). The status of the untimed GJT is less clear, however, because it is arguably more of an operationalization than a construct. It is, after all, a test. It should be moved down to the indicator level and its parent construct theoretically articulated. This parent construct could then be said to represent the nonmetalinguistic part of explicit knowledge with untimed GJTs as one of its potential operationalizations or indicator variables. In the past, the parent construct of the untimed GJT was hypothesized to be explicit knowledge. However, if that is the case, where does this place metalinguistic knowledge, if metalinguistic knowledge is a construct rather than an operationalization? There is a problem of theoretical order and conceptual clarity here, and the purpose of my explicating this is not to lead the reader around in what might appear to be taxonomic indulgence but, rather, to demonstrate how these SEM models compel rigor in theoretical argumentation.

In closing, it is worth remarking on the potential that SEM models have for research into explicit and implicit knowledge. As stated previously, CFAs are a subset of SEM models, and this subset has been the focus of this critique. SEM as a methodology in general, however, offers a powerful tool to the general project of theory construction and empirical inquiry in the area of explicit and implicit knowledge. Hair, Anderson, Tatham, and Black (1998) referred to SEM as being distinguished by two characteristics: (a) estimation of multiple and interrelated dependence relationships and (b) the ability to represent unobserved concepts in these relationships and account for measurement error in the estimation process (p. 584). These capabilities situate SEM very well for the demands of work in the area of implicit and explicit knowledge, where researchers are dealing with constructs that have multiple indicators and that are simultaneously complex, potentially correlated, and latent. To take one example, a challenge for theory building in implicit and explicit knowledge was identified by Hulstijn (2005) in his remarks on Robinson's (2005) article in the same issue: “If language aptitude, intelligence, and working memory can be conceptually related to the constructs of implicit and explicit learning and knowledge, their status might change from peripheral and correlational to central and causal” (p. 136). SEM would be well placed to test models of implicit and explicit language learning where constructs such as aptitude, intelligence, and working memory are introduced in the representation of a complex behavioral reality. Models can be specified in which these factors are represented in different configurations of correlation and causation and then tested, thus marrying the empirical and conceptual fronts and avoiding what Hulstijn termed the “pitfalls of extreme rationalism and extreme empiricism” (p. 138). Furthermore, the imperative for precise specification of the model in SEM, including all indicator variables, error, and correlation between latent constructs or factors, forces rigor in theoretical argumentation.

However, having said this, my final comments turn to the future. Even more so than standard inferential statistics, SEM is full of statistical hazards that imperil the unwitting researcher. It is an extremely complex methodology for those of us schooled in the staple of research methods courses in psychology and linguistics programs but not in the matrix algebra of a pure statistics background—and I include myself here. Indexes of model fit, for example, are biased by a variety of factors such as sample size, univariate nonnormality, multivariate nonnormality, and missing data. Model specification is a complex issue and a variety of criteria have to be met for appropriate inferences to be drawn. Overfitted models work, but cannot help but to work, and explain little. The researcher needs to be acutely aware of Monte Carlo-type studies that indicate the behavior of different indexes under different conditions of data properties. In this context, it is imperative that the review process for articles based on this methodology include the oversight of a specialist in the area. Without this, there is as much potential for damage to the research trajectory as there is for benefit.