Ellis (2005) and his coresearchers developed a number of tests with a view to providing relatively separate measures of explicit and implicit knowledge. The aim in the development of these tests was to resolve a continuing problem in SLA studies—namely the construct validity of tests used to measure acquisition—and, more specifically, to provide a basis for investigating the relationship between implicit and explicit knowledge (i.e., the strong interface, the weak interface, and the noninterface positions).
The procedure followed was first to provide notional definitions of these constructs and then operational definitions. On the basis of the latter, five tests were designed: The first three were intended to provide measures of implicit knowledge, and the second two were measures of explicit knowledge. For the reader's convenience, the five tests are described in Table 1, together with measures of reliability. These tests were administered to 91 second language (L2) learners (83 in the case of the oral narrative test). These learners were of mixed English language proficiency. A detailed account of the methodology of the study is available in Ellis (2005).
Description of the five tests
Ellis elected to analyze the data using an exploratory principal components factor analysis,1
One point not raised by Isemonger (this issue) concerns the use of a principal components analysis versus a factor analysis. There is some disagreement among statisticians regarding the preferred analysis (Field, 2005), and in order to avoid further criticism, a maximum likelihood factor analysis was also performed. The results of the obliquely rotated pattern matrix did not differ from the principal components analysis presented in this article.
In his critique of Ellis's (2005) study, Isemonger (this issue) argued that (a) assuming that an exploratory factor analysis is appropriate, an oblique rather than an orthogonal rotation was required, as the assumption was that the factors would be correlated, and, in any case, (b) given that Ellis's study had a verificational rather than an exploratory agenda (i.e., he was testing an operationalization of a particular model of L2 knowledge), a confirmatory factor analysis should have been employed. Although we feel that Isemonger has taken little cognizance of past practice in applied linguistics and SLA (as reflected in the studies mentioned in the previous paragraph),2
It is interesting to observe that Isemonger (this issue) cited Wintergerst, DeCapua, and Itzen (2001) in a note. His purpose in doing so was to claim that psychometric research examining Reid's perceptual learning style questionnaire has questioned its validity. We found ourselves wondering how Isemonger could cite the Wintergerst et al. study in this respect, given that it would seem subject to the same criticisms he has leveled at our use of factor analysis.
PRINCIPAL COMPONENT ANALYSIS WITH OBLIQUE ROTATION
The principal component analysis that we originally conducted employed a varimax orthogonal rotation.3
Isemonger (this issue) rightly pointed out that Ellis (2005) neglected to state whether the principal component analysis used in his study was rotated and went on to assume that the solution Ellis presented was not rotated. This assumption is incorrect. It should also be noted that the original manuscript sent to the editor of Studies in Second Language Acquisition explicitly indicated that varimax rotation was employed. Unfortunately, this information was deleted in the publishing process. R. Ellis takes complete responsibility for this error.
Isemonger also criticized Ellis's (2005) study for the fact that the eigenvalue of the second factor was below one, citing several studies that claimed that even solutions using eigenvalues equal to or greater than one might overestimate the number of factors that can reliably be extracted. We draw the reader's attention to the facts that the eigenvalue of factor 2 in the oblique solution is very close to one and that an inspection of the scree plot clearly justified a two-factor solution. Also, we note that there is divergence of opinion among statisticians regarding the basis for selecting the number of factors. For example, some studies have argued that an eigenvalue equal to or greater than one is too strict and, furthermore, argued for retaining all factors with eigenvalues greater than 0.7 (Jolliffe, 1972, 1986; cited in Field, 2005). Finally, Field recommended an inspection of the communalities after extraction, with values close to 1.0 being desirable. Table 2 shows that the communalities are considerably higher for the two-factor solution than for the one-factor solution.
Factor analysis communalities
The fact that each variable loads primarily on only one factor, as shown in Table 3, arguably removes the need to conduct a confirmatory factor analysis. However, given the doubts that Isemonger has expressed regarding the low eigenvalue of factor 2 as well as his insistence on the need to test alternative solutions, we will now present the results of such an analysis.
Principal component analysis of the five tests (direct oblimin)
CONFIRMATORY FACTOR ANALYSIS
Two confirmatory factor analyses were carried out using AMOS 5.0 (Arbuckle, 2004). The first analysis tested the construct solution predicted by the design of the tests (i.e., that the oral imitation test, the oral narrative test, and the timed GJT would load on an implicit factor and that the untimed GJT [ungrammatical] and the metalinguistic test would load on an explicit factor). The second analysis tested the alternative method solution proposed by Isemonger (i.e., that the oral imitation test and the oral narrative test would load on a production factor, whereas the timed GJT, the untimed GJT [ungrammatical], and the metalinguistic test would all load on a decision factor). The results are shown in Figures 1 and 2.
Implicit/explicit model.
Decision/production model.
Table 4 provides a summary of the model fit for both solutions, with the χ2 statistic, testing the goodness of fit, presented. A significant value indicates that the model is statistically unlikely to occur, whereas a nonsignificant value indicates an acceptable model (Byrne, 2001). In the current analysis, the χ2 for the implicit/explicit model is not significant; however, the χ2 value for the decision/production model is significant. Additionally, Byrne (2001) noted that a normed fit index value of greater than .95 indicates a superior fit and that a root mean square error of approximation value of less than .05 indicates a good fit for the model. Thus, according to these indicators, the implicit/explicit model is an acceptable model, whereas the decision/production model is not.
Summary of the model of fit for both solutions
OPERATIONALIZING IMPLICIT AND EXPLICIT KNOWLEDGE
Isemonger also addressed our operationalizations of implicit and explicit knowledge. He wondered how the timed GJT could provide a measure of implicit knowledge given that the test invites a judgment of the correctness of sentences—knowledge that he considers to be metalinguistic. Here, Isemonger seems to have missed the point. Speakers of a language are perfectly able to decide whether a particular usage is grammatical without any explicit knowledge of the rule or feature involved. Indeed, the plethora of studies that have utilized GJTs have been based on precisely this assumption. The rationale for the timed GJT was that learners would only be able to judge the sentences on the basis of their implicit knowledge because the speed of the response required precluded access to their explicit knowledge.
Second, drawing on Ellis (2004), Isemonger noted that explicit knowledge (as analyzed knowledge) exists independently of a learner's knowledge of metalingual terms, and he then argued that the two operationalizations of explicit knowledge (the ungrammatical sentences in the untimed GJT and the metalinguistic knowledge test) failed to differentiate these two aspects of explicit knowledge. In fact, the untimed GJT was intended as a measure of analyzed knowledge, whereas the metalinguistic knowledge test was designed to measure learners' knowledge of metalanguage. Isemonger's problem with this rests in the fact that neither of these tests measures potentially verbalizable explicit knowledge. Fairly obviously, however, no test is capable of measuring what is potentially verbalizable, but only what is verbalizable. We chose not to include a test that required learners to verbalize rules for two reasons. First, from a practical standpoint, it was easier to assess their receptive knowledge of metalingual terms. Second, when asked to verbalize a rule, learners might construct ad hoc explanations at the time (i.e., they might create explicit knowledge rather than draw on their existing explicit knowledge). In short, whereas it might be helpful to conceptualize explicit knowledge as potentially verbalizable, it might not be possible to operationalize such a notion for testing except in terms of learners' knowledge of metalingual terms.
CONCLUSION
In his critique of Ellis (2005), Isemonger also offered a list of suggestions for avoiding the psychometric shortfalls in SLA studies that both Ellis (2005) and Douglas (2001) warned against. We find this a useful list—one to which all SLA researchers who examine relationships among quantitative datasets should attend. We recognize that Ellis's study was lacking with respect to a number of the points on this list. However, we feel that the soundness of the general conclusion that Ellis reached—namely, that the tests he examined did succeed in providing relatively separate measures of L2 implicit and explicit knowledge—was demonstrated. This is even more clearly demonstrated by the additional analyses presented in this response to Isemonger.
One of Isemonger's suggestions is that higher order models be investigated. In the case of the data available from Ellis's (2005) study, this would be possible (as Isemonger pointed out) by disaggregating the test scores. For example, it would be possible to enter separate scores for the grammatical and ungrammatical sentences in the GJTs and for the three parts of the metalinguistic test. This constitutes an avenue for further research based on the same datasets that we look forward to exploring in the future.
