LITERATURE REVIEW
LANGUAGE APTITUDE
Language aptitude is a predisposition to learn languages well or rapidly (Carroll, Reference Carroll and Diller1981). This special talent for language that can predict how fast a second language (L2) is acquired (i.e., rate of attainment) or how successfully it is acquired (i.e., ultimate attainment) is componential in nature (Carroll, Reference Carroll and Glaser1962; Skehan, Reference Skehan and Robinson2002). Despite the consensus on its multicomponential nature, various aptitude models have been proposed, each with different subcomponents and ultimately a different understanding or conceptualization of the construct. One of the first models, and still the most influential, was Carroll’s (1962) four-factor aptitude model. Carroll derived aptitude components in a bottom-up fashion after running a series of factor analyses on a battery of approximately 20 verbal tests that were felt to be good predictors of success in intensive foreign language training. The tests were administered to male military personnel between 1953 and 1958. Carroll’s job was very clear and that was to help the U.S. Army find and train people who would learn foreign languages with ease in short periods, a need that became obvious with the events of World War II. The main goal of Carroll’s project was, therefore, predictive rather than construct validity. More specifically, Carroll investigated the extent to which aptitude could predict initial foreign language learning rate as measured by academic course grades.
Carroll’s model included phonetic coding ability (codifying, assimilating, and recalling phonetic material), rote memory (ability to recall words in an unfamiliar language), grammatical sensitivity (identifying grammatical functions of words or phrases in sentences), and inductive language learning ability (figuring out the rules of systematically varying language materials). Three of these four factors gave rise to the Modern Language Aptitude Test battery (MLAT; Carroll & Sapon, Reference Carroll and Sapon1959). Inductive language learning ability was measured in the initial battery of tests, but it was not measured to any appreciable degree in the final five-test battery (Carroll, Reference Carroll and Glaser1962).
Similar test batteries to the MLAT are the Defense Language Aptitude Battery (DLAB; Petersen & Al-Haik, Reference Petersen and Al-Haik1976), only in use by the U.S. Department of Defense, and the Pimsleur Language Aptitude Test Battery (PLAB; Pimsleur, Reference Pimsleur1966), for young learners between ages 12 and 17. The DLAB was developed by the Defense Language Institute after the MLAT as a battery for differential prediction, to decide in what language or language family a candidate might do best. It mainly measures the ability to learn the grammatical rules of systematically varying language materials. The PLAB, presented as an alternative to the MLAT for the junior high school level, measures vocabulary knowledge in English, language analytical ability, sound discrimination, and sound–symbol associations.
These early test batteries, which have shaped our understanding of language aptitude for decades, suffer from various shortcomings. First, they were developed for selection purposes and, therefore, had predictive validity as a goal; they were not concerned with revealing the underlying structure of the construct. Second, they are characterized by being heavily weighted in favor of explicit cognitive abilities, especially rote (or explicit associative) memory, and language analytical (or explicit inductive learning) ability. As a result, aptitude has been mostly understood as an ability to consciously reflect on linguistic structure and to memorize (a good predictor of outcomes under methods such as the audio-lingual), rather than as an ability to acquire patterns without awareness.Footnote 1
The impact of this conceptualization on the predictive accuracy of the MLAT in learning contexts different from the ones for which the MLAT was originally conceived for is obvious. In fact, an internal report by the U.S. Department of Health, Education, and Welfare that summarized Carroll’s findings already suggested that the MLAT could predict some types of learning better than others: “[R]eading and writing superiority tend to be chiefly predicted by section IV of the MLAT, ‘Grammatical Sensitivity;’ by time not spent abroad; and third college literature courses” (Markessinis, Reference Markessinis1968, p. 28), whereas “active performance is predictable by relatively lower scores on section III of the MLAT, time abroad, occasional or frequent home use of the foreign language; early language study; introductory and intermediate courses; [and] poor rote memory” (Markessinis, Reference Markessinis1968, p. 28).
More recent aptitude models lack the strong empirical support the MLAT has in the literature but have been able to provide more detailed conceptualizations of the construct. Skehan (Reference Skehan and Robinson2002) and Robinson (Reference Robinson and Robinson2002) have taken a processing stage approach in which the aptitudes needed for each stage of L2 learning may differ. Skehan (Reference Skehan and Robinson2002) proposed four broad processing stages (noticing and registering input, patterning, controlling analyzed knowledge, and lexicalizing) and matched them with relevant cognitive abilities and processing resources. Robinson’s (2002) model included a number of aptitude complexes that are differentially related to language processing under different instructional conditions. The model predicted interactions between individual cognitive abilities and instruction type, with the underlying assumption that matching learners to instructional options will optimize learning.
The strength of Skehan’s and Robinson’s models lies on the fact that both rely on key issues in SLA theory and pedagogy, such as different processing stages in acquisition and the type of information processing involved in different learning contexts. They also consider the advances in the understanding of human memory systems that have taken place since the MLAT. However, unlike the MLAT, these models have not been developed into test batteries that could be empirically tested. As a result, their predictive validity is largely unknown. A model that was developed into a test battery with the objective of being an alternative to the MLAT is the Cognitive Ability for Novelty in Acquisition of Language-Foreign (CANAL-F; Grigorenko, Sternberg, & Ehrman, Reference Grigorenko, Sternberg and Ehrman2000). Like Skehan’s and Robinson’s models, CANAL-F takes acquisition processes as its theoretical base. However, it has not enjoyed the popularity of the MLAT, perhaps due to features such as its duration (i.e., three hours) and format (i.e., paper and pencil), which might have made its administration more challenging.
The two most recent language aptitude test batteries, both computer delivered and, therefore, easier to administer, are the LLAMA (Meara, Reference Meara2005) and the High-Level Language Aptitude Battery (Hi-LAB; Doughty et al., Reference Doughty, Campbell, Bunting, Mislevy, Bowles, Koeth, Prior, Watanabe and Lee2010; Linck et al., Reference Linck, Hughes, Campbell, Silbert, Tare and Jackson2013). The LLAMA grew out of a series of projects by MA students at the University of Wales working on aptitude and its measurement and, as pointed out by an anonymous reviewer, it has not been properly validatedFootnote 2 (but see Rogers et al., Reference Rogers, Meara, Aspinall, Fallon, Goss, Keey and Thomas2016 and Rogers et al., Reference Rogers, Meara, Barnett-Legh, Curry and Davie2017, for validity-related analyses of the LLAMA). Although largely based on the MLAT, the LLAMA departs from the MLAT in two main ways. First, the tests are language independent and, therefore, suitable for speakers of any language with a Roman alphabet. Second, the battery includes a sound recognition test that is not present in the MLAT. The test measures the ability to recognize patterns in spoken language (i.e., sequence learning ability), an ability involved in the discovery of language structure by way of statistical properties of the input and that, as Meara (Reference Meara2005) argues, can help learners recognize the small morphological variations in a language. Based on the fact that it does not include a study-phase and that it relies on feelings of familiarity involving fast, automatic processes, rather than on conscious recollection involving slow, controlled search processes, Granena (Reference Granena, Granena and Long2013a) argued that the learning conditions created by the test are closer to implicit induction (i.e., acquiring patterns unintentionally through exposure) than explicit induction (i.e., figuring out rules and relations).
The Hi-LAB (Linck et al., Reference Linck, Hughes, Campbell, Silbert, Tare and Jackson2013) was created to predict high-level L2 attainment with a focus on interpersonal communication. The battery measures a set of 11 cognitive and auditory perceptual abilities grouped into six different constructs, several of which were not present in the MLAT or in any of the first aptitude test batteries. Two of these constructs, long-term memory retrieval and implicit learning, rely on abilities in the domain of implicit cognitive processes. Long-term memory retrieval is measured by a semantic priming task showing subconscious meaning associations (i.e., the ability to activate semantic networks in an unconscious way), whereas implicit learning is measured by a serial reaction time task showing subconscious pattern-learning (i.e., the ability to pick up on patterns without needing to think about them consciously). Except for processing speed, all the other constructs in the battery (working memory, explicit associative memory, and perceptual acuity) fall well under abilities that engage explicit, attention-driven cognitive processes.
These two aptitude test batteries show that the current understanding of aptitude is that of a multicomponential construct encompassing abilities from both the implicit and explicit cognitive domains. Some of these abilities differ from conventional aptitudes in that they involve cognitive processes. The possibility of considering learning and memory processes as a domain of aptitude constructs was, in fact, already entertained by Carroll and Maxwell (Reference Carroll and Maxwell1979). Their review acknowledges the promising contribution of experimental cognitive psychology in identifying cognitive processes in perception, learning, memory, problem solving, and other mental activities as cognitive aptitudes. This involves the study of individual differences in task performance as measured by reaction times, error rates, or other indicators.
In sum, the advances in the conceptualization of aptitude have followed calls for new aptitude constructs in educational psychology (e.g., Woltz, Reference Woltz2003) and cognitive psychology (e.g., Kaufman et al., Reference Kaufman, DeYoung, Gray, Jimenez, Brown and Mackintosh2010) and are more aligned with developments in our current understanding of how second languages are learned and taught (Long & Doughty, Reference Long and Doughty2009).
LANGUAGE APTITUDE AS A PREDICTOR OF L2 PROFICIENCY
There are multiple factors that account for individual variation in language learning success, but language aptitude is considered one of the best predictors, even if the remaining variation is still considerable and explained by a variety of other factors. Carroll’s research (e.g., Carroll, Reference Carroll1963, Reference Carroll1969) showed the power of aptitude as a predictor of course grades in initial foreign language learning (i.e., learning rate). The MLAT yielded correlations in the range of .4 to .6 (Carroll, Reference Carroll and Diller1981). Similar predictive validity coefficients were reported for the PLAB and the DLAB. Ehrman and Oxford (Reference Ehrman and Oxford1995) also found that aptitude was the strongest predictor of language proficiency (up to .51), among predictors that included learning strategies, learning style, personality, motivation, and anxiety in an intensive language learning course. Similar results were reported by Sparks et al. (Reference Sparks, Ganschow and Patton1995), in this case with high school foreign language learners. These studies, all of which used the MLAT as a measure, indicate that aptitude is a relevant predictor of academic language proficiency in instructed settings. In fact, a recent meta-analysis (Li, Reference Li2016) examining the construct validity of aptitude reported a strong effect size (.49) between aptitude, mostly as measured by the MLAT, and general L2 proficiency when operationalized as course grades or scores on general proficiency tests. The effect size, however, dropped to moderate for specific aspects of L2 learning such as L2 grammar learning (.32), vocabulary, or L2 writing (.34) (Li, Reference Li2016).
Most of the research conducted in relation to the predictive validity of aptitude has been done using the MLAT and, therefore, it has followed a traditional conceptualization of aptitude. There is far less research done on the predictive validity of aptitude using more recent conceptualizations and test batteries. Li’s (2015) meta-analysis investigating the relationship between aptitude and L2 grammar acquisition, for example, only included three studies that used the LLAMA out of a sample of 33. Li’s 2016 meta-analysis investigating aptitude and general L2 proficiency included two out of a sample of 66. These two studies (Abrahamsson & Hyltenstam, Reference Abrahamsson and Hyltenstam2008; Granena & Long, Reference Granena and Long2013) were carried out in naturalistic learning contexts with very advanced, near-native L2 speakers. In an instructed setting, Linck et al. (Reference Linck, Hughes, Campbell, Silbert, Tare and Jackson2013) used the Hi-LAB to predict L2 proficiency, operationalized as listening and reading skills as measured by the Defense Language Proficiency Tests. More than 522 L2 learners participated in the study, all members of the U.S. military or personnel from government agencies. The results of the study indicated that listening and reading L2 skills at the most advanced levels were related to phonological short-term memory, explicit associative memory (rote memory), and implicit learning.
These three cognitive abilities were able to distinguish high-attainment L2 learners from control (i.e., mixed-attainment) learners. Phonological short-term memory was a significant predictor of reading skills, while rote memory and implicit learning predicted both listening and reading skills. The strongest relationships identified were between rote memory and listening (.67) and between implicit learning and listening (.51). These findings were considered to give support to the hypothesis that the Hi-LAB would show stronger relations to listening outcomes than reading outcomes and to the fact that it would be a good predictor of spoken language skills, the original aim of the battery.
Despite the evidence provided, Linck et al. (Reference Linck, Hughes, Campbell, Silbert, Tare and Jackson2013) pointed out the need for additional research on the predictive validity of the Hi-LAB constructs, especially in the area of speaking proficiency and subskills such as fluency. The study of the relationship between cognitive factors and linguistic skills in L2 has important theoretical and practical implications. Theoretically, it sheds light on the type of learner internal factors that affect the speed of and success in L2 learning, thus contributing to a more accurate model of SLA. Practically, it provides valuable information that can help tailor instruction to student characteristics, inform curricular design, place or select language learners, or diagnose learning problems.
PURPOSE OF THE STUDY AND RESEARCH QUESTIONS
The present study is part of a broader project that investigated the role of cognitive aptitudes in an instructed L2 learning setting. As part of the project, tests from the two most recent aptitude test batteries available in SLA were administered (i.e., LLAMA and Hi-LAB). The current study reports on the results of an exploratory factor analysis that examined the underlying structure of the Hi-LAB and LLAMA tests employed in the project and the extent to which the observed underlying constructs could predict L2 speaking proficiency in terms of complexity, accuracy, and fluency (CAF).
This study was guided by the following two research questions, which aimed to find out, first, whether there were any constructs in common underlying the tests in the LLAMA and the Hi-LAB aptitude test batteries and, second, whether the constructs identified could predict L2 Spanish speaking proficiency in terms of CAF.
1. Are the LLAMA and Hi-LAB subtests measuring any common constructs?
2. Are any of these constructs good predictors of L2 speaking proficiency in terms of complexity, accuracy, and/or fluency?
METHODOLOGY
PARTICIPANTS
A total of 135 college-level learners of L2 Spanish took part in this study. The requirements to participate were (a) to be a native speaker of English, and (b) to have an intermediate level of Spanish (the equivalent of a 200-level course). All of them responded to a flyer advertising the study and were further screened by means of a background questionnaire. Ninety were female (66.7%) and 45 were male (33.3%). Their mean age at testing was 20 (SD = 4.3) and their mean age of onset was 13 (SD = 4.4). Six had been in Spanish-speaking countries for stays longer than one month and shorter than a year (2, 3, 4, 6, 9, and 10 months). The rest had never stayed abroad for purposes other than travel, and those stays ranged between three days and four weeks. Participants were enrolled in a variety of majors. Twelve were studying Spanish as a minor.
COGNITIVE TESTS
A total of eight tests measuring different cognitive abilities were administered. Four belonged to the LLAMA aptitude test battery (Meara, Reference Meara2005): LLAMA_B, LLAMA_D, LLAMA_E, and LLAMA_F. Four belonged to the Hi-LAB (Doughty et al., Reference Doughty, Campbell, Bunting, Mislevy, Bowles, Koeth, Prior, Watanabe and Lee2010; Linck et al., Reference Linck, Hughes, Campbell, Silbert, Tare and Jackson2013): Paired Associates, Letter Span, Available Long-Term Memory (ALTM) Synonym, and Serial Reaction Time Task. Three of these tests were selected based on the findings by Linck et al. (Reference Linck, Hughes, Campbell, Silbert, Tare and Jackson2013), which showed that, out of the 11 Hi-LAB tests, Paired Associates, Letter Span, and Serial Reaction Time Task were significant predictors of general L2 proficiency. The fourth test, ALTM Synonym, a priming test, was added to the current study as a novel and underinvestigated measure of aptitude. Each of the tests is described in the following text.
LLAMA_B
LLAMA_B is a vocabulary learning test that measures the ability to learn new words. It is loosely based on the paired-associates test in the MLAT and, therefore, a measure of explicit associative memory. The 20 words to be learned, taken from a Central American language, are presented visually and linked to a target image. Participants are given two minutes to study the word–picture associations. In the testing phase, the program displays one of the words and participants have to identify the correct picture on the screen.
LLAMA_D
LLAMA_D is a sound recognition test that measures the ability to recognize sound sequences. Participants listen to a string of 10 sound sequences that are computer generated and based on words from a British Columbian Indian language. There is no study phase and stimuli are only played once. Then, participants complete a recognition test where they have to discriminate between old and novel items.
LLAMA_E
LLAMA_E is a sound-symbol association test that measures the ability to form novel sound-symbol associations. Similar to the MLAT phonetic script subtest, it requires participants to work out relationships between sounds (i.e., recorded syllables), 24 in total, and a written representation of those sounds in an unfamiliar alphabet. Participants are given two minutes to study the associations. In the testing phase, they hear a combination of two syllables and make a choice between two possible symbol representations.
LLAMA_F
LLAMA_F is a grammatical inferencing test that measures the ability to infer or induce the rules of an unknown language (i.e., explicit inductive learning ability). Participants see a set of pictures and sentences in an unfamiliar language describing the pictures, 20 in total, and are asked to work out the rules that operate in the language. In the study phase, participants have five minutes to click on a series of buttons arranged on the screen. For each button, a picture and a sentence are displayed. In the testing phase, participants see a picture and two sentences and have to choose the sentence they consider correct.
Paired Associates
This is a test that measures explicit associative memory, or the extent to which participants can remember novel associations. Participants’ task consists in remembering pairs of words, each composed by an English word and a nonword. The test includes two phases. In the study phase, participants see each pair five times for five seconds each time. In the testing phase, participants are given the nonword and are asked to type the corresponding English word.
Letter Span
This test measures phonological short-term memory. Participants see lists of letters on the screen that they have to recall in the order they were presented. The letters are presented one by one for 900 ms each. The test includes a total of 21 lists of lengths from three to nine. There are three lists of each length, all including consonants, and they are presented in a pseudo-random order. Participants respond by clicking onscreen buttons.
ALTM Synonym
This test measures associative priming, or the extent to which the activation of one concept in participants’ lexicon will activate related concepts, thus indicating facilitation of lexical processing. The test includes two tasks. In the priming task, participants listen to a list of five words. Then, they are shown two topic words, one of which is a synonym for two of the words in the list and one of which is a synonym for the other three words in the list. For example, a participant can hear moist smart damp brilliant soggy, where three words are synonyms of wet and two words are synonyms of intelligent (Was & Woltz, Reference Was and Woltz2007). Participants’ task consists in indicating the word that had more synonyms in the list by answering a question such as “Were there more words meaning wet or intelligent?” In the comparison task, participants see pairs of words, and their task is to indicate whether the meanings of the words are similar or different. There are 18 sets in the comparison task, each of which comes after a priming list from the priming task. Each set includes 12 word pairs, four of which are unscored and eight of which are scored. Nine of the sets are primed and nine are unprimed. In primed sets, one or both words in the pairs are synonyms of one of the two topic words from the priming list (e.g., wet or intelligent). In unprimed sets, none of the words are synonyms of the two topic words from the preceding priming list.
Serial Reaction Time
This test measures sequence learning and has been proposed as a measure of implicit learning (Willingham, Nissen, & Bullemer, Reference Willingham, Nissen and Bullemer1989). Participants see four boxes arranged horizontally on the screen indicating the four positions where an asterisk may appear. Participants’ task consists in pressing the button that corresponds to the location of the asterisk on a response box. The test includes six blocks of 96 trials with an intertrial interval of 500 ms. Asterisks appear in a pseudo-random order in the first and last blocks. In the rest of blocks, asterisks follow a repeating sequence of length 12.
LANGUAGE TEST
Oral Production Task
A picture taken from a spot-the-difference task from New English File Intermediate by Oxford University Press was used as a picture description task. The picture shows a busy street full of people and cars doing various things. Seven of the ongoing scenes were marked with letters A–G, and participants were asked to describe the picture in as much detail as possible paying special attention to the seven activities marked with a letter.
PROCEDURE
Each participant was tested individually at a research lab. The oral production task was administered first. The cognitive tests were administered next on a desktop computer in a randomized order using E-prime 2.0 (Psychology Software Tools, 2011).
DATA CODING
LLAMA_B
An accuracy score is automatically computed by the LLAMA software program for every participant out of a maximum of 100. A correction for guessing is automatically done by the software. The internal consistency of the test (k = 20) for the participant sample (N = 135), according to Cronbach’s alpha, was .81.
LLAMA_D
An accuracy score is automatically computed by the LLAMA software program for every participant out of a maximum of 75. A correction for guessing is automatically done by the software. The internal consistency of the test (k = 30) for the participant sample (N = 135), according to Cronbach’s alpha, was .50.
LLAMA_E
An accuracy score is automatically computed by the LLAMA software program for every participant out of a maximum of 100. A correction for guessing is automatically done by the software. The internal consistency of the test (k = 20) for the participant sample (N = 135), according to Cronbach’s alpha, was .72.
LLAMA_F
An accuracy score is automatically computed by the LLAMA software program for every participant out of a maximum of 100. A correction for guessing is automatically done by the software. The internal consistency of the test (k = 20) for the participant sample (N = 135), according to Cronbach’s alpha, was .66. After deleting the last item on the test, Cronbach’s alpha was .69.
Paired Associates
An accuracy score was computed by adding the total number of correctly recalled English words. The internal consistency of the test (k = 20) for the participant sample (N = 135), according to Cronbach’s alpha, was .87.
Letter Span
An accuracy score was computed by adding the total number of letters participants recall in the right order for a total of 126 (21 lists, three lists of each length from three to nine). The internal consistency of the test (k = 21) for the participant sample (N = 135), according to Cronbach’s alpha, was .76.
ALTM Synonym
A rate score combining response time and accuracy for each comparison was computed. The score shows correct responses per minute, and it results from dividing the number of correct responses in a set by the total response time within that set. Separate scores for same and different comparisons within each of the nine primed sets are computed. These rate scores within each primed set are then regressed on the corresponding unprimed set. This yields a residual priming score that captures variability in the rate score of primed sets (i.e., dependent variable) that is left after accounting for the variability explained by the rate score of unprimed sets (i.e., independent variable). The 18 residual difference scores resulting from same and different comparisons within the nine primed sets are added to create the final score. A positive score means more priming and a negative score, less priming. The internal consistency of the test (k = 18) for the participant sample (N = 135), according to Cronbach’s alpha, was .70.
Serial Reaction Time
A facilitation score was derived as the difference in median reaction times in the last sequential block (block 5) and the last random block (block 6). The higher the score is, the more facilitation, and, therefore, better sequence learning. The internal consistency of the test, using split-half reliability, was .79.
Oral Production Task
The audio files were transcribed using Praat software, Praat software package (version 5.3.67; Boersma & Weenink, Reference Boersma and Weenink2014). Recordings were segmented into Analysis of Speech units (AS-units; Foster, Tonkyn, & Wigglesworth, Reference Foster, Tonkyn and Wigglesworth2000). A “Level 3” analysis, as defined by Foster et al. (Reference Foster, Tonkyn and Wigglesworth2000), was adopted. This excluded minor utterances (e.g., yeah, thanks), verbatim echoes, and certain turn-initial units without a verb, such as elliptical responses. Foster et al. (Reference Foster, Tonkyn and Wigglesworth2000) have shown that using the AS-unit is the optimal way of dividing transcribed oral data into analyzable units. As defined by Foster et al. (2000, p. 365), an AS-unit is “a single speaker’s utterance consisting of an independent clause, or a sub-clausal unit, together with any subordinate clause(s) associated with either.”
Complexity was measured syntactically and lexically, based on Skehan’s (2009) recommendation to supplement CAF measures with measures of lexical use. Syntactic complexity was measured by calculating the mean number of clauses per AS-unit (Foster et al., Reference Foster, Tonkyn and Wigglesworth2000). Lexical complexity was measured in terms of lexical diversity (Read, Reference Read2000; Skehan, Reference Skehan2003) by means of the Guiraud Index of lexical richness (Guiraud, Reference Guiraud1954), which includes a mathematical transformation of the Type-Token Ratio. The Guiraud Index is a widely used measure, which is calculated by dividing the number of word types in a speech sample by the square root of the number of word tokens produced. The square root is introduced to reduce the effect of differences in text length (Vermeer, Reference Vermeer2000).
Accuracy was measured by calculating the mean number of errors per AS-unit (Michel, Kuiken, & Vedder, Reference Michel, Kuiken and Vedder2007). Errors included syntactic, morphological, or lexical choice errors. This measure was preferred to the percentage of error-free clauses (Foster & Skehan, Reference Foster and Skehan1996) based on its greater ability to discriminate among L2 learners with very few error-free clauses.
Finally, fluency was measured as pruned speech rate per minute (i.e., Total number of syllables excluding filled pauses, reformulations, false starts and repetitions/Total seconds of speech × 60). This measure is reported to be a fluency-sensitive measure (e.g., Freed, Reference Freed and Riggenbach2000). Pruned items (i.e., those removed from analysis) included filled pauses, word fragments, repeated words, false starts, replacements, and explicit editing items (Ishikawa, Reference Ishikawa2008).
DATA ANALYSIS
The first stage in data analysis involved performing a principal components analysis (PCA) on the eight cognitive measures administered in the study: the four LLAMA subtests (LLAMA_B, LLAMA_D, LLAMA_E, and LLAMA_F) and four Hi-LAB subtests (Paired Associates, Letter Span, ALTM Synonym, and Serial Reaction Time). The resulting factor scores were saved as variables.
In the second stage of the analysis, factor scores were used as continuous predictor variables in univariate general linear models. Custom models with factor scores as covariates, including all covariate interaction terms, were computed to better understand any complex relationships among the variables. Separate analyses were conducted for each of the dependent variables (accuracy, fluency, syntactic complexity, and lexical complexity). Z-scores were used to standardize the data.
RESULTS
This study had two main goals. First, to explore whether the LLAMA and Hi-LAB test batteries had any underlying constructs in common, and, second, to examine whether any of these constructs were good predictors of L2 speaking proficiency. Table 1 displays the descriptive statistics of the eight cognitive measures,Footnote 3 while Table 2 shows the descriptive statistics of the language task in terms of CAF.
Means, standard deviations, and maximum and minimum values for the eight cognitive tests
Means, standard deviations, and maximum and minimum values for the language task in terms of CAF
Table 3 displays the correlation matrix for all the measures in the study, including predictors and outcomes.
Correlation matrix with predictors and outcome measures
* Correlation is significant at the 0.05 level (two-tailed).
** Correlation is significant at the 0.01 level (two-tailed).
A PCA was first conducted on the eight cognitive tests, to determine any factors on the basis of factor loadings. An oblique rotation method was chosen due to the theoretical expectation that any underlying factors might be correlated. The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy (a measure of how suited data are for factor analysis) was .77. Conventionally, values that are higher than .60 suggest that the proportion of common variance among the variables is low (partial correlations should not be very large if distinct factors are expected to emerge). Bartlett’s test of sphericity was significant (p < .001), indicating that the relationships among the variables were not due to chance. The analysis revealed three factors with eigenvalues greater than 1.0, which accounted for 59.62% of the total variance. The first component accounted for 28.02% of the total variance, while the second and third components accounted for 17.05% and 14.55% of additional variance. According to the rotated solution, five tests loaded on the first component with loadings greater than .44Footnote 4: Letter Span (.55), Paired Associates (.79), LLAMA_B (.75), LLAMA_E (.64), and LLAMA_F (.65). Two tests loaded on the second component with loadings greater than .4: ALTM Synonym (.76) and LLAMA_D (.77). Finally, Serial Reaction Time loaded strongly on the third component (.85) (see Table 4).Footnote 5
Loadings from exploratory factor analysis (PCA)
Note: Loadings greater than .4 appear in bold.
The next step in the analysis consisted in using the saved factor scores as predictors of L2 speaking proficiency. Separate univariate general linear models were computed for each speaking proficiency dimension with the three factors as continuous predictor variables. Custom models that included the three predictors and all possible interactions among predictors were used. The general linear model for accuracy accounted for 6% of the total variance (R 2 = .06). Factors 1, 2, and 3, labeled Explicit Learning Ability, Implicit Memory Ability, and Implicit Learning Ability, respectively, were not statistically significant: Factor 1, F(1, 121) = .689, p = .408, η 2Footnote 6 = .005; Factor 2, F(1, 121) = 2.370, p = .126, η 2 = .020; and Factor 3, F(1, 121) = 1.376, p = .243, η 2 = .010. The interactions among predictors were not statistically significant either: Factors 1 and 2, F(1, 121) = 1.437, p = .233, η 2 = .011; Factors 1 and 3, F(1, 121) = .680, p = .411, η 2 = .005; Factors 2 and 3, F(1, 121) = .024, p = .876, η 2 = .000; and Factors 1, 2, and 3, F(1, 121) = .542, p = .463, η 2 = .004.
The general linear model for fluency accounted for 8% of the total variance (R 2 = .08). In this case, interactions among predictors were not statistically significant: Factors 1 and 2, F(1, 121) = .070, p = .792, η 2 = .001; Factors 1 and 3, F(1, 121) = 2.885, p = .092, η 2 = .023; Factors 2 and 3, F(1, 121) = .321, p = .572, η 2 = .003; and Factors 1, 2, and 3, F(1, 121) = .030, p = .864, η 2 = .000. Factors 1 and 3 were not statistically significant either: F(1, 121) = .494, p = .483, η 2 = .004 and F(1, 121) = 1.241, p = .267, η 2 = .009, respectively. However, Factor 2, the factor labeled Implicit Memory Ability, had a statistically significant effect on fluency scores, F(1, 121) = 4.308, p = .040, η 2 = .034. Although this effect was statistically significant, it was associated with a small effect size, accounting for 3% of the total variance, as also evidenced by its standardized beta coefficient of .19, indicating a weak positive relationship. Specifically, for every increase of one standard deviation in Implicit Memory Ability, fluency was predicted to increase by .19 standard deviations, holding all other variables constant.
The general linear model for syntactic complexity accounted for 6% of the total variance (R 2 = .06). This model did not yield any statistically significant results. Factors 1, 2, and 3 were not significant: F(1, 121) = 1.926, p = .168, η 2 = .015; F(1, 121) = 1.585, p = .210, η 2 = .012; and F(1, 121) = .055, p = .816, η 2 = .000, respectively. The interactions among predictors were not statistically significant either: Factors 1 and 2, F(1, 121) = .143, p = .706, η 2 = .001; Factors 1 and 3, F(1, 121) = .197, p = .658, η 2 = .002; Factors 2 and 3, F(1, 121) = .608, p = .437, η 2 = .005; and Factors 1, 2, and 3, F(1, 121) = 1.218, p = .272, η 2 = .010.
Finally, the general linear model for lexical complexity accounted for 11% of the total variance (R 2 = .11). In this model, Factors 1, 2, and 3 were not statistically significant: F(1, 121) = .000, p = .986, η 2 = .000; F(1, 121) = 3.362, p = .069, η 2 = .027; and F(1, 121) = .299, p = .585, η 2 = .002, respectively. The interactions between Factors 1 and 2, Factors 1 and 3, and Factors 1, 2, and 3 were not significant either: F(1, 121) = .555, p = .458, η 2 = .005; F(1, 121) =1 .882, p = .173, η 2 = .015; and F(1, 121) = .015, p = .901, η 2 = .000, respectively. However, the interaction between Factors 2 and 3, the factors interpreted as Implicit Memory and Implicit Learning Ability, reached statistical significance: F(1, 121) = 4.374, p = .039, η 2 = .035. This interaction indicated that the effect of the Implicit Memory Ability factor on lexical complexity depended upon the value of the Implicit Learning Ability factor. The size of this interaction was small, explaining 4% of the total variance. Its standardized beta of .20 showed a weak positive relationship with lexical complexity. Specifically, the standardized beta value indicated that the contribution of the interaction accounted for shift in lexical complexity of .20 standard deviations.
To explore the nature of this interaction more in detail, separate models were run predicting lexical complexity with Factor 2 (Implicit Memory) at different levels of Factor 3 (Implicit Learning) to assess whether the regression slopes for each of these individual Factor 3 groups are statistically significant. Cohen et al.’s (2003) recommendation to use values corresponding to +1 standard deviation, mean, and -1 standard deviation was followed. Within the group with low- or mid-level Implicit Learning Ability (i.e., below 1 standard deviation or between -1 standard deviation and +1 standard deviation), Implicit Memory Ability was not significantly predictive of lexical complexity: B = .027, t(96) = .258, p = .797 and B = .051, t(104) = .508, p = .612. However, Implicit Memory Ability was significantly predictive of lexical complexity within the group with high Implicit Learning Ability (i.e., above 1 standard deviation), B = .657, t(21) = 3.699, p = .001. These results indicated that Implicit Memory Ability mattered for the development of lexical complexity for L2 learners with high Implicit Learning Ability.
To summarize, the data analysis showed that the eight cognitive tests examined, four from the LLAMA and four from the Hi-LAB batteries, were measuring three underlying latent constructs and that one of the factors, Factor 2, was a significant predictor of L2 speaking proficiency, particularly of L2 fluency as measured by pruned speech rate per minute, a measure of speed fluency (Tavakoli & Skehan, Reference Tavakoli, Skehan and Ellis2005) indicating the extent to which a speaker has a good balance of speed and the ability to keep talking. In addition, Factor 2 interacted with Factor 3 as a predictor of L2 lexical complexity operationalized as lexical diversity.
DISCUSSION
The current study had two objectives. First, to explore possible relationships between the Hi-LAB and LLAMA aptitude tests by examining whether the two batteries had any underlying constructs in common, and, second, to investigate whether any of these constructs were good predictors of L2 speaking proficiency in the context of 135 college students learning Spanish as an L2.
RESEARCH QUESTION 1: COMPARABILITY OF APTITUDE TESTS
A PCA run on eight aptitude tests, the four LLAMA tests and four Hi-LAB tests, identified three underlying constructs that accounted for 59.62% of the total variance in the data. The first factor, which accounted for 28.02% of the total variance, received significant loadings from five of the eight tests, three LLAMA tests (LLAMA_B, LLAMA_E, and LLAMA_F) and two Hi-LAB tests (Paired Associates and Letter Span). These tests have in common the fact they are measuring explicit, attention-driven cognitive abilities involving conscious and reflective learning processes. LLAMA_B and Paired Associates are both adaptations of Carroll and Sapon’s (1959) paired associates measure and require learning associations between a nonword and either a picture or another word. The central executive resources engaged in such a task, which are necessary for guiding attention, strategy selection, planning, decision making, and control and coordination of the processes involved in short-term storage (Van der Linden et al., Reference Van der Linden, Collette, Salmon, Delfiore, Degueldre, Luxen and Franck1999), should also be engaged in the other three tests that loaded on Factor 1. The Letter Span, measuring phonological short-term memory, requires participants to recall lists of letters, and, therefore, relies on the central executive to temporarily store information, while needed for reasoning and processing, and to call up such information (Kane & Engle, Reference Kane and Engle2002). LLAMA_E and LLAMA_F both involve working out relations in datasets and explicitly instruct test takers to “work out the relationship” and “work out the grammatical rules,” respectively. They have in common with paired associates and short-term memory tasks the fact that the rehearsing of testing materials is possible, as well as opportunities for strategy use and problem-solving techniques. Explicit associative learning (i.e., the learning of associations acquired consciously and intentionally) is likely to play a role in all of them. Based on the fact that the five tests loading on the first component are measuring cognitive abilities that involve explicit cognitive and memory processes, Factor 1 was labeled “Explicit Aptitude.”
The second and third factors, which accounted for 17.05% and 14.55% of additional variance, respectively, received loadings from tests that have been argued to tap implicit cognitive processes (e.g., Granena, Reference Granena, Granena and Long2013a, Reference Granena2013b; Kaufman et al., Reference Kaufman, DeYoung, Gray, Jimenez, Brown and Mackintosh2010; Woltz, Reference Woltz2003). ALTM Synonym and LLAMA_D loaded on the second factor, while Serial Reaction Time loaded on the third factor. The ALTM Synonym Task includes a semantic priming component that measures implicit (or procedural or nondeclarative) memory, a type of memory that is not available to consciousness, but that is responsible for the improvement or change in the identification, production, or classification of a stimulus as a result of a prior encounter with the same or a related stimulus (Tulving & Schacter, Reference Tulving and Schacter1990). LLAMA_D, however, is a recognition task where test takers have to discriminate between old and novel items. Recognition tasks are supposed to involve episodic (explicit) memory retrieval. However, recognition can be based on the assessment of stimulus familiarity (Yonelinas, Reference Yonelinas2002). Recognition in LLAMA_D could be familiarity based, that is, recognition based on the sensation that an item has been seen before without explicitly remembering any detailed contextual information. Familiarity-based recognition memory judgments have been shown to rely on the same process that supports conceptual implicit memory (e.g., Wang & Yonelinas, Reference Wang and Yonelinas2012). Given that both LLAMA_D and the ALTM Synonym Task could be measuring implicit memory (i.e., unconscious retention), Factor 2 was labeled “Implicit Memory Ability.”Footnote 7
It should be noted that the reliability of one of the two measures that loaded onto Factor 2, LLAMA_D, was .50. This reliability is considered “poor” according to George and Mallery (Reference George and Mallery2003) and it would be “unacceptable” below .50, according to the same standards. However, Nunnally (Reference Nunnally1967) maintained that even reliabilities of .60 or .50 might be acceptable in theoretical studies. Hair et al. (Reference Hair, Black, Babin, Anderson and Tatham2006) argued that reliability may decrease to .60 and still be acceptable in exploratory studies and research in the social sciences, while Aron and Aron (Reference Aron and Aron1999) proposed that values of .60 or even lower could be adequate in psychology research. Although there is no question that values exceeding .7 should be preferred (Plonsky & Derrick, Reference Plonsky and Derrick2016, proposed .74 as a threshold for a minimally acceptable estimate), there is evidence suggesting that the reliability of implicit learning tasks is typically lower than the reliability of explicit learning tasks because they place fewer constraints on performance processes, which results in noisier data (e.g., Dienes, Reference Dienes1992; Kaufman et al., Reference Kaufman, DeYoung, Gray, Jimenez, Brown and Mackintosh2010; Reber, Walkenfeld, & Hernstadt, Reference Reber, Walkenfeld and Hernstadt1991; Robinson, Reference Robinson1997). For example, Reber et al. (Reference Reber, Walkenfeld and Hernstadt1991) and Robinson’s (1997) replication study reported reliabilities of .51 and .52, respectively. That notwithstanding, the low reliability of LLAMA_D in terms of its internal consistency means that measurement error is present and may have affected the results of the study, undermining the claims made. Therefore, caution should be exercised when generalizing the results obtained in relation to LLAMA_D.
The only task that loaded positively and significantly on the third component, the Serial Reaction Time Task, has been proposed as a measure of implicit inductive learning ability (Nissen & Bullemer, Reference Nissen and Bullemer1987), that is, of an individual’s ability to learn a pattern or rule through simple exposure and without the intent to learn the pattern. Given the fact that Serial Reaction Time was the test with the strongest loading on the third factor (.85) and that the only other significant loading was a negative loading from the Letter Span Test (-.45), Factor 3 was labeled “Implicit Learning Ability.” A possible explanation for the negative loading of the Letter Span Test could be the different nature of the cognitive processes that characterize short-term memory and implicit learning. As a component of working memory, short-term memory engages attention-driven memory processes. While selective attention is required for implicit learning, intention to learn or executive processing resources that guide the focus of attention are not. The inverse relationship between short-term memory and the hypothesized “Implicit Learning Ability” factor could be interpreted as a competition of implicit and explicit cognitive processes in implicit learning such that restricting the contribution of the explicit processes involved in short-term memory results in an enhancement of implicit learning. The claim that implicit and explicit systems operate in a push-pull manner is defended by dual-process theories of cognition (e.g., Jacoby, Reference Jacoby1991). However, it should be noted that the same relationship was not found in the case of Factor 1, “Explicit Aptitude.” In this case, there was no association, neither positive nor negative, between Serial Reaction Time task performance and the factor (.08), perhaps suggesting that explicit cognitive processes do not require restricting the contribution of implicit processes to operate. In terms of cognitive aptitudes, this result could suggest that individuals with better implicit learning ability are those with lower executive functions, but further research is necessary to confirm this claim.
These results suggest that the tests in the LLAMA and the Hi-LAB may have some underlying constructs in common, mainly, one measuring a combination of explicit learning and memory abilities and one measuring abilities in the implicit domain, specifically implicit memory. The constructs in the two batteries, however, do not completely overlap. According to the results, the Hi-LAB is measuring implicit learning in addition to implicit memory, while the LLAMA is not. This shows differences in the conceptualization of aptitude underlying the two batteries, and it is consistent with the broader scope of the Hi-LAB, a battery of 11 tests developed to predict high-level proficiency and that measures abilities such as executive functioning and processing speed, which the LLAMA, largely based on the MLAT, was not designed to tap. The fact that both batteries are hypothesized to predict L2 learning is not in conflict with the lack of a full overlap between their underlying constructs. It may indeed affect their predictive power in particular learning contexts or for particular aspects of L2 learning, but this is precisely one of the challenges that aptitude research has ahead, understanding the role of subcomponents of aptitude for particular learning outcomes.
The exploratory factor analysis performed in this study further showed that cognitive abilities in the implicit domain are not unitary and that, at least, a distinction between implicit memory and implicit learning is necessary. Although there is no consensus in the cognitive psychology literature, one position in memory research is that implicit memory and implicit learning do not overlap when implicit memory is used to refer to a particular nondeclarative memory phenomenon, more specifically to the effects that prior experiences have on participants’ current performance (e.g., Buchner & Wippich, Reference Buchner, Wippich, Stadler and Frensch1998; Reber, Reference Reber2013; Seger, Reference Seger1994). Implicit learning, however, refers to the incidental acquisition of knowledge about the structural properties of the relations between objects or events (Buchner & Wippich, Reference Buchner, Wippich, Stadler and Frensch1998). Also, whereas implicit memory studies tend to be interested in the retrieval stage of memory, implicit learning studies are primarily concerned with the encoding stage (i.e., the acquisition of knowledge). This allows for the possibility that implicit memory and implicit learning are two different aptitude constructs resulting in nonoverlapping individual variation.
RESEARCH QUESTION 2: APTITUDE AND L2 SPEAKING PROFICIENCY
Regarding the second research question, which asked whether any of the constructs identified would be good predictors L2 speaking proficiency, the results showed that the second component, the factor labeled “Implicit Memory Ability,” with significant loadings from ALTM Synonym and LLAMA_D, was able to predict L2 speaking proficiency in terms of a subconstruct of fluency measured as pruned speech rate per minute. Implicit memory is a major form of long-term memory that does not require conscious thought. While implicit learning mostly concerns the encoding of input (i.e., the acquisition of new information), implicit memory mostly concerns the retention and, especially, the retrieval of information. It involves effortless and incidental retrieval of previously learned information that was acquired either intentionally or incidentally (Reber, Reference Reber2013). Some individuals are better at spreading activation in implicit memory and, as a result, at making use of, or retrieving, information more efficiently (Woltz, Reference Woltz2003). The fact that implicit memory ability explains differences in L2 speed fluency suggests that those learners with greater implicit memory ability are more able to recall words in a foreign language quickly and automatically. They may also be more able to build meaningful associations between words and to map meaning to form.Footnote 8
The ALTM Synonym measure, however, was not able to discriminate between high L2 attainment and mixed L2 attainment in Linck et al. (Reference Linck, Hughes, Campbell, Silbert, Tare and Jackson2013). This could be due to the fact that Linck et al. (Reference Linck, Hughes, Campbell, Silbert, Tare and Jackson2013) examined predictability of reading and listening skills, whereas the present study focused on speaking performance. Therefore, it could be that Implicit Memory Ability is more relevant for productive than receptive skills because production requires actual language retrieval. Learners with greater Implicit Memory Ability would be better at retrieving previously learned or known information effortlessly and incidentally.
The fact that the speaking task employed did not focus on language forms and, instead, involved spontaneous, real-time performance could further explain that explicit cognitive aptitudes did not predict L2 outcomes. However, this does not explain the lack of a relationship between the Implicit Learning Ability factor and L2 proficiency. Linck et al. (Reference Linck, Hughes, Campbell, Silbert, Tare and Jackson2013) found that Serial Reaction Time was a significant predictor of L2 proficiency in their study and argued that implicit learning played an important role in achieving high-level L2 attainment. One possibility for this discrepancy is that this study included intermediate-level L2 learners, or the equivalent of a 200-level language course, whereas the participants in Linck et al. (Reference Linck, Hughes, Campbell, Silbert, Tare and Jackson2013) were, at least, at an ILR level 4 on the Defense Language Proficiency Tests (DLPT; Defense Language Institute Foreign Language Center, 2009), that is, between an advanced professional and a functionally native proficiency level. Implicit Learning Ability may be a significant predictor of L2 proficiency at advanced levels, but not at intermediate language levels.Footnote 9 Similarly, more explicit cognitive abilities such as analytical ability could be relevant at initial stages, as hypothesized by Linck et al. (Reference Linck, Hughes, Campbell, Silbert, Tare and Jackson2013), and as amply shown by studies from the MLAT Manual carried out to establish the predictive validity and percentile norms of the MLAT battery (Carroll & Sapon, Reference Carroll and Sapon2002).
The results of this study also revealed a significant interaction between two of the factors (Implicit Memory and Implicit Learning Ability) as predictors of lexical complexity. Lexical complexity has been defined by Bulté and Housen (Reference Bulté, Housen, Housen, Kuiken and Vedder2012) as “the degree of elaboration, the size, the breadth, width or richness of the L2 learner’s [lexical knowledge]” (p. 25). In this study, it was operationalized as lexical variety and used as a measure of the range of an L2 learner’s productive vocabulary. The significant interaction found suggests that those learners with a broader productive vocabulary were those with higher implicit memory and implicit learning abilities. Specifically, the nature of the interaction indicated that the effects of implicit memory ability on this aspect of vocabulary acquisition do not begin to show up unless implicit learning ability is high. This suggests that as L2 learners become more able to acquire more vocabulary incidentally, their implicit memory starts playing a more important role allowing them to access and retrieve words more easily.
Although this study found statistically significant results, the effect sizes obtained by means of eta squared (η 2) were small, indicating that the effects observed could only explain between 3% and 4% of the total variance. Language aptitude has generally yielded correlation coefficients in the .40 to .60 (Carroll, Reference Carroll and Diller1981, p. 93). This means that language aptitude has been able to account for approximately 16% to 36% of the variance as an individual difference. For specific aspects of L2 learning such as L2 grammar learning, vocabulary, or L2 writing, the effect sizes have been shown to be smaller (e.g., .32 or 10% and .34 or 11%, respectively; Li, Reference Li2016). It is likely that other factors not considered in this study can account for greater variability in L2 proficiency. Future research is necessary, particularly research investigating cognitive aptitudes proposed by more recent conceptualizations of aptitude.
CONCLUSION
This study, part of a broader project on the role of cognitive aptitudes in an instructed setting, investigated the underlying structure of the two most recent aptitude test batteries in the SLA field (the LLAMA and the Hi-LAB) to see whether they had any constructs in common. The study also examined whether any of the observed underlying constructs could predict L2 speaking proficiency in terms of CAF. Results showed that the LLAMA and the Hi-LAB include tests measuring the same constructs. Specifically, the tests from the two batteries loaded onto three different factors, which were interpreted as “Explicit Aptitude,” “Implicit Memory Ability,” and “Implicit Learning Ability.” The results further showed that Implicit Memory Ability was a significant predictor of L2 fluency and that it interacted with Implicit Learning Ability as a predictor of lexical complexity. This suggested that L2 learners with greater Implicit Memory Ability might be better at retrieving previously learned or known information effortlessly and incidentally. Implicit Memory Ability was further associated with having richer, more varied vocabulary provided that Implicit Learning Ability was also high, possibly allowing learners to acquire a wider range of words available for retrieval and use in oral production.
LIMITATIONS AND FURTHER RESEARCH
This study had several limitations. First, the study did not include multiple CAF measures that could allow for more fine-grained analyses in terms of CAF subconstructs. They did not target specific language aspects or target structures either. Further research should look at the predictive validity of the factors identified in carefully controlled experimental instructed SLA studies, especially given the small effect sizes found in the current study. As an anonymous reviewer rightly pointed out, there is also a need to examine areas of overlap between different aptitude tests, not only Hi-Lab and LLAMA, as a way of further refining the aptitude construct in a bottom-up fashion, as measured by currently available instruments. Also, implicit memory is typically measured by means of priming tasks. However, it can take different forms and some researchers in SLA (e.g., Brill-Schuetz & Morgan-Short, Reference Brill-Schuetz, Morgan-Short, Bello, Guarini, McShane and Scassellati2014; Faretta-Stutenberg & Morgan-Short, Reference Faretta-Stutenberg and Morgan-Short2018; Morgan-Short et al., Reference Morgan-Short, Faretta-Stutenberg, Brill-Schuetz, Carpenter and Wong2014) have focused on individual differences in procedural memory, a subset of implicit memory in charge of sensory-motor habits or automatic skills. Future research should investigate the extent to which individual differences in implicit memory subtypes overlap and whether they are equally good predictors of L2 outcomes. Finally, the L2 learners in this study were intermediate-level learners and the measures that predicted proficiency differed from those that predicted advanced-level proficiency in Linck et al. (Reference Linck, Hughes, Campbell, Silbert, Tare and Jackson2013). Future studies looking at aptitude as a predictor of general L2 proficiency should consider including different proficiency groups (beginners, intermediate, and advanced) in the same study to gain a better understanding of the role of aptitude in L2 learning.
