Advanced speakers of languages critical for a wide variety of important professional tasks are in high demand but short supply the world over. With increasing frequency, rapidly changing geopolitical circumstances produce unanticipated surge requirements in less commonly taught languages (LCTL), in particular, not to mention rarely taught or never previously taught languages. In some cases, no trained teachers or teaching materials are available, and little is known about what to teach or in what order. These problems are not limited to LCTLs. Few data exist on the appropriate linguistic content, or on psycholinguistically defensible learning sequences—especially at the advanced levels—for courses in commonly taught languages like English, Chinese, French, or Spanish. Nor do data exist on the linguistic knowledge implicated in the global ratings on the various proficiency scales increasingly widely used to measure current learner abilities.
In many languages, with larger or smaller native speaker populations, commonly taught or not, there is an urgent need to develop new speakers and to raise the ability levels of existing ones. To give some idea of the problem, the typical American language and literature major is lucky to achieve level 2 on the Interagency Language Roundtable (ILR) scale after four years of study. Median attainment after four years of harder languages for English-speaking adults, such as Chinese, Arabic, Russian, Korean, or Japanese, is only ILR 1, with attrition typically setting in quickly thereafter. The U.S. government, however, estimates that ILR 3 is the minimum proficiency level required for most employees to be able to perform their work satisfactorily in a foreign language (Brecht & Rivers, Reference Brecht and Rivers2000, Reference Brecht, Rivers and Long2005). It is clear that there is a massive language gap to be bridged, even among graduating language majors who enter private industry or government service, and the problem is compounded by the fact that most business experts and government employees are not language majors in the first place. It would be very valuable for teachers, textbook writers, testers, and the learners themselves to know precisely what is learnable, and therefore teachable, at a given stage in development. Additionally, it would be valuable to know what is required of the individual student and, more broadly, of the general student population to move from one proficiency level to another in a given language, and how the picture differs for heritage and nonheritage learners.
Important practical applications aside, the whole notion of proficiency in a second language (L2) has long been contested in the SLA and applied linguistics literature. SLA theorists, such as Pienemann (Reference Pienemann, Hyltenstam and Pienemann1985), have questioned its utility, arguing that it is an amorphous construct, lacking operationalization and, possibly, operationalizability, whereas learners’ current stage and subsequent progress on linguistically represented developmental indices is psycholinguistically meaningful and objectively measurable. Similar arguments have been made for more than 20 years in the language testing literature, where reliability and validity problems—with the measures employed to assess learners’ abilities and with the scales themselves—have been noted by many researchers (see Alderson, Reference Alderson2005, Reference Alderson2007; Bachman, Reference Bachman1988; Fulcher, Reference Fulcher1996, Reference Fulcher2004; Hulstijn, Reference Hulstijn2007; Lantolf & Frawley, Reference Lantolf and Frawley1985, Reference Lantolf and Frawley1988, Reference Lantolf and Frawley1992; Pienemann, Reference Pienemann, Hyltenstam and Pienemann1985; Pienemann, Johnston, & Brindley, Reference Pienemann, Johnston and Brindley1988). A demonstration of relationships, if any, between L2 learners’ development of control of grammatical features and the same learners’ advances in global proficiency ratings would be of interest to theorists, researchers, and practitioners alike.
The Proficiency Problem
Despite critiques by SLA theorists and well-known reliability and validity problems, global ratings of overall second language abilities in terms of steps on various L2 proficiency scales are used in many countries by large public and private employers, such as militaries (Lett, Reference Lett and Long2005), intelligence agencies, and corporations, as well as by academic gatekeepers. Among the best known are the ILR and ACTFL (American Council for the Teaching of Foreign Languages, 1985) scales in the United States, the Canadian Benchmarks (Pawlikowska-Smith, Reference Pawlikowska-Smith2000), the International Second Language Proficiency Ratings (Wylie & Ingram, Reference Wylie and Ingram1999) in Australia, and the Common European Framework of Reference for Languages: Learning, Teaching and Assessment (CEFR; Council of Europe, 2001) in Europe. Proficiency ratings are routinely accepted, be it rightly or wrongly, in some quarters as a rough estimate of an applicant’s ability to perform a job, as a partial basis for assigning an individual to a particular rung on a pay scale, and as a criterion used to group learners for instructional purposes or to exempt them from instruction altogether.
Although such scales are attractive to end users in part because of their functional orientation, because of their (apparent) simplicity, and because ratings—usually encoded in single phrases or global numeric values—can ostensibly be calibrated, equated, and compared, they also have disadvantages. Most scales are not empirically based but, rather, depend on the intuitions of teachers or experienced testers. Proficiency ratings found in guidelines and manuals are not based (or are minimally based) on linguistic criteria (e.g., accuracy, syntactic complexity, control of specific linguistic elements, lexical depth, or collocational appropriateness), but rather are based on either functional criteria (e.g., the individual can linguistically handle a situation with complications or can give directions) or discourse structure (e.g., the individual speaks in paragraphs or uses extended discourse). In either case, they rely on the subjective judgments of testers who ultimately decide whether or not a task has been successfully completed.
From a language-learning or language-teaching perspective, scales and ratings based on the scales are unhelpful. Numerical ratings, like 1+ or 2, letter grades, like B+ or A−, or opaque descriptors, like Threshold and Vantage or Advanced Low or Intermediate High, may mean something to the insiders that administer the tests, but they mean little or nothing to either the learners who need to improve their ratings or their teachers (unless they are trained testers themselves). They are proficiency scales, after all, and are not intended to provide diagnostic information.
When a student is told that he or she is a 1+ in Arabic, Chinese, Russian, or Spanish on the ILR scale, for example, and is given a certain period of time to improve, what precisely does this student and his or her teacher need to work on to reach level 2, 2+, or 3? What sorts of phonological problems, grammatical constructions, vocabulary items, collocations, or registers must be mastered at each level? And of potentially still wider interest, do the same or similar features determine ratings across languages? Do heritage and nonheritage learners exhibit comparable profiles and share the same linguistic problems within a given proficiency range? Which linguistic features does a rating imply that a learner controls, in perception or production, and which ones remain problematic? Which ones need to be included in a syllabus and pedagogic materials for learners in the 1+ to 2 or any other range in language X, in typologically similar languages, or in all languages? In short, what are the linguistic correlates of proficiency?
One potential answer to the question lies in the intuitions of experienced classroom teachers. Unfortunately, however, intuitions differ; beginning teachers are inexperienced, and experienced ones are particularly few and far between, as are pedagogic materials, when it comes to LCTLs, especially at advanced levels. Experienced and inexperienced teachers, alike, express a desperate need for solid information on the appropriate content of courses for the languages for which they are responsible. Impressionistic judgments, most would agree, are no more justified as a substitute for empirical findings in language teaching than in mathematics or science courses, or in the provision of other important services, from architecture to medicine.
An alternative solution might be to refer to the published descriptions that accompany global proficiency ratings. This turns out to be of limited help as well, because the descriptions are vague and involve a miscellany of overlapping linguistic criteria, nonlinguistic abilities and subabilities, functions, tasks, and skills—the linguistic correlates of which are all unknown. To compound the problem, the ability to do X or Y (hold a casual conversation, deal with routine workplace language needs, etc.) often seems to have been arbitrarily assigned to a particular level on a scale; these tasks clearly have relevance at all levels, depending on the complexity of any particular instance, and can be performed by learners at all levels, albeit with greater or lesser success. In other words, the scales’ functional correlates lack face validity.
Moreover, these scales are marked by a pervasive use of undefined terms and expressions, open to widely varying interpretations. To illustrate, it is necessary to consider the ILR description of speaking abilities at ILR 2.
Speaking 2 (Limited Working Proficiency) Able to satisfy routine social demands and limited work requirements. Can handle routine work-related interactions that are limited in scope. In more complex and sophisticated work-related tasks, language usage generally disturbs the native speaker. Can handle with confidence, but not with facility, most normal, high-frequency social conversational situations including extensive, but casual conversations about current events, as well as work, family, and autobiographical information. The individual can get the gist of most everyday conversations but has some difficulty understanding native speakers in situations that require specialized or sophisticated knowledge. The individual’s utterances are minimally cohesive. Linguistic structure is usually not very elaborate and not thoroughly controlled; errors are frequent. Vocabulary use is appropriate for high-frequency utterances, but unusual or imprecise elsewhere. (Interagency Language Roundtable, n.d.)Footnote 1
Similar problems can be found in the descriptions for all four skills, at all 10 proficiency levels and sublevels, on the ILR scale, and in the comparable descriptions of levels of all such scales.
Additionally, the characterizations are sometimes so vague and general as to require considerable imagination on the reader’s part. For example, the description of CEFR level B1 (Threshold) reads as follows:
B1: Can understand the main points of clear standard input on familiar matters regularly encountered in work, school, leisure, etc. Can deal with most situations likely to arise whilst travelling in an area where the language is spoken. Can produce simple connected text on topics which are familiar or of personal interest. Can describe experiences and events, dreams, hopes & ambitions and briefly give reasons and explanations for opinions and plans. (Council of Europe, n.d.)
“Clear standard input on familiar matters regularly encountered in work, school, leisure, etc.” and “situations likely to arise whilst travelling in an area where the language is spoken,” for example, can obviously mean very different things to different people and in practice. Some notions of linguistic competencies in terms of grammatical accuracy, phonological control, and vocabulary range and control are provided along with the CEFR proficiency scale (Council of Europe, 2001), but these are very general correlates of the CEFR levels and are not language specific.
The CEFR scale was developed somewhat differently from other proficiency scales. Functional descriptors (e.g., “can produce stretches of language with a fairly even tempo” and “can define the features of something concrete for which he/she can’t remember the word,” http://www.coe.int/lang) were culled from various existing scale descriptors and were empirically scaled by North and colleagues (North, Reference North2000; North & Schneider, Reference North and Schneider1998) based on data from surveys and video assessments collected from nearly 300 teachers. Although an improvement over scales whose descriptors are ordered and leveled on the basis of pure intuition, the study of teachers’ judgments still does not capture the kind of detailed information on structural linguistic competencies based on learner performance that the Linguistic Correlates of Proficiency (LCP) project targeted.
In a sense, the fact that the scales are couched in quasi-communicative terms is a positive development. The recognition that languages are learned to be used—at least where adults are concerned—often for important, quite specialized purposes, also reflects a change in the increased popularity of communicative approaches to language teaching and the proficiency movement since the 1960s (Higgs, Reference Higgs1984; Musumeci, Reference Musumeci, Long and Doughty2009). To describe points on a proficiency scale in terms of a learner’s ability to produce utterances containing verbs marked for simple past tense, “understand utterances containing the third conditional” (North & Schneider, Reference North and Schneider1998), and so forth, with no reference to the speech acts, topics, tasks, or situations in which use of the grammatical features occurred, would be even less informative, after all, and reflect a synthetic, noncommunicative approach to language teaching and testing unsupported by theory and research findings in SLA (Long, 1991, Reference Long2007, Reference Long, Long and Doughty2009). Either forms or uses alone are inadequate; what is needed is a combination, with linguistic features linked systematically to development described in terms of some kind(s) of functional abilities.
The goal of the LCP project was to provide empirically based checklists of linguistic correlates of ILR proficiency, and thereby to increase the transparency and utility of ratings and, ultimately, improve the teaching, learning, and testing of LCTLs. The checklists would detail much of the control of linguistic elements needed to move from one level to another on a proficiency scale and would provide syllabus designers, materials writers, teachers, and learners with the information they need. The immediate goal was proof of concept: to determine the feasibility of identifying control of linguistic features, in perception and production, implicated in progress within a relatively narrow proficiency range (ILR 2, 2+, and 3), across groups of learners, and in a demonstration language (Russian). The research would not itself produce diagnostic tests with the psychometric properties required by reliable and valid measures, but it would subsequently be a relatively straightforward undertaking to produce such measures out of the data-collection batteries developed for the study.
When designing the study, the researchers were mindful of the need pointed out by Bachman (Reference Bachman1988) to avoid the circularity of defining more proficient speakers as those who can perform more demanding tasks, and defining more demanding tasks as those that can be performed (only) by more proficient speakers. The study was not designed to establish which tasks or task types are better handled by learners of higher or lower global proficiency, nor were learners assessed in terms of their ability to perform well on the tasks, per se. What was measured was their control of linguistic features; the tasks were simply carriers. Some of the same tasks (translation, elicited imitation, etc.) were used for different target features and in different linguistic domains (morphology, collocations, etc.). The same learners might exhibit poor control of some features assessed using a translation task, for example, but acceptably high control of other features assessed via the same or another translation task. Thus, the focus throughout was on control of linguistic elements, not success or failure with particular tasks or task types.
It should also be noted that oral proficiency on the ILR scale is assessed via the Oral Proficiency Interview (OPI) globally, without any reference to the accuracy of individual features that occur in a speech sample. Additionally, given the well-attested avoidance strategy in L2 performance, the absence of a feature in a L2 learner’s output may itself be indicative of underlying difficulty, although some features, such as the use of case marking in Russian, are impossible to avoid. In this study, correlates can consequently be interpreted as correlates of proficiency, not aspects of the criteria that proficiency raters are (impressionistically) using.
Previous Research
The problems that have been outlined have inspired at least one other major attempt to deconstruct global proficiency ratings: the project for the development of diagnostic language tests (DIALANG) (Alderson, Reference Alderson2005; Alderson & Huhta, Reference Alderson and Huhta2005). DIALANG is a free and open (i.e., publically funded and not secure), Internet delivered, and no-stakes or low-stakes test, of which the volunteer test taker can opt out at any time. DIALANG targets 14 European languages, with results linked to the six levels on the CEFR scale. Crosslinguistic comparability is desirable, but hard to achieve. The test begins by asking learners to assess their own proficiency via a 75-item vocabulary test (50 real words, 25 pseudo-words), in which learners are required to identify which are real words and which are not. Either alone or combined with the learner’s (optional) responses to a set of 18 can do statements for each of four language skills, the instantaneously scored and communicated results guide the learner to the appropriate level of the main test—easy, medium, or difficult—which covers listening, reading, writing (indirectly), grammar, and vocabulary. There is no speaking component.
The DIALANG tests are organized in terms of subskills. For example, in the area of reading, one is tested on the ability to distinguish a main idea from supporting detail, the ability to understand text literally, and the ability to make appropriate inferences; all are reported separately, not as an integrated score. Four types of test items are employed: multiple-choice questions, drop-down menus, text entry, and short-answer questions. Test results are delivered to learners in terms of one of the six CEFR proficiency levels, not as test scores, along with differences between their self-assessed rating and their actual test performance if they completed the optional can do and advisory feedback sections. The advice on how to progress to the next CEFR level, again in terms of subskills, takes the form of tables that show the differences between what learners at the test taker’s current level and at the preceding and following levels on the CEFR scale can supposedly do, along with the kind of texts they can handle, and under what conditions, or with what limitations.
Some of DIALANG project director Alderson’s (Reference Alderson2005) conclusions are sobering. Of particular importance for the present study is the fact that proficiency-type items (e.g., reading subskills) do not pattern with CEFR level and do not get at what underlies proficiency:
The results of the piloting of the different DIALANG components showed fairly convincingly that the subskills tested do not vary by CEFR level. It is not the case that learners cannot inference at lower levels of proficiency, or that the ability to organize text is associated with high levels of proficiency only. This appeared to be true for all the subskills tested in DIALANG. If subskills are not distinguished by CEFR level, then one must ask whether the development of subskills is relevant to an understanding of how language proficiency develops and hence to diagnosis. . . . The possibility is worth considering that what needs diagnosing is not language use and the underlying subskills, but linguistic features, of the sort measured in the Grammar and Vocabulary sections of DIALANG. (Alderson, Reference Alderson2005, p. 261)
Both the proficiency levels in the CEFR scale and assignment of items in the diagnostic tests to levels are based on expert judgments of difficulty—that is, the intuitions of experienced teachers and testers of the languages concerned—not on data on the acquisition of those languages. Empirically based acquisition data, in the form of inventories of objectively recognizable linguistic features, is the desirable approach and the one followed in the LCP project. Because relationships between mastery of such features and proficiency levels are unknown, a priori assignment of features or proficiency subskills to levels is unjustified. Charting mastery of sets of features relative to one another is, however, the first order of business, and the relationships of feature mastery to proficiency scale levels are determined by the data, not by intuition or a priori standard settings, even when, as in the DIALANG project, acceptably high interrater reliability agreement was obtained among the judges of item difficulty. To set the standard on the basis of the judgment of item raters, however expert, means that one set of intuitions—the original proficiency scales—is being equated with another set of intuitions, when what is needed is an empirical basis to both. In the LCP project, the purpose of the research is to unpack levels on the ILR scale and to empirically determine the degree of control of linguistic features, linguistic constructions, and the other objectively measurable holistic abilities (e.g., the ability to comprehend speech in noise) that differentiate them. Subsequently, it would also be very useful to know which classes of problems occur across languages at these levels, and which tend to be specific to particular typologically defined groups of languages. Such information would be of considerable help to those involved with languages for which no tests are available at advanced levels (or at all), yet for which rapid ramp-up may be required.
A second research effort of relevance to this work, also targeting the CEFR, is described briefly by Hulstijn and Schoonen (Reference Hulstijn and Schoonen2006):
It must be borne in mind that the scales of the CEFR are to a large extent based on the intuitions of language teachers about descriptors derived from a number of existing widely used scales of language proficiency. What the CEFR does not indicate is whether learner performance at the six functional levels listed in Chapter 4 of the CEFR (Council of Europe, 2001) actually does exhibit the linguistic characteristics listed in chapter 5, and, more specifically, which linguistic features are typical of each of the levels. Answers to these questions are essential if the CEFR is to be successfully implemented across Europe. (p. 7)
This project proposes to focus on seven target languages: English, Finnish, French, Swedish, German, Italian, and Dutch. The aims are
1. To find out what learners can and cannot do in all four skills (listening, speaking, reading, and writing) at each of the six levels in the CEFR scales, which requires administration of tasks of a close format (not just “free” speech in an oral interview, for instance) and the measurement of both accuracy and reaction times
2. To develop diagnostic tools
3. To identify commonalities and differences in the linguistic (and paralinguistic) profiles of naturalistic and instructed learners
The clearly stated intention is to focus on empirical studies of learner performance at the various CEFR levels, although results are not yet available. The report’s title, Scientific Report of ESF-Sponsored Exploratory Workshop: Bridging the Gap between Research on Second Language Acquisition and Research on Language Testing, demonstrates precisely the orientation of the LCP project. The most obvious difference is that whereas the CEFR is a relatively new kid on the block, the ILR scale has been used for many years and is a widely accepted proficiency yardstick in most branches of the U.S. government, and through the related ACTFL scale, in academia and elsewhere.
THE LINGUISTIC CORRELATES OF PROFICIENCY PROJECT
In light of these considerations, the LCP project was undertaken using Russian as the demonstration language. The purpose was to identify linguistic features implicated in progress on the ILR scale—specifically, from ILR 2 to 2+, and from 2+ to 3—and the project ultimately aimed to improve the teaching, learning, and testing of high-level listening and speaking abilities. The study focuses on the 2–3 range because of a U.S. government initiative that sought to raise the proficiency level—from 2 to 3—of the tens of thousands of employees for whom command of a foreign language is considered critical for job performance. The research was intended to demonstrate proof of concept; that is, that despite idiosyncrasies at the level of individuals due to the well-documented substantial variability in interlanguages, it is possible to specify the linguistic features implicated at various proficiency levels across groups of learners. If successful, the same methodology could then be applied to other proficiency ranges, as well as to new languages. Given these idiosyncrasies, it was considered useful if the same battery of measures developed to define LCP could also be used to provide feedback to individual learners (and their teachers) on their particular abilities and problems.Footnote 2
For the purposes of the study, the products of such work, LCP, or linguistic profiles, are detailed inventories of linguistic features—phonology, morphology, syntax, vocabulary, and collocations—in perception and production, and of abilities such as comprehension of speech in noise, that are typically mastered by, or still posing problems for, adult English-speaking learners of foreign languages at various separately assessed levels on a proficiency scale.
Although both the DIALANG and LCP projects are diagnostic (see Kunnan & Jang, Reference Kunnan, Jang, Long and Doughty2009), they differ in a number of ways. Perhaps the most crucial difference is that whereas DIALANG focused on subskills (e.g., the ability to distinguish a main idea from supporting details), the LCP project focuses on language descriptors. The findings of the LCP project make it possible to follow the development of individual features. For some features tested, an acquisition point was established for a given testing modality (perception or production) that used the traditional 80% accuracy threshold in SLA research. Sensitivity to violations in the appropriate use of verbs of motion does not, for example, develop even in ILR 3 learners, whereas sensitivity to violations in derivational morphology (illegal derivations) develops at ILR 2+. Any given ILR proficiency can be characterized by a certain level of attainment for a set of features. The study’s findings for Russian illustrate the typical errors in feedback to learners and teachers:
1. ILR 2: Can deal with low noise level with 80% accuracy only in supporting contexts, but not in low-predictability contexts (60% accuracy).
2. ILR 2+: Can conjugate real and nonce regular default -aj- class verbs with 80% accuracy but has difficulty with irregular -a- verbs, with accuracy only at 20%.
3. ILR 3: In subjectless sentences, is sensitive to mismatches in case in impersonal sentences with 80% accuracy but is only sensitive to mismatches in verb agreement with the expletive subject with 50% accuracy.
Several other features differentiate the two projects. Whereas DIALANG focuses on listening, reading, and writing, the LCP project focused on spoken production and perception only. Because this was a feasibility study, and because some end users of other LCTLs—such as Chinese—might not possess or be interested in L2 literacy, no use of the L2 writing system was allowed in the computer-delivered test batteries themselves (although non-Roman alphabets would eventually be usable in feedback to teachers and some individual learners). Whereas DIALANG focuses on 14 languages, some commonly taught, some not, the LCP project targeted three proficiency levels in just one, Russian. The aim was to determine whether fine linguistic distinctions could be made reliably within a relatively narrow proficiency range, as opposed to the entire proficiency range from beginners to near-native, targeted in the DIALANG work. Whereas several of the 14 DIALANG languages have substantial corpora, expertise in SLA, and well-documented language teaching and language testing behind them, relevant literature on Russian was comparatively rare. Thus, the project team, when generating initial sets of items, was forced to rely on a mix of introspections and retrospections by advanced learners and experienced teachers, on linguistic analyses in books and journals, on the relatively few available SLA studies of the acquisition of Russian, and on inferences drawn from better-studied L2s, like English, German, Spanish, and Japanese, as to what might be worth probing.
There were important methodological differences as well. DIALANG, for example, used expert judgments to assign items to this or that CEFR proficiency level, but the LCP project was performance based; that is, it focused on what learners can or cannot do at a certain ILR proficiency level. This approach is labor intensive, but it avoids the potential circularity of raters classifying linguistic features according to the use they make of them in assigning proficiency ratings when administering the OPI or ACTFL interview. Additionally, DIALANG employs four main types of test items: multiple-choice questions, drop-down menus, text entry, and short-answer questions; the LCP project experimented with more than 20 task types, some familiar in L2 assessment, others drawn from research in SLA involving very advanced learners (e.g., work on the critical period hypothesis) and in experimental L1 psycholinguistics.
THE STUDY
The two most significant challenges to increasing the talent pools in foreign languages, especially LCTLs, are that (a) L2 learners often find functional proficiency at level 2 and above on the ILR scale difficult to achieve, and (b) most teachers, testers, and learners lack anything more than intuitive knowledge of the level of control of the linguistic features—phonology, morphology, syntax, lexis, and collocations—implicated in progress on the ILR scale, particularly ILR 2, 2+, and 3. The intuitions of experienced professionals are often sound but sometimes are not, and in any case constitute an inadequate basis for such an important aim to meet what are often critically important foreign language needs.
The LCP project examined the linguistic features of the interlanguages of English-speaking learners of one LCTL, Russian, in the ILR 2–3 proficiency range. The study was guided by two principle research questions:
1. Which linguistic features of Russian correlate with ILR proficiency levels 2, 2+, and 3 on the ILR scale?
2. At what level of control do these linguistic features correlate with ILR proficiency levels 2, 2+, and 3 on the ILR scale?
Through an instrumentation study conducted in 2007, in addition to testing the viability of a battery of perception and production tasks, the research team had tentatively identified a set of linguistic features of Russian in the phonological, morphological, syntactic, lexical, and collocational domains relevant for progress by learners and heritage speakers in the ILR 1+ to 4 proficiency range. The main study employed a larger subject pool within the narrower ILR 2–3 range, along with a revised version of the data-collection battery (improved through standard item analyses, etc.), which again consisted of a number of tasks that tapped both perception and production abilities. The production involved was very limited and amounted to just that which was required to assess active control of the targeted features. Each of the 33 tasks in the revised battery was designed to probe knowledge of one or more features. Relationships between linguistic features and the ILR proficiency levels of interest, which emerged from the first study, would be confirmed, or not, as the case may be.
METHOD
Instrumentation: Task Types, Tasks, and Items
The design of instruments that measure L2 abilities becomes harder with each proficiency increase, especially when LCTLs are involved. Due to options in topic control and avoidance—more skillfully exploitable by advanced learners—such relatively open-ended procedures as oral interviews and picture descriptions tend to elicit samples of what speakers can do and are less useful for finding out what they cannot do. This means that a satisfactory data-collection battery or diagnostic test, especially one suitable for use with advanced learners, will need to include a number of closed tasks designed to probe for the small, often unnoticed flaws in advanced language proficiency, with avoidance precluded by the nature of the elicitation tasks. Discrete-point tasks of several kinds will be needed to identify gaps in learners’ linguistic repertoires, and to measure accurate and appropriate usage and understanding in various phonological, morphological, syntactic, lexical, and collocational domains. Several task types with proven track records in SLA research (grammaticality judgment, elicited imitation, etc.) were employed again in the LCP research, although a number of task types were imported from other fields, notably first language (L1) research in psycholinguistics, and adapted for use with adult L2 learners where necessary. A full description of, and rationale for, each of the 38 task types trialed in the instrumentation study is beyond the scope of this article but is available in Long et al. (Reference Long, Jackson, Aquil, Cagri, Gor and Lee2006).
There were additional restrictions on what was possible with regard to instrumentation. The data-collection batteries created for the LCP research, like any diagnostic tests that might eventually be based on them, were required to be deliverable via computer, so as to facilitate their use by students and language testers in some situations (e.g., distance learning). The batteries could employ written English stimuli (e.g., in a translation task) but had to avoid use of written forms of the target language, given that some learners and test takers at lower proficiency levels might only need, and only possess, functional ability in listening or speaking. The emphasis, therefore, was on combinations of English visual and Russian oral stimuli.
Just as innovations would be required to develop the task types suitable to investigate and assess advanced foreign language proficiency, so would creativity be needed in selection of the linguistic domains and features to be targeted. The most comprehensive diagnostic studies of advanced proficiency when the project began had dealt with two East Asian languages, Japanese and Korean, with inevitable biases in the choice of linguistic features included and excluded in their assessment. Kanno, Hasegawa, Ikeda, Ito, and Long (Reference Kanno, Hasegawa, Ikeda, Ito, Long, Brinton and Kagan2007) and Lee, Kim, Kong, Hong, and Long (Reference Lee, Kim, Kong, Hong and Long2005) had tested performance on honorifics and mimetics, for example, features absent in Indo-European languages.Footnote 3 There is a need for both general and language-specific approaches in the search for LCP levels. A major goal of the LCP project was to develop instruments that included both types of features and were broad in spectrum and flexible in their application to individual languages.
The global nature of ILR OPI testing precludes the use of any nonfunctional descriptors, and consequently, no lists of language-specific grammatical difficulties that correspond to the ILR levels are available. A potential option, a so-called noncompensatory core of grammatical features (e.g., basic use of inflections in cases, verbal conjugation, and verbal aspect), is not employed in ILR testing so could not be used either. Several alternative sources of ideas were consequently tapped when items for the pilot version of the data-collection battery for the project were developed. They included the reflections of experienced teachers of Russian, introspections (obtained through interviews) of high-achieving learners of the language concerning residual linguistic difficulties, Russian corpora, Russian textbooks for advanced learners, extensive consultations with Russian language instructors in the academic and government sectors, research findings on late-acquired items in L1 acquisition and late-acquired or rarely acquired features in adult L2 acquisition, and, because of its frequent focus on very advanced learners, the literature on critical periods in SLA, although most of that work concerns English. The features selected for testing were not linked explicitly to descriptors of the ILR scale, of course, because prior to this study, no such linkage had ever been established.
The pilot version of the Russian data-collection battery consisted of 38 tasks—phoneme production, AXB discrimination, accent detection, speech in noise, grammaticality judgment, lexical decision with and without priming, multiword unit completion, and so forth—with between 10 and 400, although usually 20–50, items per task. Considerable care was taken to control such potentially confounding variables as utterance length, lexical frequency, and structural complexity.Footnote 4 The battery consisted of four blocks, each of which was approximately 1 hr in length and contained speech perception or production tasks in the following sequence: listening, speaking, listening, and speaking. The four blocks were separated by pauses that allowed subjects to take short breaks. Four different versions of the test battery were created to balance different conditions, which varied from task to task, and the oral input of the whole battery included alternating male and female voices.
The 2006 instrumentation study obtained data from a total of 70 participants: 36 learners of Russian and 24 heritage speakers—with ratings that ranged from 1 to 4+ on the ILR scale (39 in the 2–3 range)—as well as 10 L1 Russian speaker controls. Participants were recruited through newspaper advertisements and by word of mouth. All were college educated and ranged in age from 18 to 56. Standard item analyses and additional statistical analyses (Cronbach’s alphas and ANOVAs) showed that most tasks had acceptably high internal reliability, that students at different levels on the ILR proficiency scale differed in their scores on most tasks, and that significant effects for ILR level were obtained on some tasks in the lower proficiency ranges, some in the higher ranges, and some across the entire range.Footnote 5 These results had been obtained despite the fact that the data were collected (a) from a truncated sample in terms of proficiency (most subjects were in the ILR 2–4 range), thereby reducing the scope of variation possible and the likelihood of obtaining statistically significant findings; (b) sometimes with fewer items than could easily be included in the future if a task were retained (e.g., because of the rigorous native speaker acceptance criterion for item retention—no more than two L1 speakers could disagree on an item for it to be retained—a number of items in some tasks had to be discarded); and (c) with data missing as a result of an occasional technical failure in the recording process.
The main study employed a revised and rerecorded version of the data-collection battery and was conducted in 2008. Of the original 38, 30 tasks—improved as a result of the item analyses or by creating additional items—were retained. Three additional elicited imitation (EI) tasks were added, for a total of 33. Each of the 33 tasks tested one feature (often with several subfeatures), and most tasks were used to test a variety of linguistic features.Footnote 6
Participants and Procedures
Within the narrower ILR 2–3 proficiency range, a larger subject pool was employed than in the instrumentation study. Data were collected from a total of 68 participants: fifty-seven learners (21 at ILR 2, 18 at 2+, and 18 at 3), all of whom were also tested for oral proficiency via an ILR oral proficiency telephone interview conducted by certified testers, and 11 L1 speaker controls (see Table 1). For all the analyses, only the learners’ data are presented, because the L1 speakers were relevant only as controls to check that the tests operated as expected. For the linear discriminant analyses (LDAs), one ILR 2 participant (female) was dropped from the analysis for missing several of the tests due to technical problems. Potential participants were recruited via newspaper advertisements, through word of mouth, and from among faculty and students in Russian programs at two American universities. They were then screened into the study on the basis of their performance in an OPI telephone interview by a certified tester. The data-collection battery, programmed using the DMDX software package, was administered via computer in four 1-hr blocks, with short breaks between blocks.Footnote 7
Russian participant distribution by age, gender, and proficiency
Note
OPI = Oral Proficiency Interview.
ANALYSES
Error rates were analyzed using generalized linear mixed models with a logit link function (henceforth logit mixed models). These models allow for important random effects, such as subject and item, to be modeled, unlike ordinary logistic regression, and they are better suited to analyze categorical data, such as error rates, than repeated-measures ANOVA (see Agresti, Reference Agresti2002, and Jaeger, Reference Jaeger2008, for discussion and further references). Models were fitted in the statistical platform R (R Development Core Team, 2008), using the lmer function from the lme4 library (Bates & Sarkar, Reference Bates and Sarkar2007). Model selection and testing were performed such that ILR level was always retained in the model (to test the hypothesized effects) and was treated as a categorical, ordinal variable, using different contrast codings to test the three comparisons shown in the summary chart (ILR 2 vs. 2+, 2+ vs. 3, and 2 vs. 3).
Because many tasks included one or more fine-grained factors in the design, model comparison was used to trim unnecessary predictor variables. Model comparison was also used to evaluate the inclusion of random effects of subject and item on both intercept and slope of the other variables in the model.Footnote 8 Following Pinheiro and Bates (Reference Pinheiro and Bates2000), models were compared using χ-square likelihood ratio tests (a complete model of the results is available on request from the author), and predictors were only retained when these tests reached significance at p < .05. The result was that many of the fine-grained distinctions designed into the tests did not reach significance and thus were not part of the final models. For each of the final models, ILR rating was used to predict accuracy in order to test whether there is some relationship between general proficiency and test performance.
LDAs were carried out as a complement to the logit mixed model analyses, in order to assess how successfully these measures correctly classify participants into the ILR levels determined by the OPI. In general, tests’ accuracy scores were moderately to highly correlated with one another; thus to enter in large numbers of tests as predictors would run afoul of multicollinearity problems. Out of the 465 unique pairwise correlations between measures, 164 were correlated at r > 0.50. The pairwise correlations of more than 0.70 are listed in Table 2.
Pairs of tests correlated at r = 0.70 or greater
Because of these relatively high levels of correlations between tests, LDAs were built to start with single predictors, rather than with entire sets of tests, and predictors were added only when classification accuracy improved. Three separate LDAs were performed, for three different binary contrasts: classification of ILR 2 versus ILR 3, ILR 2 versus ILR 2+, and ILR 2+ versus ILR 3. Finally, in order to assess internal reliability, Cronbach’s alpha was computed for each test.
RESULTS
Overall, the results indicated that every test except for one—picture-word elicitation, a phonology production test—showed significant differences between ILR levels 2 and 3 at p < .05. Out of 33 tests, 14 tests showed significant differences between ILR levels 2 and 2+, and 21 showed significant differences between ILR levels 2+ and 3. The perception tests that showed significant effects are summarized in Table 3, and the production tests are summarized in Table 4. Complete tables of detailed results for all the tests are available on request from the author.
Perception tests that demonstrated significant effects of ILR level on accuracy at p < .05
Production tests that demonstrated significant effects of ILR level on accuracy at p < .05
The values of Cronbach’s alpha for the majority of the production tasks are in the high range (0.711–0.978, M = 0.873, Mdn = 0.899), which indicates high internal consistency. Cronbach’s alpha values are typically lower for the perception tasks (0.261–0.925, M = 0.693, Mdn = 0.728), and for tasks with smaller numbers of items. Production tasks tend to differentiate one more sublevel contrast, in addition to levels 2 and 3, whereas some of the perception tasks do not differentiate, or only marginally differentiate, sublevels within the 2–3 range. These results, taken together, suggest that the production data are more efficient and reliable. However, given that production data need a human scorer and are therefore more resource consuming, a trade-off of higher reliability and lower cost-effectiveness should be considered in the future development of similar diagnostic tasks.
A comparison of accuracy scores in the perception and production tasks confirms the well-documented observation that accuracy in production experiments typically lags behind accuracy in perception. Additionally, the fact that some tasks were used with different features makes it possible to directly compare learners’ control of the features in question. For example, in three EI tasks, participles, verbal adverbs, and different types of subordinate clauses were targeted within exactly the same testing format. Sentence length and target structure position—participle, verbal adverb, and conjunction—were strictly controlled, and vocabulary from the same frequency ranges was used. The accuracy scores show that participles and verbal adverbs were considerably more difficult for speakers in the ILR 2–3 proficiency range than subordinate clauses. Comparison of the different subordinate clause types within the same EI task shows that, at the same time, the unreal condition presented more difficulties for level 2 and 2+ learners than the other types. Finally, of all the domains included in the battery, accuracy scores on tasks that tested control of some kinds of idiomatic language (e.g., collocations, idioms, proverbs, and sayings) were the lowest. This finding supports the general notion that idiomatic use remains a learning objective for highly proficient L2 learners at ILR level 3 and above.
Even though the individual test results of the logit mixed models are informative, LDAs provide estimates of how effectively the tests classify subjects into appropriate ILR levels. In these analyses, only overall accuracy within each test was analyzed; the fine-grained conditions were collapsed. Single-predictor LDAs for each of the 33 tests were used to calculate base rates of classification accuracy, defined as the number of correct classifications divided by the number of subjects. For all of the analyses, the reported classifications are based on leave-one-out cross-validation. The result is a more conservative, robust estimate of the classification accuracy of the model. The tests that resulted in a classification of 0.7 or better on their own in cross-validation are listed in Tables 5–7, for each of the ILR contrasts.
Tests that classified ILR 2 vs. ILR 3 at better than 0.70 accuracy
Note
ILR = Interagency Language Roundtable.
Tests that classified ILR 2 vs. ILR 2+ at better than 0.70 accuracy
Tests that classified ILR 2+ vs. ILR 3 at better than 0.70 accuracy
Multivariate LDAs were constructed by starting with the predictor that had the best classification on its own (i.e., the top in each of Tables 5–7, respectively) and then adding additional predictors one at a time, which were only retained in the model if they improved the classification accuracy of the cross-validated LDA. Three models ultimately emerged as among the best possible classification models for this set of data. Note that these are not the three uniquely best models, and that it is possible that some tests could be substituted for these same tests and still achieve similar classification results. These models simply illustrate the most one could reasonably expect from this battery of tests in terms of overall classification accuracy. The models for the three ILR contrasts are given with the classification tables for each contrast in Tables 8–10.
Number of actual vs. predicted ILR levels for the best LDA model predicting ILR 2 vs. 3, with predictors of tests 6, 7, and 31 (collocations/proverbs, verbal prefixes, participles)
Number of actual vs. predicted ILR levels for the best LDA model predicting ILR 2 vs. 2+, with the predictors of tests 6 and 18 (collocations/proverbs, reflexives)
Number of actual vs. predicted ILR levels for the best LDA model predicting ILR 2+ vs. 3, with the predictors of tests 3 and 31 (nonimperative imperative constructions, participles)
These results show that even with a small fraction of the measures tested in this project, classification rates are quite high. The especially high accuracy of classification between ILR 2 and 3 is somewhat expected, given that this represents a very substantial difference in overall proficiency. More interesting is that a small number of measures were also able to classify ILR 2 versus 2+ at 76%, and ILR 2+ versus 3 at better than 80%. This distinction is especially slippery in the ILR system, and 2+ is often characterized as being almost 3 rather than representing some well-defined position between 2 and 3, so the level of success in classification by these measures is interesting and somewhat surprising. Additionally, it is interesting that the set of measures that discriminates best between 2 and 2+ is distinct from the set that discriminates 2+ and 3. These results, at this level of granularity, make a strong case for the viability of this kind of research program, designed to flesh out the fine linguistic details of progression through a proficiency scale such as the ILR, and the potential utility for such measures to augment current testing paradigms.
Despite the robustness of these findings under a cross-validated analysis, it should be noted that, given the relatively small sample size and the fact that the large selection of measures may allow for capitalization on variance—which may be unique to this sample—the study is careful not to overinterpret these results. There is clearly strong potential to add valuable information to what is known about learners’ abilities at different ILR levels, and it is clear that even a relatively small number of such measures can effectively discriminate learner groups. To be more confident that the particular measures that are successful in this study represent the most stable and reliable discriminators at the linguistic construct level or at the task-type level, further replication and validation is needed.
DISCUSSION AND CONCLUSIONS
Having been refined by item analyses and other operations since the first round of data collection and analysis, 32 of the 33 tasks in the data-collection battery (18 in perception and 14 in production) significantly differentiate ILR proficiency levels 2 and 3. This is the case despite the truncated global proficiency range in focus. This suggests that the data-collection battery is sensitive and psychometrically sound. For perception, 9 tasks also differentiate levels 2 and 2+, and 7 differentiate 2+ and 3. In production, 3 tasks also differentiate levels 2 and 2+, and 13 differentiate levels 2+ and 3.
Most learners performed better on almost all perception tasks than they did on production tasks. Perception exceeds production at the levels studied, advanced and beyond, and thus reflects the general tendency for production to lag behind perception in both L1 and L2 acquisition. For some participants, this tendency was reinforced by the way they had learned Russian and the tasks for which most use the language in their work. This is true in all domains—phonological, morphological, syntactic, lexical, and collocational. Although steady improvement has been shown from ILR 2 to 2+, and from 2+ to 3, level 3 learners still have difficulty with many of the linguistic features assessed in the study.
Additionally, this study supports the idea that acquisition of the features investigated here is a gradual process relative to the advancement through proficiency, and that particular features do not belong to particular ILR levels. Figures 1 and 2 show the distributions of scores across ILR levels for a selection of perception and production tests, respectively. The histograms show rates of correct responses (expressed as log odds) across participants. Improvements in performance are seen as the distribution shifts rightward, which indicates that more people are achieving higher odds of correct responses. The tasks shown here are representative of the largest effect sizes between proficiency levels, across a mix of domains (morphology, syntax, and lexicon). These distributions are somewhat sparse, given the 18–20 participants in each of the ILR levels. It is clear from these figures that in the majority of tasks, there is not an abrupt shift in performance between proficiency levels, but rather a more gradual shift in the distribution of scores across the range. Even though development appears gradual, the LDAs demonstrate that changes in performance can still act as effective discriminators. These results are inconsistent with the idea that a given linguistic feature belongs to a particular level on a proficiency scale, but they confirm the general claim that relative performance on targeted tasks of the type employed in this study track very well with progression across ILR proficiency levels.
Histograms of proportion of correct responses (expressed as log odds) by ILR level, for selected perception tests.
Histograms of proportion of correct responses (expressed as log odds) by ILR level, for selected production tests.
On the basis of the results of the study, checklists of linguistic features for L2 learners that match typical levels of control of those features, measured as percentages accurate, to proficiency levels on the ILR scale are available for the first time. That is, for the wide range of features investigated in this study, it is now known which are controlled by L2 Russian learners, and to what degree, as learners advance from level 2 to 2+, and from 2+ to 3 on the ILR scale. It has also been demonstrated that changes in performance in many of these measures are correlated sufficiently with the construct of general (speaking) proficiency that they can act as effective predictors of proficiency level.
The checklists, in addition to helping learners (both in groups and as individuals), will be useful to syllabus designers, materials writers, classroom teachers, and, eventually, testers in language programs. To the best of the authors’ knowledge, this is the first time such information has been established for any language, including ESL, let alone at relatively advanced proficiency levels in a LCTL. A note of caution, however, is appropriate here. To examine language-specific features alongside a scale that is intentionally language generic is a double-edged sword. It provides, on the one hand, much-needed detail to otherwise uninformative statements about difficult or complex structures; on the other hand, it introduces the possibility that the results will be irrelevant for other languages. To extend this research into other languages and to pursue crosslinguistic comparison and generalization would be of obvious value; nevertheless, language-specific features constitute a starting point for such a program.
Additionally, if the kind of information identified by this study could be provided as part of feedback on a diagnostic test, it could potentially be provided to the learners themselves. To illustrate with ESL, hypothetical examples and typical errors represent what the individual learner can do:
1. ILR 2: Can use subject, direct object, and indirect object relative clauses with 90% accuracy but has difficulty with 50% of object of preposition relative clauses.
2. ILR 2+: Can deal with high noise level only in supporting contexts, at 75% accuracy.
3. ILR 3: Inverts subject and verb correctly 80% of the time in utterances beginning with simple adverbials expressing negative polarity (e.g., Never before had she heard someone with an accent like that).
4. ILR 3+: Inverts subject and verb correctly 100% of the time in utterances beginning with simple adverbials expressing negative polarity, and 70% of the time following complex adverbials (e.g., Only after checking with his supervisor did he agree to the request).
The ability to obtain such results for Russian, plus the experience obtained in the process about the optimal procedures and methodology—especially instrumentation—for doing so, suggests that similar information could now be secured fairly quickly for a broader range of proficiency levels, with any of the commonly used scales, and in a variety of other languages of interest in the occupational and vocational sectors, as well as in academia. Such information would render the proficiency scales in question both considerably more informative, and more informative to a wider range of end-users.
