The article corpus

As the papers in this journal, in the 14 years of its existence, clearly illustrate, the study of bilingualism (and multilingualism) from a cognitive perspective attracts both linguists and psychologists. The cognitive study of bilingualism covers various bilingual populations, differing in age (from one-year-old infants to seniors), type of bilingualism (early vs. late, balanced vs. unbalanced) or cognitive functioning (selection and control (facilitation, competition, inhibition) of languages in bilinguals). The topics are concerned with, among others, language development (in bilingual children), the ways in which the languages in bilinguals affect each other in various linguistic domains in language reception and production (including code switching), the so-called critical-period issue, the linguistic and cognitive advantages and disadvantages of bilingualism, and language impairment (e.g., typically developing vs. cognitively impaired bilingual children, normal vs. brain-damaged bilingual adults). The linguistic domains, in grammar and lexis, vary from phonetics, phonology and morpho-syntax to semantics. The data range from corpora of extemporaneous speech (e.g., in studies of code switching and language contact) to participant performance (accuracy and/or speed) on researcher-designed and experimentally-controlled tasks, sometimes with additional measurement of eye movements, hand or head movements, or brain activity (ERP, fMRI). With the exclusion of single-case studies and corpus studies, typical studies, reported in this journal, pertain to the comparison of groups of participants on some researcher-designed task or tasks (see below). In most of such comparison studies, the researchers apply participant-selection criteria. Typical criteria are age, language-acquisition history and environment (e.g., age of arrival in L2 country, length of residence in L2 environment, years of L2 instruction) and self-assessed proficiency in one or several domains of language use (listening, speaking, reading, and writing). Sometimes, selection criteria include performance on a researcher-administered language test or possession of a language certificate. Thus, proficiency in either or both languages of bilinguals, whether assessed by participants or measured in a more objective way, plays a crucial role in the cognitive study of bilingualism.

Under the assumption that the empirical studies published in Bilingualism: Language and Cognition form a representative sample of the literature of bilingualism (from a cognitive perspective) at large, I looked at all papers published in this journal from the first issue of volume 1 (1998) up till the last issue of volume 14 (2011). The journal distinguishes between (i) keynote articles followed by peer commentaries, (ii) guest-editor introductions to special issues, (iii) research articles, and (iv) research notes. Empirical studies, presenting new data or new analyses of existing data, fall into categories (iii) and (iv). Not all research articles and research notes are empirical, however. Some research articles provide a review of the literature and a new model or theory, while some research notes present critical comments of a paper published in an earlier issue or a reply from the authors of the article criticized. The papers, empirical in the sense just defined, number 224 (i.e. in categories (iii) and (iv)). Of those 224 empirical papers, 142 articles are concerned with various types of group comparisons. The most frequent types of comparisons are the following: (i) comparison of bilinguals dominant in either of two languages, (ii) comparison of bilinguals (languages A and B) with native speakers of either language, having no knowledge of the other language or with monolinguals of either language (knowing no other languages), (iii) comparison of L2 learners at different levels of L2 proficiency with native speakers of their L2, (iv) comparison of groups of bilinguals or groups of L2 learners with different first languages, (v) comparison of bilinguals of different ages, or (vi) combinations of these designs. Of the 142 comparison studies, 140 studies fall into what I call the LP category. An LP study is a study for which, in my personal judgment, the assessment of LP as an independent or moderating variable is a necessary or preferred requirement.Footnote ¹

The assessment of language proficiency

Of the 140 papers presenting group comparisons of the types just mentioned, 37 appeared in volumes 1–7 (1998–2004) while 103 appeared in volumes 8–14 (2005–2011). These 140 papers would have benefited – in my view – from LP assessment in one way or another. In only seven of the 37 papers in volumes 1–7, LP was assessed with an objective test (19%). Of the 103 LP papers in volumes 8–14, 56 (54%) reported the use of an objective LP measure – a significant increase. In 28 (20%) out of the 140 LP studies (13 of 37 studies in volumes 1–7 and 15 of 103 studies in volumes 8–14), LP was not assessed at all, i.e. neither with an objective nor with a subjective procedure.

Objective LP assessment used in participant selection

In the studies that used an objective test to assess candidate participants’ LP, the procedure was in one of two forms. One procedure requested that candidate participants have an LP certificate issued by a testing institute of good reputation (e.g., in the case of English, Cambridge Certificate of Proficiency (UK), College English Test (China), IELTS (Australia and UK), Oxford Quick Placement Test (UK), Test for English Majors (China) and TOEFL (USA)). This screening procedure was adopted by Antón-Méndez (Reference Antón-Méndez2011), Athanasopoulos (Reference Athanasopoulos2006), Chen, Shu, Liu, Zhao and Li (Reference Chen, Shu, Liu, Zhao and Li2007), Cole and Pickering (Reference Cole and Pickering2010), Dong, Gui and MacWhinney (Reference Dong, Gui and MacWhinney2005), Felser, Sato and Bertenshaw (Reference Felser, Sato and Bertenshaw2009), Kormos and Sáfár (Reference Kormos and Sáfár2008), Salamoura and Williams (Reference Williams2006, Reference Yan2007), Silva and Clahsen (Reference Silva and Clahsen2008), Snape (Reference Snape2008), Trenkic (Reference Trenkic2008), and Williams (Reference Williams2006).

Many studies using child participants administered the Peabody Picture Vocabulary Test (PPVT) to assess productive vocabulary (Dressler, Carlo, Snow, August & White, Reference Dressler, Carlo, Snow, August and White2011; Goetz, Reference Goetz2003; Nicoladis, Reference Nicoladis2006; Reyes & Hernández, Reference Reyes and Hernández2006; Uchikoshi, Reference Uchikoshi2006; Yan & Nicoladis, Reference Yan and Nicoladis2009). Some studies using infants matched infants on Mean Length of Utterance (MLU) in oral production. PPVT and MLU in young children can be seen as reflecting a combination of language development and language proficiency. Reyes and Hernández (Reference Reyes and Hernández2006) and Hernandez and Meschyan (Reference Hernandez and Meschyan2006) administered the Boston Naming Test in both languages of their bilingual participants to establish language dominance. Several studies (Ping & Liow, Reference Ping and Rickard Liow2011; Von Studnitz & Green, Reference Von Studnitz and Green2002a, b), examining processing of English, screened participants with the LLEX vocabulary test developed by Meara (Reference Meara1994), which can be described as a computer-administered written lexical-decision task, with word stimuli taken from different frequency classes. Athanasopoulos (Reference Athanasopoulos2009) and Athanasopoulos, Damjanovic, Krajciova and Sasaki (Reference Athanasopoulos, Damjanovic, Krajciova and Sasaki2011) screened subjects on the basis of their scores on Nation's Vocabulary Level test. Luk, de Sa and Bialystok (Reference Luk, Sa and Bialystok2011) used the PPVT (3rd edition) in screening adult subjects (early and late bilinguals and monolingual controls).

Some other types of objective LP assessment to screen subjects with are translation test (Hyltenstam, Bylund, Abrahamsson & Park, Reference Hyltenstam, Bylund, Abrahamsson and Park2009), panel judgement of the speech of participants as near native (Pika, Nicoladis & Merentette, Reference Pika, Nicoladis and Marentette2006) or a cloze test, or C-test (see below).

Fluency tasks to establish language (non)dominance

The adult Spanish–English bilinguals participating in the study of Knupsky and Amrhein (Reference Knupsky and Amrhein2007) were selected on the basis of a category naming task and a reading-speed task, administered in both languages (derived from measures developed by Soares & Grosjean, Reference Soares and Grosjean1984). In the category-naming task, subjects had to produce as many exemplars of a certain category (e.g., animals) as possible in 30 seconds, in both their languages. This type of task, in the form of naming as many words beginning with F, A and S as one can, administered in participants’ L1 and L2, was also used by Ayçiçegi-Dinn and Caldwell-Harris (Reference Ayçiçegi-Dinn and Caldwell-Harris2009), who call it a language fluency task.

Examining processing of morphologically complex Finnish words in monolingual speakers of Finnish and Swedish–Finnish bilinguals in Finland, Lehtonen and Laine (Reference Lehtonen and Laine2003) not only collected self-assessment data from their participants but they also administered a digit-reading task in Finnish, thus providing two types of evidence “that the Finnish skills of the bilinguals were at the same level as those of the monolinguals” (p. 219). The researchers refer to a study of Chincotta, Hyönä and Underwood (Reference Chincotta, Hyönä and Underwood1997), who found that digit reading times in bilinguals were faster in their dominant language when effects of the digits’ word length in the two languages was controlled.

In a study of attention control in L2 processing, Taube-Schiff and Segalowitz (Reference Taube-Schiff and Segalowitz2005) screened their potential subjects (English–French bilinguals) not only on self-assessed L1 and L2 proficiency but also on a speeded word classification task (animacy judgment), which subjects performed faster in their L1 than in their L2. (For a treatise of fluency issues in bilingualism, see Segalowitz, Reference Segalowitz2010.)

Examples of careful screening

A fine example of careful screening is the study conducted by Tremblay (Reference Tremblay2011). Adult English–French participants were selected and assigned to three levels of L2 proficiency by means of (i) language-history information, (ii) self-assessed L2 proficiency, and (iii) an L2 cloze test (validated in another study). Factor analysis and regression analysis, using these three types of information, were conducted to produce three distinct proficiency groups. Another example of a study that applied many criteria, is the study of Sundara, Polka and Baum (Reference Sundara, Polka and Baum2006), investigating the acoustic phonetics of coronal stop production by adult simultaneous bilingual and monolingual speakers of Canadian English and Canadian French. The researchers applied five selection criteria: three language-history criteria, self assessment (confirmed by a fluent bilingual in an oral interview), and rating of speech samples by panels of native listeners on a seven-point scale ranging from no ability to native-like ability.

Self-assessment of language proficiency

Four studies (Lee & Williams, Reference Lee and Williams2001; Salomoura & Williams, Reference Williams2006, Reference Salamoura and Williams2007; Williams, Reference Williams2006) used the self-assessment questionnaire of Bachman and Palmer (Reference Bachman and Palmer1989). This questionnaire consists of 24 questions, divided in a 3 × 3 fashion involving three traits and three question types. The three traits are grammatical competence (morphology and syntax), pragmatic competence (vocabulary, cohesion, organization) and sociolinguistic competence (register, nativeness, nonliteral language). The three question types are ability (e.g., How much English grammar do you know?), difficulty (e.g., How many different kinds of grammar mistakes do you make in English?), and recognition (e.g., How often can you tell when someone makes a grammar mistake?). (For a complete list of all questions, see the appendix in Bachman & Palmer, Reference Bachman and Palmer1989.) The researchers administered the questionnaire to 116 mixed-ability nonnative speakers of English. Age ranged between 17 and 67 years, while length-of-residence in the US ranged between a few months and over ten years. Reliabilities of the nine scales ranged between .86 and .91 with the exception of grammar ability (.54). Exploratory and subsequent confirmatory factor analyses showed good fit of the 3 × 3 LP model. While this finding is quite remarkable given the literature on LP models (Bachman & Palmer, Reference Bachman and Palmer1996; Hulstijn, Reference Hulstijn2011; Purpura, Reference Purpura, Shohamy and Hornberger2008), and while the findings might support the use of this questionnaire as a placement test in a language program, it might be of limited use in the cognitive study of bilingualism (depending on the study's research questions). First, note that the grammar ability scale was not reliable (.54) in the tested sample. Second, note that the global scales of this questionnaire – and the scales in many other questionnaires used in studies of bilingualism – (i) do not differentiate between the four languages skills (listening, speaking, reading and writing), (ii) do not differentiate, within the grammar dimension, between control of phonetics, prosody, phonology, morphology and syntax, (iii) do not elicit information concerning fluency of language control, and (iv) in the answer options of the ability questions, only refer to “most native speakers” as a reference group and not to different types of native speakers in terms of educational or professional profile. Furthermore, it is doubtful whether non-experts are able to give a valid assessment of proficiency dimensions such as pronunciation, fluency of speech, size of productive vocabulary, and grammar control. Thus, global questionnaires, with their lack of detail, might be of limited use for many cognitive studies of bilingualism.

Language-program enrolment

Some studies used enrolment in different years, or levels of a language program to distinguish between proficiency groups (e.g., Ayoun, Reference Ayoun1999; Sopata, Reference Sopata2005; Toribio, Reference Toribio2001). For example, Ayoun (Reference Ayoun1999) compared verb movement in French in three groups of English learners of French, who were in the 2nd, 3rd or 4th year of a French program. The results might have been different, however, had the participants been grouped on the basis of one or several LP tests, because it cannot be ruled out that, for example, the best student in the 2nd year was more proficient in French than the poorest student in the 3rd year. A similar non-optimal design, using enrolment as the between-group criterion rather than an objective proficiency test, was adopted in a study that the present author was involved in (Hulstijn & Marchena, Reference Hulstijn and Marchena1989).

Cloze test and C-test

Several studies used a cloze test, i.e., a paper-and-pencil procedure whereby the test taker is given a text in which some words have been deleted and replaced by gaps and is instructed to fill acceptable words in the gaps (e.g., Kim, Montrul & Yoon, Reference Kim, Montrul and Yoon2010; Montrul, Reference Montrul2004; Montrul & Ionin, Reference Montrul and Ionin2010; Slabakova, Reference Slabakova2005; Yan, Reference Yan2007). In the C-test version, every second word of the text is gapped and the first letters (50% of the letters) are given. The candidate has to complete the incomplete words. (See Hulstijn (Reference Hulstijn, Blom and Unsworth2010) for examples and discussion of cloze and C-test.) Hopp (Reference Hopp2009) used a timed web-based C-test (Grotjahn, 2002) to screen subjects. More often than not, papers give insufficient information with respect to matters such as the type of text(s), the word-deletion procedure, number of deleted words, test instructions, and time allowed. More often than not, information is missing with respect to performance of a native-speaker control group, or the test's internal consistency (e.g., Cronbach's alpha). In contrast, Montrul (Reference Montrul2004) and Montrul and Ionin, (Reference Montrul and Ionin2010) used both a reliable cloze test and a vocabulary test of Spanish (taken from an exam issued by the ministry of education in Spain).

Dealing with the danger of circularity

In 63 studies in the corpus, many of which examined the development, fossilization or attrition of interlanguage grammars, an LP test was used as a means to establish different LP groups. Many studies of this type give the impression that LP is a clear construct that can be used independently in the grouping and comparison of interlanguage grammars. However, especially in the case of studies with a linguistic focus, eliciting the presence or absence of certain phonotactic, phonological, morphosyntactic, morpholexical, semantic or pragmatic features in the interlanguage of L2 learners as the dependent variable(s), it is not immediately obvious if and how LP should be measured independently from the feature under investigation. For instance, in a hypothetical study of the oral production of grammatical noun gender, how should LP be assessed without noun gender being part of the measure, especially when groups of bilinguals are compared with different language pairs, differing in the presence or absence of grammatical noun gender? This question illustrates the main argument of the present article, namely that LP assessment not “only” constitutes a methodological issue, but forces researchers to reflect on what LP actually is, the association of which with the linguistic feature under study they seek to establish.

The study of Guion, Harada and Clark (Reference Guion, Harada and Clark2004) gives a fine example of how the danger of circularity in LP assessment can be addressed. The researchers examined stress placement in bisyllabic nonwords of English (contrasting or not contrasting with Spanish stress placement) in native speakers of English and early and late Spanish–English bilinguals. The experimental task, producing the data for the dependent variable, required subjects to orally produce a concatenation of two aurally presented nonwords. Both bilingual groups were screened on their high proficiency in English by means of various language-history criteria and their performance on two LP tests: a listening-vocabulary and a listening-grammar subtest of the Test of Adolescent and Adult Language (TOAL; Hammill, Brown, Larsen & Weiderholt, Reference Hammill, Brown, Larsen and Weiderholt1994).Footnote ² In addition, as a check of participants’ familiarity with stress patterns of real English words, they had to read aloud 18 words with regular stress and 18 words with irregular words. Mean correct scores by groups and word type were 96% or higher. The researchers succeeded in screening the bilingual subjects (i) with LP measures relevant to the purpose of the study (because knowing stress patterns in some common existing words is a prerequisite for performing in a challenging task eliciting stress patterns in pseudowords), and (ii) without falling into the trap of adopting a circular design (because knowing stress patterns in some common existing words is not the same as predicting stress patterns in pseudowords).

Use of LP assessment to explain variance in the dependent variable

A striking feature of almost all comparison studies using self assessment and/or objective testing of LP is that the information gathered with these procedures is only used for selecting and deselecting potential subjects or for assigning selected subjects to different proficiency groups (e.g., intermediate and advanced subjects). For example, in the former case, selected subjects performed above or below a certain criterion score to demonstrate evidence for near-native or non-native LP, respectively. The most common statistical analysis in the comparison studies is ANOVA. Thus, in the sample of studies surveyed, it is common practice to compare participant groups on whatever the dependent variable of the study is with ANOVA-type analyses. In other words, notwithstanding Grosjean's (Reference Grosjean1998, p. 135) advice to “make bilingual assessment measures covariate variables during the analysis of results or allow participants to be their own control when the study permits it”, within-group individual differences are seldom taken into account in interpreting the results of the analyses. In most cases, the focus is on the interpretation of between-group differences. Thus, studies that found significant differences, for some dependent variable, between bilinguals and native controls or between intermediate and advanced L2 learners do not report how much variance in the dependent variable was actually accounted for by this significant difference. The exception to this observation is the study of Tokowicz, Michael and Kroll (Reference Tokowicz, Michael and Kroll2004). These researchers examined errors in English–Spanish and Spanish–English word translation in English–Spanish and Spanish–English bilinguals as a function of their study-abroad experience (SAE) and their working-memory capacity (WMC). Hierarchical linear regression was performed on the error data, not only with SAE and WMC as predictor variables, but also with self-assessed LP. Only some more recent studies report effect sizes. No studies were found that applied multilevel linear modelling (capable of handling hierarchical dimensions in participant groups), or structural equation modelling.

Comparing apples and oranges

Let us assume for the moment that experts agree that test A is a valid and reliable test to assess control of a given component of LP in language A and that test B is an equally valid and reliable test to assess control of the same component in language B. Let us further assume that both tests consist of 100 independent items and that each response can be awarded with either 1 (correct) or 0 (incorrect). Can we say now that a score of 61 in each test reflects the same amount of control in both languages? Before answering this question, let us first consider the case of two tests measuring the same ability in the same language. For instance, with two versions of the listening subtest of the Test of English as a Foreign Language (TOEFL), one can empirically ascertain whether the two versions are equally difficult by administering them to the same individuals, representing a broad ability range. In this way, one can empirically ascertain test equivalence, both in terms of content (the ability assessed) and difficulty of the test for a given population. However, in the case of two tests measuring the same ability in different languages, there is no way of ascertaining whether a score of 61 on each test reflects equal ability in the languages tested. The main reason is that languages differ in the number and nature of elements in the phonotactic, phonological, morphological, syntactic, lexical and pragmatic domains. In other words, tests of knowledge or control in each of these domains in two languages measure different things (apples and oranges) although they may be referred to by the same label.

A proposal for assessing LP between languages in bilinguals

The ways to compare LP within bilinguals and between languages mentioned above all appear to have their problems, albeit some more than others. Perhaps a more feasible way to assess LP in bilinguals’ languages, for the purpose of establishing language dominance in bilinguals, is to administer tests designed to tap roughly the same LP component in each language and compare bilinguals’ performance to the performance of native-speaker (NS) reference groups in each language. It is crucial, however, that the NSs be selected with great care. First and foremost, never should a NS reference group consist of individuals with a higher intellectual profile (level of literacy, level of education, level of profession, intellectual level of leisure-time activities) than the profile of the bilingual target group. The effect of the linguistic norms of the standard language, in most societies, on NSs receiving education at the secondary and tertiary level should not be underestimated. Second, if the study's research question require LP measures in linguistic cognition that all NSs are assumed to possess (so-called basic language cognition, see next section), three scenarios offer themselves. Under the first scenario, only NSs of lower intellectual profiles are selected to ensure that the LP tasks do not require the bilinguals to do what monolingual NSs are not able to do. Under the second, more ambitious scenario, NSs of a variety of intellectual profiles are selected. This increases the likelihood of variance in NS performance, which makes this scenario more challenging. Under the third scenario, which only applies when the bilingual target group has a high intellectual profile, the NS reference groups consist of individuals of the same, high intellectual profile. (Note that many cognitive studies of bilingualism in adults avail themselves of university students as participants.) In sum, several feasible solutions of the problems in LP assessment in bilinguals appear to exist, all based on comparison with NS reference groups, provided due attention is given to the LP component(s) involved (within or beyond basic language cognition) and the intellectual profile of the NS references group(s).

As the reviewers of this paper point out, an additional method in the assessment of LP in bilinguals’ languages would involve repeated administration of language tests, i.e. in longitudinal within-subject designs. This method is demanding but forms a very important complement to cross-sectional designs.

Levels of proficiency

To my knowledge, there is no linguistic, psycholinguistic or sociolinguistic theory on the basis of which one could define levels of LP. Similarly, if I am not mistaken, there are no theories on the basis of which one could define levels of intelligence, levels of language-learning aptitude, levels of working-memory capacity, levels of language-learning motivation, or levels of socio-economic status (SES). Performance in tests developed for measuring these attributes in individuals is normally expressed as a score on one or several scales. Using scores, one could say that a person with an IQ of 107 is more intelligent than a person with and IQ score of 101, but not that the former person finds himself or herself at a higher level of intelligence than the latter. Similarly, someone with a backward digit span of 6 can be said to have a larger working memory span than someone with a score of 4, but not at a higher working-memory level. Although no theoretical basis exists for the distinction between levels of LP, in educational practice it is quite common, perhaps even unavoidable, to speak of LP levels, such as beginning, intermediate and advanced levels, or to adopt the level labels used by companies that produce language-proficiency tests. Under the auspices of the Council of Europe, a Common European Framework of Reference for Languages (CEFR) was developed, defining six global LP levels, best known as A1, A2, B1, B2, C1, and C2 (Council of Europe, 2001). In Hulstijn (Reference Hulstijn2011), it is argued that the six levels of the CEFR fail to consistently distinguish between language development and language proficiency. The six levels cannot be regarded as forming a unidimensional ladder of language development because the higher levels (B2, C1 and C2) can only be attained by people with higher intellectual capacities.