2. Test constructs and policy

Test scores tend today to be interpreted in terms of real world abilities and skills. In earlier norm-referenced approaches to assessment, an estimate of a candidate's ability, in the form of a test score, was interpreted with reference to the performances of other test takers, that is, with reference to the scores of the population of test takers for that test. Thus, test score meaning was always relative. In current criterion-referenced approaches, scores are interpreted in relation to pre-determined standards of knowledge and capacity. Such standards are typically couched in practical, functionalist terms, reflecting the functionalist, communicative tradition of language teaching. For example, the widely used and very influential Common European Framework of Reference for Languages (CEFR: Council of Europe 2001) consists of an ordered set of statements about aspects of communicative ability, grouped into six levels (from lowest to highest: A1, A2, B1, B2, C1, C2) according to the complexity of the skills concerned. It is claimed, for example, that a speaker assessed as meeting the standard for level A2

while a speaker at B2

The wording of these standards is important, as they represent the test construct, the assumed view of language proficiency which is assessed in the test and which is the target of measurement in any individual case. (The reference to the native speaker in the definition of level B2 is crucial to the argument of this paper and will be considered below; cf. Seidlhofer Reference Seidlhofer2003.)

Increasingly, the design of an assessment and the construction of a test instrument begin with such a set of standards. These may be rather general, as in the case of the CEFR, or more specific, as in standards relevant to a particular workplace or educational setting; we will consider English language standards for the aviation workplace later in the paper.

Policy initiatives expressed in the wording of standards are increasingly central to language testing. The fact the standards set by policy act as the basis for the test has changed the nature of language testing. Previously, test developers, informed by the latest theories of language proficiency, of communicative interaction, and of the way in which language knowledge intersects with and interacts with other aspects of professional or educational competence, defined the test construct themselves in a stage preparatory to the development of a procedure which would provide evidence relevant to that construct. This ‘thinking stage’ of test development has now been removed. Instead, given that the test construct, and its wording, are determined by complex policy procedures, evolving discussions within applied linguistics on the nature of language proficiency and communicative interaction are far less likely to inform test constructs once the standards as initially formulated have been adopted – they are cast in stone. While tests have always been sensitive to policy and market considerations (see Spolsky Reference Spolsky1995 on the history of TOEFL and Davies Reference Davies2008 on the history of IELTS), these pressures are now directly and explicitly expressed within the very wording of the test construct, from which everything in a test flows.

The CEFR is a case in point. The wording of the framework inevitably reflects the policies and values of its original sponsor, the Council of Europe, and its current users, governments and educational institutions throughout the world. The CEFR has its origins in the work of the Council of Europe in the early 1970s. The Council was looking for a way of thinking about achievement in language study which was independent of the structural particularities of particular languages, in the interests of labour mobility and readily transportable and interpretable credentials. The policies of the Council at that time were critical in the development of the notional/functional syllabus, and the definition of levels of syllabus in such functionalist terms; the Council was the principal influence on the European version of communicative language teaching. It can therefore be argued that the functionalist orientation of communicative language teaching and the curriculum and assessment frameworks that go with it are a reflection of the values of the Council – broadly, European integration. In other words, the terms in which the outcomes of language education are specified, and which are the construct of language assessment, are expressions of policy. And the formulations of such frameworks, including the way in which they lend themselves to a reduction to simple numbers – A1, A2, etc. – are designed to meet the needs of those responsible for the accountability of educational systems.

The credit transfer (‘unit credit scheme’: Trim Reference Trim1980) and labour mobility motivation of the Council of Europe is explicable in terms of its policy goals, as described on its website in the period in which the CEFR was adopted (they have since been revised to focus more on the promotion of human rights and of democratic governance systems):

The cultural context of the goal of an integrated European economy, a regional form of globalization, is clear, and is strongly reminiscent of the mission of the Organization of Economic Cooperation and Development (OECD), an organization more explicitly committed to furthering the processes of globalization. It may be worth comparing the actual wording of the goals of the two organizations (the goals that seem shared are shown in bold):

The OECD specifically focuses on economic development, and the Council of Europe on education, but in a deeper sense the goals of the two organizations in the period leading up to the formulation and adoption of the CEFR reflect the same mission. Education is seen as centrally concerned with economic development, and the curriculum frameworks determining the shape of the accompanying assessment regimes for educational achievement, including that for languages, are central to the fulfilment of the goals of each organization. The OECD also has a testing regime, the Program for International Student Assessment or PISA, which is having profound impacts on the educational systems of member countries (McNamara Reference McNamara, Candlin and Crichton2011) – it tests the literacy of 15 year olds in the language or languages used as the medium of education in the member states, as well as their knowledge of mathematics and science.

This linking of language policy to educational goals through processes of examination and certification is, of course, not new. In a discussion of the linguistic unification of France in the eighteenth century, the French sociologist Pierre Bourdieu writes:

3. Elaborating the construct

Increasingly, then, test constructs, which are the heart of tests, are dictated to us by policies governing teaching, and hence assessment practices, in educational systems. The wording of the statements of test constructs in standards documents tends to be brief and directed at the audience of those using and interpreting test scores, rather than those responsible for constructing and conducting tests: in other words, the wording is user-oriented rather than constructor- or assessor-oriented (Alderson Reference Alderson, Alderson and North1991). Further elaboration is often provided to help teachers, syllabus developers and test developers interpret the meaning of the standards and to guide them in the development of syllabuses and the operational tests that accompany them.

The CEFR has considerable elaboration of the descriptors set out in the global scale, as described above, in the form of a manual designed to assist in the interpretation of the levels, and the application of the scale in different contexts is set out in a book-length publication (Council of Europe 2001). More broadly, a great deal of current work in language testing is designed to provide interpretative commentary on the meaning of standards such as the CEFR, and more and more detailed advice to test developers and teachers about relating the necessarily abstract and general statements of standards to the day-to-day realities of test design and classroom assessment. Language testers thus play a mediating role between policy statements and the needs of teachers and learners, though it should be recognized that in many respects the needs of administrators and those of teachers and learners are at odds, placing an impossible burden on language assessment (not to mention on the teachers who have to do the assessing) (McNamara Reference McNamara2001).

Now the impact of the CEFR will in some respects be positive. The spread of policies favouring communicative methods of teaching for a wide range of languages throughout Europe can be attributed to the influence of the CEFR. A shared framework across languages means that shared professional development of teachers of different languages is possible, at least so far as familiarizing them with the approach of the CEFR is concerned. The articulation of the goals of language education and the course of development of the learner's competence leads to valuable discussion among teachers. There is already some documentation of these benefits (Morrow Reference Morrow2004).

The policy goals of the CEFR are now dominating language education policy at every level in Europe, in a striking example of policy-driven assessment. It is worth noting in passing that the appeal of this management function of the CEFR to other governmental systems, and of course the power and influence of Europe, mean that it is being used as a crucial reference point – in fact the only reference point authorized by policy – well beyond Europe: for example in North and South America, Australia and Asia. And the impact of the CEFR extends to areas beyond language education. The levels of the CEFR are enshrined in law, in policies incorporating the use of language tests in granting immigrants rights to entry, residency and ultimately citizenship (Van Avermaet Reference Van Avermaet, Hogan-Brun, Mar-Molinero and Stevenson2009).

4. Imposing a common currency: Calibrating tests against the CEFR

The overwhelming political authority of the CEFR means that increasingly, language assessments in many, indeed most, areas of education must be calibrated against it. This is a complex and technically demanding process. Frameworks like the CEFR are like the Euro, or like the European Committee for Standardization (CEN). They make exchange across boundaries easier, but reduce local variation, and render unintelligible other accounting systems, or sets of cultural values, or formulations of the goals of language education, which cannot be directly translated into the language of the CEFR. Moreover, if the construct of a curriculum or an assessment cannot be calibrated against the CEFR, it has no currency.

It is becoming imperative for published tests to be explicitly related to the CEFR. There is a clear commercial advantage for those which can do so. Cambridge ESOL, for example, has linked the levels of its main suite of examinations to the levels of the CEFR: in its own words:

This leads to mapping exercises such as that shown on Cambridge ESOL's website at www.cambridgeesol.org/assets/img/exams/cefr-diagram-large.jpg; this is also included as supplementary material on this journal's website (http://journals.cambridge.org/lta).

But linking is no simple matter. The question of the calibration of individual tests to the CEFR is very demanding and very complex, even for an organization like Cambridge. Recent changes to immigration law in Britain tie admission under certain visa categories to the applicant's demonstration of a certain level of proficiency on the CEFR (www.ukba.homeoffice.gov.uk/policyandlaw/immigrationlaw/immigrationrules/appendixb/). This means that the calibration of levels on a widely available test such as IELTS becomes hugely consequential. Cambridge itself is unclear about the correct calibration. On the one hand, it states

but on the other,

So what is C1 on IELTS? 6? 6.5? 7? 8? These differences are very important for implementing an immigration policy where C1 is a crucial level for admission decisions. There is evidence from other carefully conducted linkage studies of competitor tests to IELTS such as TOEFL iBT and the new Pearson Academic test that there may be significant problems with the IELTS calibration. For as long as it lasts, Cambridge has a significant commercial advantage over its competitors, as it is far easier to meet the C1 standard in terms of the claimed IELTS equivalence than it is to meet the same C1 standard by using TOEFL iBT or Pearson Academic. This discrepancy needs to be resolved urgently.

5. The costs of a ‘common currency’

Even if it were possible to impose a common currency for standards in language education, such a development would come at enormous cost, the discussion of which has barely begun (Fulcher Reference Fulcher2004). The quote from Bourdieu reminds us of what this cost might be. The cost of unification is the devaluing of the local interpretation of the goals of education. One issue is that the limiting of the goals and meaning of language learning to functional, communicative objectives ignores the role of language learning in the subjective experience of the learner as an individual with a history, both personal and cultural.

Research in applied linguistics is beginning to address the meaning of language learning to individuals. The theme of identity and second language learning is currently being explored in a number of major publications (for a survey, see Block Reference Block2007). In particular, Pavlenko's studies of literary autobiographies (Pavlenko Reference Pavlenko2007) have shown how language learning is deeply embedded in the life trajectories of the authors; functionalist accounts are inadequate to deal with the complexities revealed in these accounts. Her work on language learning and emotions (Pavlenko Reference Pavlenko2005) has extended the scope of this research into other autobiographical accounts of language learning, language use and language loss. Kramsch's discussion of subjectivity in language learning (Kramsch Reference Kramsch2009) interprets narratives of bilinguals and other learners within contemporary social theory, and shows how ‘individual’ experience is constructed within the operation of larger social forces. Judith Butler's work on desire in the construction of subjectivity (Butler Reference Butler1997) points to a new framework for understanding such tired concepts as attitudes and motivation in language learning, which have lacked an appropriate social theory to guide research in this field.

Although recent work on language learning and individual identity is increasingly located within contemporary social theory, and adopts a social perspective, it is still focused on interpreting individual experiences of language learning, albeit in the light of social forces. We have yet to give an account of how language learning in different societies and cultures will have specific social meanings, and hence potential meanings for individuals, within the history of contact, cultural and political, of those societies with the societies and communities in which the target language is spoken. These historical conditions will mediate the encounters of individuals from different societies with the same language, so that the learning of a particular language may have different meanings according to the specific social and cultural background of the learner.

The imposition of a single set of cultural meanings and social and political values for language education, for each setting in which the CEFR is adopted, eviscerates the traditions of language teaching which are incompatible with the CEFR. In cultural and historical terms, learning English is simply not the same for a Singaporean, an Indonesian, a Vietnamese, a French person, a Dutch person, or a Hungarian. And different languages are indeed different, and vary in socio-political role, range and purpose of use, and carry very different histories of contact, often violent, between speakers of the speech communities involved. The CEFR puts all foreign languages into one and the same category, thereby erasing the fact that they are ‘foreign’ in very different ways. Learners from those backgrounds may be aware of the cultural and political significance of the act of learning the language of a cultural group with a particular historical relationship to their own, unless they are invited, in an act of collective historical amnesia, to wipe the slate clean and re-identify themselves as citizens of the new globalized world.

While the CEFR, in other words, is conceived of as something like the Euro, the currency of language is not simply convertible. Managers love the CEFR and other standards-based frameworks all over the world because they make available a certain kind of quantitative accountability in a way that was not possible before. But this all-too-ready surface translatability denies the untranslatable, the part of language and language learning which gives the enterprise value beyond labour force mobility. And being capable of ‘participation in an international society’ is not best achieved through historical amnesia, tempting though that may be given the horrors of the previous century.

A further problem with the imposition of universal language test constructs by managerial systems is that the constructs cannot easily be changed or challenged, because researchers and theorists of language testing are no longer responsible for determining the test construct. The evolution of notions of language proficiency will only slowly be reflected in test constructs, for two reasons. Firstly, because the test construct is embedded in policy processes, it is only through policy processes that the construct can be challenged or updated. Academic researchers are not necessarily very good at influencing policy debates. Secondly, because existing accountability frameworks such as the CEFR are deeply embedded in existing administrative procedures, there will be a natural reluctance to change them because of all the administrative and even legislative work that is entailed.

6. The need for change: English as a lingua franca

It seems to me that there is a great need for re-thinking the construct involved in language assessment. I think we are at a moment of very significant change, the sort of change that only comes along once in a generation or longer – the challenge that is emerging in our developing understanding of what is involved in ELF communication (Jenkins Reference Jenkins2000, Reference Jenkins2006, Reference Jenkins2007; House Reference House2003; Seidlhofer Reference Seidlhofer, Gnutzmann and Intemann2005, Reference Seidlhofer, Rubdy and Saraceni2006; Seidlhofer, Breiteneder & Pizl Reference Seidlhofer, Breiteneder and Pitzl2006; Canagarajah Reference Canagarajah2007; Mauranen & Ranta Reference Mauranen and Ranta2009; Seidlhofer Reference Seidlhofer2009; Pitzl Reference Pitzl2010). I feel that this will be as big a change for language testing as the communicative revolution was for language teaching and for language testing 35 years ago.

Let me explain what I mean. Hyejeong Kim is currently completing a Ph.D. at Melbourne on the policy of the International Civil Aviation Organization (ICAO) governing English language testing, which has recently introduced English proficiency standards to be met by pilots and air traffic controllers working in international aviation settings (Kim & Elder Reference Kim and Elder2009). Kim's study investigates attitudes to this policy within the aviation industry in Korea, which has in various ways resisted and sought to weaken the policy, through, for example, controversially revealing material from the English test used in Korea on the web well in advance of test dates, to assist those facing the test to pass. She has gained access to recordings of interaction between Korean air traffic controllers at airports in Korea (non-native speakers of English), and the pilots with whom they are communicating (a mix of non-native and native speakers of English). In some of these interactions, difficulties of mutual understanding emerged, which had potentially dangerous implications. Among the situations she studied are the following: a conversation with the Russian pilot of a Russian passenger plane who wanted to change course because of a technical problem, and who needed permission to do so; a conversation with an Australian pilot, a native speaker of English, whose plane had suffered a mechanical failure after landing and who had no power to leave a busy runway, and needed his plane to be towed to safety; a conversation with a French pilot who misheard an instruction and had therefore made a runway incursion, disrupting the landing path of incoming planes. I think we can agree that in all these situations, the communication breakdowns which occurred, but which were ultimately resolved, were potentially significant for airline and passenger safety.

What is the nature of the miscommunication in these incidents? Kim played recordings of the incidents and showed transcripts of the recordings to two groups of informants, Korean air traffic controllers, and Korean pilots, and using a form of think-aloud methodology, elicited their perspective on these incidents. The informants were asked to articulate their understanding of what was going on, and the nature and sources of the miscommunication, as the situations unfolded. These comments form the main data for the thesis. Kim then coded the data to see what they focused on. (The subject of study is thus informants’ attitudes to examples of lingua franca communication in this setting.)

The main conclusion from the analysis is that while the language proficiency of some of the Korean air traffic controllers is an issue in some of the situations, the reasons for the communicative breakdown go far beyond the language proficiency of one of the participants. A number of factors come into play. First, the professional competence of the pilot and the air traffic controller is a crucial factor in avoiding or successfully resolving miscommunication. Professional competence is an important factor in shaping discourse, to the point that it is difficult to separate them. Second, and most relevant here, the responsibility for communication in English as a lingua franca is a joint responsibility, and failure is rarely the fault of the second-language speaker alone. The rules of communication in the aviation industry set out clear procedures for routine communication, involving the required use of set phraseology for routine requests, situation reports and explanations, and required procedures for reading back what has been heard, and confirmation that what has been read back is in fact what was originally said. For less predictable situations, spontaneously formulated ‘plain English’, that is, the ordinary code, is permitted. Kim's informants commented on two aspects of ELF communication. First, the native speaking or fluent non-native speaking pilots did not observe the rules about using routine phraseology, and unnecessarily used ordinary conversational English with the Korean air traffic controllers in entirely routine and predictable situations, which sometimes resulted in miscommunication. Second, the native speaker pilots did not know how to accommodate to their ELF interlocutors in terms of accent features, lexical choice, speed of delivery, and so on. In turn, air traffic controllers did not always acknowledge their lack of understanding, which subsequently complicated the misunderstanding.

We can see then three features of ELF communication in this example (and these are amply borne out in the extensive existing literature on the subject, a sample of which was cited above):

1. (1) The interlocutor of the second-language speaker of English will not necessarily be a native English speaker (though he/she may be), but a speaker of another second language.

2. (2) Factors other than simple language proficiency – for example, professional competence – are an essential component of the communication.

3. (3) The responsibility for successful communication is a joint responsibility between the participants in the communication – it does not lie entirely with the non-native speaker.

Let us now consider the ICAO English proficiency standards (ICAO 2004). As you might expect, the wording of standards reflects to a certain extent the demands of the workplace setting. The following is the definition of the operational proficiency required to gain a professional licence:

Although the ICAO proficiency standards do not refer explicitly to the fact that the relevant work setting involves ELF communication, the mention of the need to ‘recognize and resolve misunderstandings’ is potentially relevant to that construct. In order to further elaborate the meaning of the standard, six levels of proficiency are defined in terms of the degree to which they meet the standard. Air traffic controllers and pilots must reach at least Level 4 in order to get a certificate allowing them to continue working in their chosen profession. In defining the levels, various aspects of performance are identified, and descriptions of performance are provided at the relevant level for each aspect. The following dimensions of proficiency are identified: Pronunciation, Structure, Vocabulary, Fluency, Comprehension and Interactions. Apart from Interactions, this is quite a traditional set of rating categories for the assessment of speaking, and the descriptions within each are informed by understandings of language learning and language pedagogy that are quite general and not necessarily specific to the particular work setting. Here are the definitions of two of these dimensions, Comprehension and Interactions, at Level 4, the Operational Level:

Distinctive features of the work setting are acknowledged at various points: note the mention of a ‘situational complication or an unexpected turn of events’, and there is an acknowledgement that communication will be among ‘an international community of users’, that is, that the target of the test is ELF communication as much as, or more than, communication with native speakers of English. The phrase ‘deals adequately with apparent misunderstandings by checking, confirming, or clarifying’ again acknowledges the mutual nature of the communication.

Nevertheless, it is interesting to note that as a matter of policy only one side is subjected to the examination – the native speakers are exempt, although Kim's study shows their contribution to miscommunication. Moreover, the scale of proficiency defines subsequent higher levels in terms of increasing control of the code, not increasing ability to deal with increasingly complex situations of miscommunication.

For example, the distinction between subsequent levels (5 and 6) does not appear to be specifically related to the work setting. The criterion of comprehension at Level 5 refers to ability ‘to comprehend a range of speech varieties (accent and/or dialect) or registers’ whereas Level 6 states ‘Comprehension is consistently accurate in nearly all contexts and includes comprehension of linguistic and cultural subtleties’. One might have here expected some further reference to the conditions of communication in terms of greater demand – unfamiliar accent, styles of delivery, and so on. Interestingly, in their actual working life, the pilots themselves are oriented to the demands of lingua franca communication. Kim (personal communication) reports that when flying to a new destination, the pilots listen to tapes of communication in English by speakers of ELF from that region, in order to familiarize themselves with the accent of the air traffic controllers in that destination. They do not listen to more tapes of native speakers in order to improve their comprehension of native speaker speech. Native speaking pilots might be recommended to listen to tapes of local air traffic controllers, but there is no ICAO policy to enforce that – in fact it is never discussed, because of the presupposed privileging of native speaker norms.

There have been several developments in response to the ICAO policy. ICAO's framework and the definition of operational competence are not tied to any particular test; each national system can choose to meet the standard in whatever way it wants. Many national civil aviation authorities have endorsed certain existing tests, usually general proficiency tests, as suitable instruments for eliciting evidence of the standing of individual pilots and air traffic controllers in relation to the standard. These tests are often associated with general proficiency scales that are in turn linked to the CEFR. There are two problematic aspects of this development. The first is that general proficiency scales do not articulate the relevant construct, which is ELF communication. Secondly, they locate the issue in the second language proficiency, narrowly defined, of the non-native speaker. The International Language Testing Association is currently attempting to establish itself as an accrediting agency for the tests being adopted by different national civil aviation authorities. While this has many positive aspects, in that it will deal with aspects of the fairness of the test, it will not deal with the lack of justice embodied in the construct (McNamara & Ryan Reference McNamara and Ryan2011). And because to change the construct involves a complex political process, it is unlikely to change in a hurry. In response to this impasse, the Korean civil aviation industry have taken the controversial position of publishing the test online in advance so that pilots and air traffic controllers can more easily pass it. They are indicating in this way their own disbelief in the construct of the test; it is not that they want to conduct civil aviation in Korea recklessly or dangerously – in fact Korea has an excellent civil aviation record – but they resent the injustice of very experienced and competent air traffic controllers losing their jobs needlessly.

Turning to the CEFR, to what extent is the construct of ELF communication adequately represented? As Seidlhofer (Reference Seidlhofer2003) pointed out, it is absent from the framework – not surprising, given its intellectual antecedents in the European version of the communicative movement of the 1970s. Here are the definitions of the two most advanced levels of the CEFR, Level C, Proficient User, divided into two levels, a higher (C2) and a lower (C1).

For a more detailed understanding of how the levels are defined for the skills of listening and speaking, we can turn to the descriptors in the self-assessment grid:

It is clear that the concept of ELF communication does not inform these descriptors. In terms of comprehension,

1. (1) The interlocutors are assumed to be native speakers.

   While the wording ‘provided I have some time to get familiar with the accent’ might lead us at first to think that this means non-native speaker accents, we realize that this cannot be right, as the references to ‘at fast native speed’ and ‘a good familiarity with idiomatic expressions and colloquialisms’ make clear.

2. (2) The role of other competences as an essential basis for comprehension is not considered.

3. (3) The responsibility for successful communication is held to lie entirely with the non-native speaker.

In terms of production, the references in the definition of Spoken Interaction are revealing: ‘I can express myself fluently and convey finer shades of meaning precisely. If I do have a problem I can backtrack and restructure around the difficulty so smoothly that other people are hardly aware of it’. The problem is presumably ‘mine’ alone, not a problem caused by the competence of my interlocutor. What is required in such a situation is not the ability to ‘convey finer shades of meaning’. Again, the ever more demanding native speaker interlocutor is the presumed target of the communication efforts.

My point here is not only the need for change, the need to challenge the construct, but what it would take to move the mountain of the CEFR in the direction of change. The arguments of applied linguists are unlikely to be persuasive in themselves – policy statements need policy action to alter them.