1. Purpose of the Common European Framework of Reference

First, let us remind ourselves what the Common European Framework of Reference (CEFR) (Council of Europe 2001) is about. Published in 2001 after a period of piloting, it consists of a descriptive scheme, common reference points expressed as six proficiency levels, descriptor scales for many aspects of that descriptive scheme, advice on curriculum scenarios and considerations for reflection. The aim of the CEFR is to stimulate reflection on current practice and to provide common reference levels to facilitate communication, comparison of courses and qualifications, plus, eventually, personal mobility as a result. In this process, in relation to assessment, the CEFR descriptors can be of help:

Before the CEFR there was a practical ‘Tower of Babel’ problem in making sense of course certificates and test scores. A teacher, school or examination body would carry out a test and report a result in their own way as ‘19’, ‘4.5’, ‘516’, ‘B’, ‘Good’, etc. It is no exaggeration to say that twenty years ago a teacher of Spanish in a secondary school in southern France, a teacher of French to Polish adults and a teacher of English to German businessmen would have taken ten to twenty minutes to establish any common ground for a discussion. The CEFR labels help.

In taking notice of the CEFR, most people therefore start with the levels. In contrast to those used in other language proficiency scales, the CEFR descriptors and levels are the product of serious validation in pre-development studies (North Reference North1995: Reference North2000a; North & Schneider Reference North and Schneider1998) and post-development studies (Jones Reference Jones2002; Kaftandjieva & Takala Reference Kaftandjieva and Takala2002; North Reference North2002). These confirmed that language teachers and learners interpret the descriptors consistently across a sample of educational contexts, regions and target languages. However, the existence of a scale of levels does not mean that situating learners, courses and examinations on that scale is straightforward. Experimentation with the CEFR descriptors contained in checklists for each level in the European Language Portfolio, of which more than 100 versions have now been produced (Schneider, North & Koch Reference Schneider, North and Koch2000; Little Reference Little2005; www.coe.int/portfolio), has helped schools to become reasonably confident in their judgement as to the level of their learners and courses. But relating examinations and test scores to the levels is a more serious matter that people have had more difficulty with. Therefore, in response to requests from member states, the Council of Europe put together a working party to develop a Manual for relating language qualifications to the CEFR, which after publication in pilot form in 2003 is now available on the Council of Europe's website (Council of Europe 2009) accompanied by further material on exploiting the scaling of teacher assessments (North & Jones Reference Jones2009), a reference supplement (Takala Reference Takala2009), and sets of case studies from the piloting now published by CUP (Martyniuk Reference Martyniuk2010). I will come back to the procedures recommended in the Manual later in the paper.

The pre-CEFR Tower of Babel problem masked a second, more theoretical problem: the relation of assessment results to real-world practical language ability. Tests each reported their own scale and left users to work out what different bands/scores on the scale meant in terms of real-life ability. As Jones, Ashton & Walker (Reference Jones, Ashton, Walker and Martyniuk2010: 230) point out, the CEFR Manual helps language testers to address this central concern of criterion-referenced assessment. The CEFR promotes an action-oriented approach: seeing the learner as a language user with specific needs, who needs to act in the language in real-world domains. The CEFR descriptor scales can provide the vertical continuum of real-life ability needed as an external criterion for valid criterion-referenced assessment. The Manual offers sets of procedures to help in this process. This point is returned to later the paper.

However, it is important to remember that the prime function of the CEFR is not to get all tests reporting to the same scale but to encourage reflection on current practice, and thus to stimulate improvement in language teaching and learning (and testing). The CEFR was developed to contribute to reform and innovation and to encourage networking; it is certainly not a harmonisation project, as we made very clear:

Nor is the CEFR a panacea. It is a heuristic model intended to aid communication, reflection and focused research. This fact was recognised in various articles in the special issue of the journal Language Testing on the CEFR, best summarised by Norris in his introduction:

Neither has the Council of Europe or any of the authors ever claimed the CEFR to be perfect or complete. The Council has, in fact, repeatedly stated that all the lists and sets of descriptors are open-ended. No descriptors at all were published for several important aspects of the descriptive scheme and some 10% (40) of those that were published are not based on research, half of those being at C2. I was in a position in which I just had to create most of the descriptors for communicative language activities at C2. The English Profile Project is currently focusing on bringing more precision to descriptors for the C levels. I personally find it disappointing that we have had to wait 15 years for a follow-up project to further extend the calibrated descriptor bank.

In any case, as Alderson once pointed out (personal communication), the descriptors are designed to describe learner behaviour, not test tasks. Operationalising them into a specification for a test task requires a process of interpretation that is not always straightforward. As Weir (Reference Weir2005) and Alderson (Reference Alderson2007) state, a lot of work is involved in that process and the CEFR (2001) is of only limited help in this respect. This need for further specification, development and underpinning was underlined by various contributors to a series of articles in a special edition of Perspectives in The Modern Language Journal (Byrnes Reference Byrnes2007). The CEFR cannot just be applied; it must be interpreted in a manner appropriate to the context and further elaborated into a specification for teaching or testing.

Personally, I think that the two biggest dangers with the CEFR are, firstly, a simplistic assumption that two examinations or courses placed at the same level are in some way interchangeable and, secondly, a rigid adoption – rather than adaptation – of the descriptors. A label such as ‘A2’ serves only as a convenient summary of a complex profile. The CEFR/ELP descriptors are intended for orienting learning and profiling developing competence, not just for determining what overall level someone is considered to be. Every person and every test that is said to be A2 is said to be A2 for different reasons. As regards the descriptors, I deliberately kept these as context-neutral as possible, so that the scale value of each individual descriptor would be more stable across contexts. The descriptors must be adapted and further elaborated to suit the context, as in the 100 versions of the European Language Portfolio that have been endorsed by the Council of Europe. Fortunately, adapting things is not something teachers are shy about doing. A rich bank of such adapted descriptors is now available on the Council of Europe's website as a source of inspiration (www.coe.int/portfolio). A recent project in EAQUALS (European Association for Quality Language Services) reviewed that bank in order to further develop and fill the gaps in a set of general language descriptors for the main levels (criterion levels) and for the plus levels (levels between the main levels). The results are also available at the same web site.

2. Good uses of the CEFR

What might, then, be regarded as good use of the CEFR? Apart from the question of a serious engagement with the communality of the reference levels, which is the subject of the second half of this paper, I think there are two main points: firstly, taking as a starting point the real world language ability that is the aim of all modern language learners and, secondly, the exploitation of good descriptors as transparent learning objectives in order to involve and empower the learners.

3. Procedures for relating assessments to the CEFR

More and more schools and teachers are using the CEFR levels to communicate with learners, and examination boards increasingly refer to them. It is clear that what exactly is meant in practice by a set of verbally defined levels of proficiency like the CEFR Common Reference Levels cannot be entirely separated from the current process of implementation, training workshops, calibration of illustrative samples, adaptation of CEFR descriptors, and linking of tests to the CEFR. However, the levels are not intended as a free-for-all for users to define as they wish. As was emphasised at the 2007 intergovernmental Language Policy Forum held to take stock of the implementation of the CEFR, the levels should be applied responsibly, especially if national systems and international certificates are being aligned to them (Council of Europe 2007: 14). This means taking account of established sets of procedures, such as those recommended in the Manual, when designing any linking project.

The fact that good practice guidelines are necessary for linking high-stakes assessment to the CEFR can be demonstrated by a simple example. In response to the plethora of proficiency standards that have developed in the UK, two British researchers were commissioned to produce a so-called alignment of all the different language proficiency scales, including the CEFR. The study, called ‘Pathways to Proficiency’, claimed a relationship between CEFR Level B1 and the British National Language Standards (BNLS) Level 2. As the authors state (Department for Education and Skills 2003: 12–14), this alignment was done on the basis of no more research than placing the documents on a table next to each other and eyeballing the descriptors. Even then, to my mind, the authors do not always seem to have selected the most appropriate CEFR descriptor scales to compare to the British ones. If this rather intuitive approach had been taken by a language education provider or publisher, this would be a minor issue: their interpretation would be confirmed or adjusted over time. However, in a nationally commissioned, high-stakes project, things are more serious. The English school-leaving certificate, the GCSE, is placed at BNLS Level 2. The unsubstantiated suggestion that BNLS Level 2 = GCSE = B1 is by no means entirely unconnected to the fact that in February 2010 the UK Border Authority declared B1 to be the minimum for a UK student visa (required for a stay of more than six months). As reported in the EL Gazette at the time, GCSE was the comparison used in both the British Parliament and the press to justify the measure. In fact a pass at GCSE (Grade C) is probably A2. The top Grade A – supposedly representing the standard of the previous O Level – may be B1. Nobody knows exactly, because nobody has bothered to do a study. However, the identification of GCSE with B1 helped to deny tens of thousands of language students the opportunity to come to the UK and cost the British English Language teaching industry 10% of its business quite literally overnight – and almost certainly erroneously. Even worse, the UK Border Authority has since proposed raising the minimum for a student visa from B1 to B2, a step which would have dire consequences for the number of international students at UK universities.

The Manual recommends four sets of procedures for linking to the CEFR: familiarisation, specification, standardisation and validation. In the context of operational school assessment one could also put a special emphasis on moderation, to limit and/or adjust for subjectivity in assessments by teachers.

3.2 Specification

Specification in this context includes defining the coverage of the course or examination in relation to the CEFR descriptor scales, in terms of both the curriculum and the assessment tasks and criteria used to judge success in them. This involves selecting communicative activities, perhaps guided by the descriptor scales in Chapter 4 of the CEFR (2001), summarised in CEFR (2001: Table 2), designing tasks and writing items. Valid assessment requires the sampling of a range of relevant discourse. For speaking, this normally means combining interaction (spontaneous short turns) with production (prepared long turns); for writing it may mean eliciting written-spoken language (interaction: email, SMS, personal letter) as well as prose (production: essay, report). For listening and reading it may mean some short pieces for identifying specific information (listening/reading for orientation) and one or two longer pieces for detailed comprehension.

The formulation of criteria may or may not be related to the descriptors in Chapter 5 of the CEFR (2001), which are summarised in CEFR (2001: Table 3). However, the criteria should be balanced in terms both of extent of knowledge and degree of control and of linguistic competence and pragmatic competence, as CEFR (2001: Table 3) attempts to do. The assessment instrument might be a single grid of categories and levels, such as CEFR (2001: Table 3), especially for standardisation training or a programme in which teachers teach classes at different levels. Alternatively, it might focus only on the target level, with one descriptor per chosen category; a simple example is given in Table 1. The advantage of this approach is the ease with which the criteria can be explained to learners. This makes it easier to highlight the competences they must acquire for communicative success, rather than just focusing on lists of things they can do.

Table 1

Assessment at one level

Note: 3 is the mark given if the learner exactly meets the criterion-descriptor; no more and no less.

3.6 Validation

Validation has two aspects: internal validation of the intrinsic quality of the assessment and external validation of the claimed link to the vertical continuum of real-life language ability operationalised in the CEFR descriptor scales. For reasons of space I shall only discuss the latter, since the entire language testing literature concerns the former. Many of the moderation techniques referred to above are simple forms of external validation: the fundamental principle is to exploit collateral information and independent sources of evidence. The advice in the Manual is to use two independent methods of setting the cut scores between levels. Then, if necessary, one can use a cyclical process of adjusting the cut scores, examining them in the light of a decision table like that shown in Table 2 in order to arbitrate between two provisional results. The table shows a low-stakes (Eurocentres) worked example cited in the Manual (Council of Europe 2009: 111–113). Here the pattern was very regular, with 73.5% matching classifications, so no correction from the provisional cut scores set for the item bank on the basis of item-writer intention seemed necessary.

Table 2

A CEFR Manual ‘Decision Table’ for validation of cut-scores on a Eurocentres item bank for German (North Reference North2000b)

This contrastive technique can be exploited in many different ways: for example, contrasting the original claim based on item-writer intention against the result from formal standard setting (=ways of setting the cut scores); contrasting the results from two independent standard-setting panels; contrasting the results from two different standard-setting methods (e.g. between a test-centred and a candidate-centred method), and finally confirming the result from standard-setting (or the original claim based on item-writer intention) with a formal external validation study. For teacher assessment, the external criterion could be operationalised in CEFR-related examination results for the same students; for a test under study the external criterion could be ratings by CEFR-trained teachers of the same students in relation to CEFR descriptors. In fact, many of the Manual case studies recently published (Martyniuk Reference Martyniuk2010) did successfully use two methods in order to confirm their claim to CEFR linkage. Both the ECL study (Szabo Reference Szabo2010) and the TestDaf study (Kecker & Eckes Reference Kecker and Eckes2010) contrasted original item-writer intention with formal standard-setting; both the City & Guilds study (O'Sullivan Reference O'Sullivan2010) and the ECL study (Szabo Reference Szabo2010) contrasted the mean average difficulty of their own items with that of the illustrative items; both TestDaf study (Kecker & Eckes Reference Kecker and Eckes2010) and the Bilkent COPE study (Thomas & Kantarcioğlu Reference Thomas and Kantarcioğlu2009; Kantarcioğlu et al. Reference Kantarcioğu, Thomas, O'Dwyer and O'Sullivan2010) contrasted panel-based standard-setting results with external teacher judgements of the candidates in relation to CEFR descriptors. The Surveylang study (Verhelst Reference Verhelst2009) contrasted results from a sophisticated data-based panel Cito bookmark method (Council of Europe 2009: 82–83) with external teacher CEFR judgements. Finally, both the Pearson Test of English – Academic (De Jong Reference De Jong2010) and the Oxford On-line Test (Pollitt Reference Pollitt2009) also contrasted item-writer intentions with external teacher judgements.

In contrast to these sensible approaches, Cizek & Bunch (Reference Cizek and Bunch2007), the current US text book on standard-setting, explicitly advises against using two methods of standard-setting, because these might yield different results. They state that ‘a man with two watches is never sure’ and ‘use of multiple methods is ill advised’ (Cizek & Bunch Reference Cizek and Bunch2007: 319–320). Yet replication is the basis of Western academic thought: if you cannot replicate a result you do not have a result. Good practice would dictate corroboration of what is, for a high-stakes test, an important decision that will affect many people's lives.

4. Criterion-referencing and standard-setting

This reluctance to question the decision of a single panel highlights a general confusion about standard-setting and criterion-referenced assessment. Standard-setting is very often undertaken by a panel of experts who estimate the difficulty level of items in order to set the cut-score for pass/fail or different grades in a test. In EALTA, this conventional approach to standard-setting seems to be considered essential for relating assessments to the CEFR. Eli Moe, for example, in her paper at the EALTA standard-setting seminar in The Hague began by saying:

However, neither I myself nor Neil Jones nor John De Jong, to name but three people present, would agree that panel-based standard-setting is a must when linking tests to the CEFR. The preliminary Manual (Council of Europe 2003) made it clear that it was perfectly feasible to jump from the specification phase direct to the kind of empirical, external validation discussed in the previous section without bothering with panel-based standard-setting at all. It recommended using the judgements of CEFR-trained teachers for validation and presented box plots and bivariate decision tables provided by Norman Verhelst, the Cito statistical expert from the DIALANG project, as useful tools in that process. When I met Norman at the first meeting of the Manual group, we actually had a one-hour discussion in which I expressed my difficulty in buying the idea that someone with his experience of Item Response Theory (Rasch modelling, henceforth IRT) could seriously believe that such guesstimation by panels really worked.

Validating the relationship of a test to the CEFR requires what is technically known as linking to the continuum of ability acting as the criterion. In our current discussion, the validated CEFR descriptor scale provides that continuum. Criterion-referenced assessment (CR) places persons on such a continuum, independent of the ability of others. This is in contrast to norm-referenced assessment, in which the ability of candidates is evaluated relative to that of their peers, or to a standard set in relation to their peer cohorts over many years. CR was developed by Robert Glaser in a seminal article from which the crucial passage is the following:

This is not at all where the conventional, US-style standard-setting represented by Cizek & Bunch (Reference Cizek and Bunch2007) is coming from. In the US, CR started in the 1960s, at almost exactly the same time as the behaviourist instructional objectives movement, so unfortunately the two concepts merged in setting the performance standard for mastery learning in the US minimum competence approach (Glaser Reference Glaser1994a: 6; Reference Glaser1994b: 9; Hambleton Reference Hambleton1994: 22). Such a performance standard is actually a norm: a definition of what it might be reasonable to expect from a newly qualified professional, or from a ninth-year high school student in a specific subject in a specific context. Over time that norm-referenced standard became confused with the criterion – which is supposed to be the continuum of real-world ability. This is an important point, because it means that the referencing of the assessment became entirely internal; the link to the continuum of ability in the area concerned in the world outside had been lost. Standard-setting in North America then became the process of setting the pass/fail norm for minimum competence on a multiple-choice test assessing a given body of knowledge in the subject concerned for the particular school year or professional qualifying exam.

Since it was the subject experts (panel of expert nurses; committee of ninth-year teachers) who defined that body of knowledge, they were also in a position to give an authoritative judgement on whether the test was fair. Fairness relates to what such experts feel it is reasonable to expect from a specific cohort of candidates in relation to the closed domain of knowledge concerned. Whether an individual's result is considered to be good or bad therefore depends entirely on how that result relates to the score set as the expected norm for their cohort. This is fair enough. However, it is neither criterion-referenced assessment nor proficiency assessment in the sense in which the word ‘proficiency’ is used in the expression language proficiency. There is no relationship to an external criterion: the continuum of ability. The referencing is purely internal: to what is fair; to what was taught.

As Jones (Reference Jones2009: 36) pointed out in his paper at the EALTA standard-setting seminar, there really is almost nothing in common between setting such a pass norm for a closed domain of subject knowledge, on the one hand, and linking a language test to the continuum of language proficiency articulated by the CEFR, on the other. In addition, as Reckase (Reference Reckase2009: 18) suggested in his EALTA presentation, panel-based methods were not designed for the multiple cut-scores necessary for linking results to different language proficiency levels; there is an inevitable dependency between the decisions.

Nevertheless, 23 out of the 26 articles on case studies of relating tests to the CEFR in Martyniuk (Reference Martyniuk2010) and Figueras & Noijons (Reference Figueras and Noijons2009) took a panel-based standard-setting approach, mostly citing Cizek & Bunch (Reference Cizek and Bunch2007). Fortunately, as mentioned earlier, many also replicated their panel-based findings with a second method. However, the predominance of panel-based standard-setting demonstrates the extent to which many language testers and many people involved in linking assessments to the CEFR are not aware of the confusion between criterion-referencing and mastery learning described above, nor that panel-based standard-setting is a norm-referencing technique, nor that it is not innately suitable as a means for setting multiple cut-scores on a test. Nor are many language testers aware that there is 30 years of literature suggesting that such panel-based standard-setting is flawed, even within its own context (e.g. Glass Reference Glass1978: 240–242; Impara & Plake Reference Impara and Plake1998: 79).

These problems with estimations by panels have recently been rediscovered in an EALTA context (Kaftandjieva Reference Kaftandjeva2009) in the evaluation of the so-called basket method used in the DIALANG project and included in the preliminary, pilot version of the Manual. The basket method is one of many variants on the classic Angoff Method of standard-setting through estimation of item difficulty by a panel. Whereas the Angoff Method asks panellists to estimate percentages, the Basket Method takes a simpler approach. It asks each panel member to decide which basket (A1, A2, etc.) to put the item in, by posing and answering a question like ‘At which CEFR level will a candidate first be able to answer this question correctly?’ Many variants of the Angoff method feed data to panellists between rounds of estimation. Usually data is provided on item difficulty (facility values or IRT theta values) and then on impact (‘How many people would fail if we said this?’) and the provision of such data was in fact recommended in the preliminary Manual. As Kaftandjieva (Reference Kaftandjeva2009: 30) indicates, such a modified basket method works much better. But to my mind this approach really amounts to little more than an exercise in damage limitation. If people cannot accurately guess the difficulty of items without being given empirical data, why not use the empirical data to determine difficulty in the first place? If, in order to avoid excessive strictness or leniency and arrive at a sensible result, panellists need data on what percentage of the candidates fail as a result of their guesswork, can one place any faith at all in the judgements?

This point is illustrated by the attempts made by ETS to relate TOEFL to the CEFR. After a 2004 panel-based standard-setting project to relate ETS exams to the CEFR using a classic Angoff method (Tannenbaum & Wylie Reference Tannenbaum and Wylie2004), TOEFL reported that 560 on the paper-based TOEFL (PBT), the equivalent of 83 on the internet-based test (iBT), was the cut-score for C1 (ETS 2004). On their website, ETS currently report equivalences based on a second panel-based study (Tannenbaum & Wylie Reference Tannenbaum and Wylie2008) and state that iBT 57–86 (PBT 487–567) is B1 (ETS 2008). That is to say, according to the guesstimates of the first panel, PBT 560–567 is B1 and according to the guesstimates of the second panel, PBT 560–567 is C1. A similar switch occurs with the claimed equivalences for TOEIC. Common sense, the corroboration technique from the CEFR Manual illustrated in Table 1, the comparative scores that ETS publish on their website for IELTS (IELTS 6.5 = iBT 79–93/PBT 550–583) and Eurocentres' institutional experience of working with IELTS, TOEFL and the CEFR would all suggest that 560–567 might well be somewhere between the two results, i.e. B2. But Tannenbaum & Wylie are not interested in corroboration. They fail to mention the results of the first study, though they refer to its existence (Reference Tannenbaum and Wylie2008: 11); they cite Cizek & Bunch's (Reference Cizek and Bunch2007) dismissal of comparing two results as grounds for not undertaking an external validation study using teacher ratings of CEFR proficiency to obtain convergent evidence, as recommended in CEFR Manual (2008: 30). They even see no need to demonstrate through the specification procedures recommended in the CEFR manual that TOEFL has any content link to the CEFR either (2008: 3).

The ETS approach to CEFR linking was heavily criticised at the EALTA meeting in Sitges in 2007 because it completely ignored the procedures recommended in the Manual (specification and external validation in addition to conventional standard-setting) and their own experience of linking TOIEC to the American ILR (Interagency Language Roundtable) scale through ratings of candidates with descriptors from the scale. This criticism appears to have prompted the second study but not, unfortunately, an understanding of the need to corroborate claims, lay open findings and resolve contradictions. Above all there is a complete failure to appreciate that calibrating to a common criterion requires a different approach from traditional US methods for setting a pass score on a test. In a context in which the UK Border Authority is using the CEFR to set the proficiency threshold needed in order to receive a student visa to undertake higher education in the UK, knowing whether TOEFL 560–567 is B1, B2 or C1 is not an academic matter.

5. Calibrating to a common criterion

The fundamental problem is that the panel-based approach normally estimates the difficulty of the items in a single form of the test by a single panel. Yet examination institutes should be relating their reporting scale to the external criterion provided by the CEFR descriptor scale so as to guarantee the link over different test administrations. They should not be relating items on one particular test form on the basis of the views of one particular panel. This is essentially the problem with the TOEFL projects. Best practice in linking a high-stakes test to the CEFR involves calibrating the scale behind the test or suite of tests to the CEFR with what is technically called vertical scaling or vertical equating using IRT. Simple introductions to IRT are offered by Baker (Reference Baker1997), McNamara (Reference McNamara1996) and Henning (Reference Henning1987).

Cizek & Bunch (Reference Cizek and Bunch2007), however, devote just 7% of their text to the issue of standards at different stages on a continuum of ability – only to then reject the concept. They discuss what they describe as vertically moderated standard-setting (VMSS), which is a way of smoothing out infelicities when stringing together a series of norms for different school years, each determined independently by standard-setting panels. They conclude that ‘none (of the VMSS methods) have any scientific or procedural grounding to provide strong support for its use’ (Cizek & Bunch Reference Cizek and Bunch2007: 297). Vertical scaling to a continuum of ability (IRT) they reject out of hand on the basis of a study by Lissitz & Huynh (Reference Lissitz and Huynh2003). Yet Lissitz & Huynh cited six specific reasons why vertical scaling with IRT was inappropriate for their context. Only one of them – the fact that it is technically complicated – actually applies to the context of relating language assessments to the CEFR.

There is a literature on linking assessments, and Angoff's (Reference Angoff and Thorndike1971) article – which initiated conventional panel-based standard-setting – was part of this literature. It was entitled ‘Scales, norms and equivalent scores’. The so-called Angoff standard-setting method was in fact a remark in a footnote. That footnote was written 15–20 years before computer development made practicable the scaling and the establishment of equivalent scores that IRT promised. But there were other equating methods before IRT became widespread.

Before becoming involved in the development of the Manual, I wrote a modest article (North Reference North2000b) which described how over the years various people in Eurocentres had addressed the question of equating tests and linking them to the Eurocentres scale of proficiency, a precursor of the CEFR descriptor scale. I had read most of the then standard-setting literature in bibliographic research before the development of the CEFR descriptor scale, but didn't see how panel-based, judgemental methods were relevant to a common framework scale of levels, except for rating spoken and written samples. Even then it seemed clear that the many-faceted variant of IRT scaling (Linacre Reference Linacre1989, Reference Linacre2008) would be needed to handle inconsistency and subjectivity on the part of the experts operating as raters. This was the approach applied – after extensive qualitative research on the descriptors – in the CEFR research project (North Reference North2000a) and in calibrating the CEFR illustrative spoken samples (North & Lepage Reference North and Lepage2005; Breton, Lepage & North Reference Breton, Lepage and North2008).

I certainly think that the experience of participating in standard-setting seminars is a very enriching one. It is good and very valuable practice for a team of test developers and item writers to consciously evaluate and judge the difficulty of items and then be confronted with empirical data on item difficulty. As Moe (Reference Moe2009: 137) suggests, this process may also help make the levels more concrete by teasing out their criterial features. But why use such guesstimation to set cut-scores? There is a data-based alternative that exploits vertical scaling and the assessments of learners by CEFR-trained teachers. As mentioned above, the technique has been used in several CEFR linking projects (Oxford Online Placement Test: Pollitt Reference Pollitt2009; Pearson Tests of English: De Jong Reference De Jong2010; the UK Languages Ladder project: Jones et al. Reference Jones, Ashton, Walker and Martyniuk2010; the TestDaf study: Kecker & Eckes Reference Kecker and Eckes2010; and the European Survey of Language Competence: Verhelst Reference Verhelst2009). The technique is explained in the Further Material (North & Jones Reference North and Jones2009) provided to accompany the Manual, which is buried in the small print on the Council of Europe's website (www.coe.int/lang), sandwiched between the link to the Report of the CEFR Forum underneath the presentation of the Manual and the text introducing the Reference Supplement. I thoroughly recommend it to you.