2.1 Representations of the L2 construct

Theoretical representations of communicative L2 ability provide a reasoned basis and useful starting point from which to define the construct of interest, design appropriate assessment tasks, and justify the score-based inferences and uses of tests. Bachman (Reference Bachman, Fox, Wesche, Bayliss, Cheng, Turner and Doe2007) characterizes prevailing theoretical representations of the L2 construct as ‘ability-focused, task-focused, and interaction-focused’, and he submits that while they ‘are not mutually exclusive’, it may not be possible to reconcile them – at least theoretically – in part because they are based on different ‘values and assumptions’ (p. 41). Nonetheless, construct-related research that investigates the nature of L2 ability constitutes some of the most fundamental research to be pursued in language assessment.

For several decades, the consensus view has been that communicative L2 ability is multicomponential, consisting of a set of highly interrelated subcompetencies that interact in a particular language use situation (Purpura Reference Purpura, Shohamy and Hornberger2008), although it is unclear exactly how many factors are involved, how they are interrelated, and how they affect language performance. Therefore, empirical studies that substantiate and elucidate the subcompetences believed to underlie the nature of L2 ability not only contribute valuable evidence to support theoretical representations of the construct but also provide crucial support for the design and use of particular L2 assessments.

The investigations of both the paper-based (PBT) and internet-based (iBT) versions of the Test of English as a Foreign Language™ (TOEFL®) are a case in point. Based on a review of studies that explored the factor structure of the TOEFL, Sawaki, Stricker & Oranje (Reference Sawaki, Stricker and Oranje2009) concluded there was empirical support for the multicomponential perspective, although the number and combination of factors identified across the different studies varied. Moreover, in a confirmatory factor analysis of a prototype of the iBT, Sawaki et al. found the factor structure of the new test to be best represented by a higher-order factor model that included a general factor (L2 ability) and four group factors, each related to one of the parts of the test (Listening, Reading, Speaking, and Writing). In another study of the same sample of test-takers, Stricker & Rock (Reference Stricker and Rock2008) found the same factor structure reported by Sawaki et al. across subgroups, despite differences in the study participants’ number of years of classroom instruction in English and their first language. Because these studies were conducted on a prototype of the iBT TOEFL, studies of an operational version of the test using a representative sample of the iBT test-taker population are warranted. Additionally, if similar studies were conducted on other L2 assessments, the results would expand the available evidence for or against the multicomponential model and possibly further explicate the contributions of various factors to the construct.

Of course, factor-analytic studies represent only one form of quantitative evidence and one approach to investigating language assessments. Test developers recognize the need to utilize multiple forms of evidence, collected throughout the test development and implementation process. While quantitative procedures have often been emphasized in test validation, qualitative procedures are increasingly being used as well.

Survey methods (questionnaires and interviews), discourse and conversation analysis, verbal report (or recall) protocol analysis, and observations are among some of the most frequently cited qualitative procedures, and researchers have employed them in a number of recent studies of international assessments of L2 English ability. For example, Lazaraton (Reference Lazaraton2002) used a conversation analysis procedure to collect and analyze interlocutors’ discourse on the Key English Test (KET) and the Certificate in Advanced English (CAE), and Galaczi (Reference Galaczi2008) used the procedure to investigate the oral discourse produced by paired test-takers on the speaking component of the First Certificate in English (FCE). TOEFL researchers used discourse analysis procedures and verbal report analysis protocols to explore the academic language ability construct and the knowledge, processes, and strategies used by iBT test takers (Brown et al. Reference Brown, Iwashita, McNamara and O'Hagan2005; Cohen & Upton Reference Cohen and Upton2006; Cumming et al. Reference Cumming, Kantor, Baba, Erdosy and James2006). Orr (Reference Orr2002) employed the verbal protocol procedure to investigate rater behaviors on the speaking performances of FCE test takers. In a recent International English Language Testing System (IELTS) study, Wigglesworth & Elder (Reference Wigglesworth and Elder2010) used discourse analysis and survey data (questionnaires and group interviews) to gain insights into the effect of planning time on test takers’ speaking performance in Part 2 of the IELTS speaking section. Although each of these studies investigated assessments of L2 English ability, the research designs are relevant to investigations of other L2 assessments.

In an effort to better understand the nature of L2 ability, researchers have examined a number of the underlying constructs thought to contribute to it. Among these are the knowledge, skills, and cognitive processes L2 users engage when listening, reading, speaking, and writing in a language use situation or when responding to language assessments. For example, Swain et al. (Reference Swain, Huang, Barkaoui, Brooks and Lapkin2009) investigated strategic competence, which is considered to be a key aspect of communicative language ability, by examining test takers’ reported strategy use in response to tasks on a modified version of the speaking section of the iBT. The researchers used a questionnaire and an immediate recall procedure (following the completion of each of six assessment tasks) to collect data from test takers, and the results of the study offer some empirical support for the role of cognitive processes in the L2 speaking construct. Weir et al. (Reference Weir, O'Sullivan, Yan and Bax2007) also used an immediate recall procedure following each task on a practice version of the IELTS, to investigate the cognitive processes and strategies test takers reported using in response to tasks in the reading section.

The results of these studies offer empirical evidence of the processes and strategies elicited by certain assessment tasks, and provide additional insights into the L2 reading and speaking ability constructs. They also represent approaches one can undertake with other L2 assessments. In fact, without the results of investigations of more L2 assessments conducted in more contexts, it will be difficult to attain a complete understanding of the contributions of the factors involved in interactive L2 communication, or how they interact.

In the past decade, some scholars have conceptualized L2 ability as a set of hierarchically arranged communicative functions, situations, topics, and acts and the relative competence displayed by L2 users across a range of proficiency levels. This perspective is exemplified in the Common European Framework of Reference (CEFR). The CEFR emerged from a European initiative intended to facilitate transparency and coherence among those engaged in language education by creating a common scheme for communicating, organizing, reflecting on, monitoring, and promoting L2 learning, teaching, and assessment (North Reference North, Taylor and Weir2008: 22). However, its influence now extends far beyond Europe. Given the global impact of the CEFR and some of the concerns raised by assessment specialists, it is important to continue to investigate the framework and to accumulate construct-related support for its use (for critiques of the CEFR, see, for example, Fulcher Reference Fulcher2004, Weir Reference Weir2005b, Hulstijn Reference Hulstijn2011, and Little Reference Little2011).

One concern with the CEFR has to do with the proficiency level descriptions and the adequacy of the illustrative scale descriptors for some competences in the framework. To address these limitations, North (Reference North, Taylor and Weir2008) advocates undertaking investigations in local contexts using a process that collects and categorizes samples of performances and test items across proficiency levels, compares the data to CEFR descriptor scales, and uses the results to support descriptors or justify modifications to them based on empirical evidence.

Since ability in one component of the L2 construct can differ from the knowledge and skill displayed in another and because competence in different components (such as linguistic versus strategic competence) can vary in terms of both the rate of L2 development and the level of attainment, it is important to understand more fully the contribution of various components to overall L2 ability and how they interact in different language use contexts. Research that addresses the following task will contribute additional insights into the nature of L2 ability and may provide construct-related evidence for the CEFR.

CEFR designers intended the framework to be language- and context-neutral and capable of being adapted to meet local circumstances. While a number of studies have reported on its use within a European context, there are limited published accounts of how it has been applied or adapted for use in other regions. Wu & Wu (Reference Wu, Wu and Martyniuk2010) describe a project undertaken in Taiwan that sought to align a locally developed L2 assessment with the CEFR, based on the approach advocated in a draft document distributed by the Council of Europe (2003). Yet without more case studies of how the CEFR has been used in other contexts and how locally developed L2 assessments have been effectively aligned with the framework, it will be difficult to empirically support claims that the CEFR is truly capable of being adapted to contexts anywhere in the world. A related but somewhat different issue has to do with how much modification can reasonably occur in applications of the CEFR to local circumstances before L2 standards cease to be comparable or assessments cease to be aligned with the CEFR. Studies that respond to the following research task will help address these issues and also provide some of the validity evidence needed to justify the use of local L2 assessments.

The context of language use figures prominently in current representations of the L2 construct, including the CEFR. Brown (Reference Brown2008) has speculated that the language ability construct may actually be composed of ‘a cluster of constructs that vary from setting to setting depending on variations in purposes, approaches, syllabuses, testing methods, constraints, and so on’ (p. 288). If this is correct, it is important to conduct studies proposed in research task 4 using frameworks that promote systematic and comprehensive analyses of the test use contexts and facilitate identification of the most salient features in the context. For a detailed description of a context analysis conducted on a local assessment of L2 English ability (EIKEN) used in Japan, see Brown (Reference Brown2008).

Given the increased use of performance-based tasks in L2 assessment, it is important to have empirical evidence both to inform the design and to justify use of the tasks. At present we have an incomplete understanding of how task and test taker characteristics affect task performance or how the various factors related to these characteristics interact and contribute to performance (Bachman Reference Bachman2002). Recent research has also revealed the influence of assessors on test takers’ performances and some of the factors that appear to affect the awarding of scores on performance-based tasks. Research task 5 proposes investigations that will increase understanding of key issues related to performance-based task assessment. To date, most studies have focused on a few high-stakes tests of L2 English ability, and it is important to explore the use of these tasks in a broader range of L2 assessments and contexts, including their use in large-scale national and international testing programs and in classroom-based assessment.

Unfortunately, there are too few accounts of how task characteristics and test taker characteristics affect performance on assessments designed for local assessment purposes. A recent example of research that explores this is Gan's (Reference Gan2010) school-based case study of test takers’ performance on a group oral assessment. Gan used a conversation analysis procedure to investigate the discourse that emerged from higher- and lower-scoring groups of L2 learners on a classroom-based group oral assessment. Data from the study revealed features of the interactional discourse produced by groups with different levels of L2 proficiency and provided empirical support for use of a group oral assessment task. See the 2009 special issue of Language Testing (26.3) for current approaches to investigating paired and group assessments of L2 performance and Taylor & Wigglesworth's overview of key issues.

In another example of a local study with implications for L2 classroom-based assessment, Matsuno (Reference Matsuno2009) used multifaceted Rasch measurement to compare self- and peer-ratings of L2 writing performance with teacher ratings. In addition to offering some empirical support for the use of peer assessment of classroom-based writing tasks, the study represents a research design that could be used to investigate assessor effects on other types of performance-based tasks and L2 assessments. Studies that investigate the extent to which L2 learners can function effectively as assessors of L2 performance have implications for efforts to empower learners and to integrate assessment alternatives into L2 classroom practices. Research task 6 seeks to collect empirical evidence to support the use of practical classroom-based task designs and assessment procedures.

2.2 Validation theory and practice

As Chapelle (Reference Chapelle1999) observed, there are numerous obstacles to advancing work in validity, including the lack of adequate models for defining the L2 construct, the difficulty of building a validity argument to justify use of an assessment, and the paucity of practical frameworks to help test developers and users translate validation theory into concrete actions that improve assessment practices. Despite these challenges, current standards of professional practice compel test developers to justify (validate) the interpretations and uses of test scores. Much of the recent work related to validation theory in L2 assessment has been informed by Kane (see Kane Reference Kane and Brennan2006 and Reference Kane2010 for descriptions of an argument-based approach to validation), Mislevy and colleagues (Mislevy, Steinberg & Almond Reference Mislevy, Steinberg and Almond2002), Bachman (Reference Bachman2005), and Weir (Reference Weir2005a), among others.

The case study of the TOEFL iBT project compiled by Chapelle et al. (Reference Chapelle, Enright and Jamieson2008) exemplifies the current argument-based approach to validation. In addition to rendering a comprehensive, longitudinal account of the validation activities undertaken to support the design and the interpretive and validity arguments for the new TOEFL test, the case study presents one of the most detailed accounts of L2 test development available. Bachman & Palmer (Reference Bachman and Palmer2010) proffer a framework designed to guide practitioners in the development of L2 assessments and the arguments used to justify the inferences and uses of scores. The assessment use argument that emerges from the Bachman & Palmer framework incorporates recent theoretical developments in validation and the pattern of logical reasoning and argument structure of Toulmin (Reference Toulmin2003). Finally, Weir (Reference Weir2005a) has integrated recent developments in validation theory into an evidence-based approach to test development and validation based on six key considerations (components): test taker, cognitive validity, context validity, scoring validity, consequential validity, and criterion-related validity. His socio-cognitive framework focuses on the collection of multiple forms of complementary evidence for each of the six components of the framework throughout the test development and implementation process. The accumulated body of validity evidence is used to support claims about the quality and usefulness of the assessment. Chapter 11 of Weir (Reference Weir2005a) presents a number of illustrative applications of the socio-cognitive framework, as well as a variety of quantitative and qualitative procedures available for the investigation of L2 assessments.

Since the frameworks described above are relatively new, there are limited reports of their impact on language assessment practices. Therefore, case studies that examine how these frameworks have been implemented in different contexts and how they have contributed to the validation practices associated with local L2 assessments will establish the extent to which validation theory is capable of promoting improvements in validation practices. Several recently published accounts suggest that these frameworks are affecting the development of local L2 assessments. For example, Pardo-Ballester (Reference Pardo-Ballester2010) reports on how Bachman's (Reference Bachman2005) assessment use argument framework was used to construct the validation argument for a local placement test of L2 Spanish listening ability, and Llosa (Reference Llosa2011) describes a study that built a validity argument for the interpretation and use of student performances on a classroom-based assessment that was aligned with state-mandated L2 proficiency standards. Llosa's multi-method research design employed confirmatory factor analysis and a verbal protocol procedure to obtain data and determine if a standardized measure of students’ L2 proficiency assessed the same construct as a classroom-based assessment that was based on teacher judgments.

The global movement in standards-based L2 teaching, learning, and assessment has led to a need for locally developed standards-based classroom assessments; however, there are limited published case studies of empirically validated standards-based assessments. Thus, studies of how teachers have applied appropriate validation frameworks to the development of standards-based classroom assessments will not only reveal the extent to which existing validation frameworks can be adapted or applied to the development of standards-based classroom assessments but will also establish an empirically based justification for the use of the assessments. As a number of scholars have emphasized, however, the validation of classroom-based assessments may require new ways of looking at key test qualities such as reliability and validity (McKay Reference McKay2005; Rea-Dickens Reference Rea-Dickens, Shohamy and Hornberger2008; Davison & Leung Reference Davison and Leung2009). Research task 7 is intended to determine the impact of validation frameworks on L2 assessment practice and to encourage the validation of locally developed L2 assessments.

Current approaches to validation emphasize the need to collect multiple forms of evidence to justify test use to stakeholders. In order to gauge the impact of validation theory on the professional activities of assessment specialists, Cizek, Rosenberg & Koons (Reference Cizek, Rosenberg and Koons2008) reviewed the test descriptions contained in the 2005 edition of the Mental measurements yearbook (Spies & Plake Reference Spies and Plake2005). The researchers concluded that the reported validity information did not conform to current views of validity theory and that consequential validity evidence was essentially absent. A comparable study of the test descriptions of assessments of L2 English ability or other major L2 assessments, as proposed in research task 8, would establish a baseline from which to monitor subsequent progress in the adoption of professional standards relative to validity evidence, and would document the effect of developments in educational measurement on the field of language testing.

2.3 Technology

The application of computer technology to large-scale assessment offers great promise, but raises a number of concerns as well. One fundamental issue is how the application of computer-assisted language testing (CALT) affects test taker performance and, more narrowly, whether test taker performances on paper-based and computer-based versions of the same language test are comparable. While the narrower question has been investigated in several studies conducted on large-scale assessments of L2 English ability, several assessment specialists have called for more studies that explore the extent to which there might be group differences in performance on computer-mediated assessments (Chapelle & Douglas Reference Chapelle and Douglas2006; Leeson Reference Leeson2006; Douglas & Hegelheimer Reference Douglas and Hegelheimer2007). Douglas & Hegelheimer (Reference Douglas and Hegelheimer2007) posited that computer familiarity may be a factor for some groups of test takers, and it is incumbent on test developers to collect evidence that test taker characteristics such as gender, first language, geographical region, or socioeconomic status do not lead to significant differences in the degree of familiarity with computer technology, and that this in turn is not a source of unmotivated variation in test performance.

A study conducted by Choi, Kim & Boo (Reference Choi, Kim and Boo2003) represents a multi-method approach to investigating the comparability of scores on paper- and computer-based versions of the same L2 assessment. The researchers collected empirical evidence of comparability by using a variety of means, including content analysis, correlational analyses, analysis of variance, and confirmatory factor analysis. In addition to providing empirical support for the claim that performances on two versions of the same assessment were comparable, the study represents a research design that can be replicated in other contexts where both computer-based and paper-based versions of the same L2 assessment are used.

Weir et al. (Reference Weir, O'Sullivan, Yan and Bax2007) used a multi-method design to investigate the performances of test takers on a paper and a computer-mediated version of IELTS writing tasks. The results produced empirical support for the claim that the test method in this case does not significantly affect assessors’ ratings of performance and also supported the claim that test takers engage in similar cognitive processing when responding to paper-based and computer-mediated versions of the same task. Additionally, based on study participants’ questionnaire responses, test takers were found to be generally familiar and comfortable with computers. However, given the characteristics and number of participants in the study, the researchers recommended replicating it with a larger and more representative sample of test takers. In another study that explored computer familiarity and the effect of test method on performance, Yu (Reference Yu2010) reported minimal differences in the performances of test takers on an integrated reading and writing task and the test takers’ degree of computer familiarity did not appear to affect performance. While the three studies cited above demonstrate some empirical support for the use of computer-mediated versions of tests, they need to be complemented by studies that examine large representative samples of the test taker population for a given assessment. In order to ensure the validity of score inferences for an assessment, it is also important to investigate the effects of all the task types used in an assessment and to determine the extent to which paper and computer-mediated versions elicit the same knowledge, skills, cognitive processing, and oral and written discourse from test takers. Studies undertaken in response to research task 9 will contribute empirical evidence to justify the use of computer-mediated assessments of L2 ability.

Advances in technology make it possible to include a broader range of tasks in computer-mediated assessments than was possible even a few years ago. Today a variety of media are available to enhance the authenticity of a task, but the media input (e.g. video, audio, and graphics) may also increase the processing demands placed on test takers and possibly compromise their performance. A study by Wagner (Reference Wagner2007) investigated how much time test takers spent viewing video input on listening assessment tasks, and he found the participants spent an average of two-thirds of the allotted time watching the video input. In another part of the same study, Wagner (Reference Wagner2010) compared test takers’ performances on a listening assessment based on whether the task provided video or only audio input, and found that the group that received the video input scored higher and the difference between the two groups was statistically significant. These data offer some preliminary empirical evidence of how test method characteristics and technology-enhanced media input affect test takers’ performance, but the issue remains largely uninvestigated. Studies undertaken in response to research task 10 will offer insights into this aspect of computer-mediated assessment.

In a study of computer-mediated assessment, Wise & Kong (Reference Wise and Kong2005) sought to investigate test takers’ motivation. If test takers are not motivated to put forward their best effort on a test, it undermines test score validity. Empirical investigations of test motivation have typically relied on self-report data collected from test takers during or after the test. In contrast, Wise & Kong used the principle of response time effort (RTE) as an indicator of test takers’ motivation and analyzed the amount of time test takers spent on test items. The researchers then combined this quantitative data with self-report data they collected from test takers. This study demonstrates how a multi-method research design using complementary data collection procedures can collect empirical evidence of how test taker motivation and test method characteristics (computer-mediated test) affect performance on an assessment. Because this issue has not been explored with regard to L2 assessments, research task 11 proposes investigating it.

2.4 Consequences of assessment

The terms washback and impact are often used to describe studies that investigate the consequences associated with tests and test use. While precise definitions of the terms remain somewhat elusive, the emerging view is that impact is an overarching concept that refers to the broad influences of tests on individuals, educational programs and systems, and society at large. washback, on the other hand, is more often used to describe the effects of assessment on language teaching and learning (Hamp-Lyons Reference Hamp-Lyons1997; Hawkey Reference Hawkey2006).

Several assessment specialists have proposed conceptual frameworks for investigating the impact of language assessments. Chalhoub-Deville (Reference Chalhoub-Deville2009) argues that impact studies need to begin before the language assessment is introduced rather than after the fact, which is commonly the case. She advocates conducting a social impact analysis (SIA) and contends that this will more effectively integrate key stakeholders into the process of considering the consequences of test use and testing practices. The testing-context analysis framework proposed by Brown (Reference Brown2008) also emphasizes the need to integrate stakeholders into a process that includes data collection and analyses of test use consequences. Given the limited number of reports on how these frameworks have been implemented, research task 12 proposes that case studies be undertaken to demonstrate the efficacy of these frameworks.

Few longitudinal impact studies have been conducted on major L2 assessments. Two notable examples are the multi-phase impact studies conducted on the IELTS and iBT tests. The IELTS impact study employed a multi-method approach to collect survey (questionnaire and interview) and observation data from informants in order to determine the influence of the test on stakeholders in the teaching-learning context as well as those beyond it. Data from the three-phase study and from other impact studies commissioned by the IELTS research program have contributed consequential validity evidence to justify use of the test.

iBT researchers also adopted a multi-method approach to investigate the impact of the new TOEFL. They utilized survey methods (questionnaires and interviews) and classroom observations to obtain data from key stakeholders (teachers, students, and program administrators), and used the data to document the classroom teaching and learning practices in a sample of Central and Eastern European contexts prior to the introduction of the test. The data also revealed how test users acquired information about the iBT, how well they understood the new TOEFL, and how they responded to the new features of the assessment (Wall & Horák Reference Wall and Horák2006; Wall & Horák Reference Wall and Horák2008). The third phase of the impact study was designed to collect information on classroom teaching and learning following the introduction of the iBT and to support claims that the test fulfilled its intended purpose. Both the iBT and IELTS impact studies represent designs that can be adopted for longitudinal studies of other L2 assessments.

One aspect of test impact that has not been sufficiently explored is the extent to which test users are able to understand the information disseminated about the test by the test publisher. Based on a review of the test reports distributed by large-scale assessment programs, Goodman & Hambleton (Reference Goodman and Hambleton2004) called for more empirical studies of how information about major tests is communicated, to score users and their ability to understand the information. The researchers suggested that think aloud protocols could be used to gain insights into how certain subgroups interpret and understand reports. Test publishers can use the results to improve and tailor the information disseminated to test users based on the characteristics of different test user groups. Studies undertaken in response to research task 13 will provide a form of consequential validity evidence not currently available for L2 assessments.

Most of the research on test impact has focused on the effects of major national and international L2 assessments. This has led some to call for more studies that explore the impact of assessment on individual students and classroom learning (Purpura Reference Purpura, Taylor and Weir2009; Cheng & Curtis Reference Cheng, Curtis, Coombe, Davidson, O'Sullivan and Stoynoff2012). In fact, very little is known about how different assessments and assessment practices affect individual test takers’ cognitive processing and their acquisition of key L2 competences. Consequently, there is a need for studies that investigate the effects of assessment on students’ learning and L2 development; such research will not only provide consequential validity evidence to support the use of specific local L2 assessments but will also foster stronger connections between assessment practices and L2 learning.

The language-testing community has devoted considerable attention to both professional standards and ethics. While progress has been made in codifying and promulgating guidelines intended to promote the development of robust tests and the appropriate use of test scores, there is a paucity of empirical research regarding the effect of professional standards on professional practice. Hence, studies undertaken in response to research task 15 will provide preliminary evidence of the degree to which test developers and test users report they are familiar with professional standards and acknowledge the influence of those standards on how they conduct their work (Alderson Reference Alderson2006).

test fairness concerns receive major treatment in the standards and codes of practice disseminated by authoritative professional groups in the international language testing community. The centrality of fairness to language assessment has led several researchers to propose frameworks to facilitate its consideration (Kunnan Reference Kunnan, Milanovic and Weir2004, Reference Kunnan, Taylor and Weir2008; Xi Reference Xi2010). Using the iBT as an example, Xi (Reference Xi2010) demonstrates how a fairness framework can be used to identify, order, and integrate various kinds of empirical evidence into a fairness argument that supports ‘the comparability of the score-based interpretations and uses for different relevant groups’ (p. 158). Recent articles by Cheng & Qi (Reference Cheng and Qi2006) and Shih (Reference Shih2008) demonstrate how Kunnan's fairness framework can facilitate consideration of large-scale national assessments of L2 English ability. However, there is a need for reports on how fairness frameworks can be integrated effectively into the entire validation process from the design of the assessment through its implementation and continued use, and research task 16 addresses this.

Recently, fairness concerns have been raised about large-scale international assessments of L2 English ability. In particular, some assessment specialists believe the tests may be biased in favor of native speaker norms and have called for more empirical investigations to ascertain if this is indeed the case (Chalhoub-Deville & Wigglesworth Reference Chalhoub-Deville and Wigglesworth2005; Davidson Reference Davidson, Kachru, Kachru and Nelson2006; McNamara & Roever Reference McNamara and Roever2006; Hamp-Lyons & Davies Reference Hamp-Lyons and Davies2008). In a study conducted by Hamp-Lyons & Davies (Reference Hamp-Lyons and Davies2008) on the Michigan English Language Assessment Battery (MELAB), the researchers sought to determine if there is evidence the MELAB favors English native speaker norms and that the test is biased against certain non-native speakers’ English. Because of limitations in the design of the study (sample size, rater characteristics, rater selection, and text selection), the researchers were unable to draw any definitive conclusions and they recommended additional investigations. Moreover, they argued it is important to continue to examine ‘whose norms should be imposed in these tests and what the consequences for test takers are if the norm imposed by the test is not the “normal” variety accepted in their own society’ (p. 26). Taylor (Reference Taylor, Taylor and Weir2009) discusses some of the theoretical and practical considerations teachers and testers face with regard to language standards and language variation and the results of a survey of ALTE members on this issue (Taylor Reference Taylor, Taylor and Weir2008). Given the significant consequences of major international assessments of L2 ability and the limited number of empirical studies of this issue, investigations related to research task 17 are warranted.