2.1 Background

In 2001, the Ministry of Education promulgated the Experimental Program of Compulsory Education Curriculum, stating that EFL courses should be generally offered from Grade 3 in primary schools in China. In fact, students in the more affluent cities have even begun to learn English in kindergarten. Normally, students in China receive at least four years of compulsory EFL education at elementary level (Grades 3–6), three years each at junior high (Grades 7–9) and senior high levels (Grades 10–12), and at least one year at tertiary level.

Accompanying the 11 years of EFL learning are numerous English tests, to which great importance is attached by test stakeholders, as the tests are, more often than not, closely linked with high-stakes decisions such as access to higher education and opportunities for better employment or career advancement. Despite the long history of tests and a testing-oriented culture, the establishment of language assessment as an independent field in applied linguistics in China, however, came quite late. It was not until the late 1990s that researchers began to devote considerable efforts to developing valid and reliable tests for the huge test-taking population in China (Jin, Reference Jin2010). This was almost a decade behind the emergence of language assessment as a field of applied linguistics in the 1980s (Bachman, Reference Bachman2000). Late, maybe, but the past decade has witnessed a flourishing interest in language assessment in the field of applied linguistics in China, especially after hosting the 30th Language Testing Research Colloquium (LTRC) by Zhejiang University in 2008. Chinese scholars also enjoy an increasingly high profile in the international language assessment community, evidenced in the growing number of presentations from Chinese scholars at the annual LTRC, as well as the increasing number of publications by Chinese scholars in recognized international journals such as Language Testing and Language Assessment Quarterly.

2.2 High-stakes EFL tests in China

In China, the function of English tests as a gatekeeper for higher education or better employment are primarily realized through the operations of the National Education Examinations Authority (NEEA), which was established in 1987, in response to the call for an official and unified testing system in China. The NEEA, as the only education examinations authority under the Ministry of Education, the People's Republic of China, oversees almost all the high-stakes education examinations in the country, including EFL tests at various levels and for different purposes, such as the National Matriculation English Test (NMET), the Graduate School Entrance English Examination (GSEEE), the College English Test (CET), and the Public English Testing System (PETS).

The NMET is one of the three compulsory sub-tests in the university entrance test battery, the other two being the Chinese test and the math test. The university entrance test battery is often nicknamed the ‘Footslog Bridge’ in China, namely, ‘the only route to academic success via a long and laborious walk or march’ (Cheng & Curtis, Reference Cheng, Curtis, Cheng and Curtis2010, p. 267). The primary purpose of the NMET is to measure test takers' English language proficiency, based on which university admission decisions are made, together with scores in other subtests. The NMET, introduced in 1985, is a norm-referenced test, administered to high school graduates wishing to attend colleges and universities in China. The number of test takers was about 3 million in the early 1980s (Cheng, Reference Cheng2008) and reached 10.31 million in 2019. As one of the two highest-stakes English tests in China, the NMET exerts a great influence on all stakeholders and it is no exaggeration to say that ‘the test influences how society evaluates the schools, and how schools, parents and students evaluate the teachers’ (Cheng, Reference Cheng2008, p. 19). Over the past years, there have been several rounds of reform in the NMET administration and syllabus for various reasons. One recent reform has to do with the Decision on Major Issues Concerning Comprehensively Deepening Reforms, adopted at the close of the Third Plenary Session of the 18th Communist Party of China Central Committee in November 2013, which stipulated that educational equality should be vigorously promoted in China. One of the initiatives is to give students multiple test-taking opportunities in the NMET, aiming to eradicate the drawback of ‘one's fate being determined by one single examination’. Since 2014, Zhejiang Province and Shanghai have been selected as pioneering areas (Liu, Reference Liu2017), where students take the NMET twice a year and the higher scores are used for admission decisions. The reform signals that a comprehensive evaluation and diversified college admission mechanism is to be established, where students have multiple opportunities in the college entrance examinations.

The GSEEE is another high-stakes test designed and administered by the NEEA for the purpose of selecting candidates for Master's programs in educational and research institutions in China (He, Reference He, Cheng and Curtis2010). Since its first administration in 1989, the GSEEE, as a norm-referenced English language proficiency test, has played a significant role in promoting graduate education quality. Test takers need to score above the cut-off points set for the majors they choose in order to be admitted into the programs. The results of the GSEEE, therefore, determine the career path for most test takers and even change the life prospects for some. Test takers of the GSEEE consist of two groups, namely, students who are in the final year of their undergraduate programs, and past students who graduated one or more years ago. The test-taker population of the GSEEE has been on the rise and reached a record high of 2.38 million in 2018, which is in alignment with the increase in the number of students enrolled in colleges and universities from 1999 onwards. In contrast to most English tests at tertiary level, the current format of the GSEEE does not include a listening section, which was removed in 2005. The unexplained removal of the listening section, however, seemed to have brought a negative washback effect on EFL listening teaching and learning and withdrawn attention from language educators and researchers.

In addition to the two highest-stakes English tests in China, the CET, a test boasting the largest number of test takers in the world, has caught the most attention from Chinese researchers. The CET is a national English language test for college and university students in China, administered by the National College English Testing Committee, authorized by the NEEA. The purpose of the CET is to provide an objective evaluation of college and university students' overall English proficiency in accordance with the National College English Teaching Syllabus and bring a positive washback effect to EFL teaching and learning at the tertiary level in China (Jin & Yang, Reference Jin and Yang2006). It is a test battery consisting of two levels, the CET Band Four (CET-4) and the CET Band Six (CET-6), as well as their corresponding CET-Spoken English Test (CET-SET). It is administered twice a year, generally in June and December. Since its inception in 1987, the CET, as a criterion-related norm-referenced test (Yang & Weir, Reference Yang and Weir2001), has attracted a growing number of test takers, rising from 0.1 million in 1987 to an annual test-taker population of nearly 20 million in recent years. The target test takers of the CET-4 are second-year college and university students, though some students, especially those in key universities, may take the CET-4 in their first year. The CET-6 is available to college and university students who have passed the CET-4. Regardless of their performance, students can take the CET multiple times until their graduation. The CET, albeit not a high-stakes entrance test like the NMET and GSEEE, has gained increasingly higher stakes as more and more uses, mainly unintended, are made of the test. For instance, the CET-4 certificate was one of the graduation requirements for obtaining a bachelor's degree in some universities until the Ministry of Education banned this practice in 2005. In the job market, the CET-4 certificate has become a prerequisite for employment in some companies. In educational settings, the CET is reported to have exerted an enormous washback effect, both positive (Gu, Reference Gu2004) and negative (Cheng, Reference Cheng2008), on English language teaching and learning in tertiary education.

The PETS is a national English language testing system, jointly developed by the NEEA and the University of Cambridge Local Examinations Syndicate and first administered in 1999. It is a criterion-referenced testing system, incorporating five distinct but coherent levels ranging from Level 1, the level of English expected at junior high school, to Level 5, the level of English required of people aiming to study or work abroad. The PETS is available to all English learners, regardless of age, profession or academic background, with an annual test-taker population of 1 million (Liu, Reference Liu, Cheng and Curtis2010). The purpose of the PETS is to promote English learning nationwide (Cheng, Reference Cheng2008), but test results of some levels of the PETS are also used for high-stakes decisions. For instance, scores on the listening section of PETS Level 2 (PETS-2), in some provinces, are used as an indicator of English listening ability for university admission decisions, together with scores in the Matriculation English Test.

In addition to these tests designed and administered by the NEEA, there are some other well-established English tests in China. For instance, the Test for English Majors (TEM), administered by the National Advisory Commission on Foreign Language Teaching, on behalf of the Higher Education Department of the Ministry of Education, is a national English language test specifically targeted at English major undergraduates in Chinese colleges and universities. As a criterion-referenced test, the purpose of the TEM is to measure English majors' language proficiency, in accordance with the teaching requirements stipulated by the Teaching Syllabus for English Majors in China (Zou, Reference Zou2003) and promote English language teaching and learning among English majors. The TEM, officially launched in 1992, is a test battery, which comprises TEM Band Four (TEM-4), TEM Band Eight (TEM-8), and two corresponding oral tests, i.e. TEM-4 Oral and TEM-8 Oral. TEM-4 is administered to English major undergraduates at the end of their second year, while TEM-8 at the end of their fourth year. The test-taker population of the TEM-4 and TEM-8 increased to 459,000 in 2010 from 13,167 in 1992 when the test was first administered (Jin & Fan, Reference Jin and Fan2011), and this figure has remained stable in recent years. Since its inception, the TEM certificate has enjoyed wide social recognition, with the TEM-8 certificate in particular, being regarded as a document certifying that the EFL learner has reached the highest English proficiency level in China.

In November 2016, the English Test for International Communication (ETIC), developed by Beijing Foreign Studies University, was officially launched. The intended use of the ETIC is to provide a reference for personnel recruitment and talent in China and, in particular, assist in the selection of urgently needed talents with global vision and expertise in intercultural communication. The ETIC is administered twice a year, in May and November respectively. The impact of the ETIC is not apparent yet, as it is relatively new, with no relevant studies published.

2.3 China's standards of English

The numerous national or provincial English tests, some of which are discussed above, are designed in accordance with different standards for different purposes. One problem continually bothering English educators and researchers in China is the lack of a common scale which various tests can be aligned to. Additionally, in language teaching, more and more teachers are concerned about the overlaps and inconsistencies in the EFL curriculum requirements at different educational stages in China. Recognizing that a common ability standard for reference is the key to maintaining efficient and coherent teaching and assessment practices, China's Standards of English (CSE) project was officially launched by the NEEA on September 30, 2014. This is also an active response to The Implementation Opinions on Deepening the Reform of the Examination and Enrollment System, issued by the State Council on September 4, 2014. Since then, over one hundred language teaching and assessment researchers have been working on the development of the CSE, which encompasses nine levels (1–9) of English proficiency. More than 30,000 teachers and 130,000 students in 28 provinces, municipalities and autonomous regions in China have been involved in the large-scale questionnaire survey for the scaling of descriptors.

The two major breakthroughs of the CSE, as compared to the Common European Framework of Reference for Languages and other international standards and frameworks, is its emphasis on the cognitive aspect of language proficiency, and the development of the three subscales of pragmatic competence, interpreting and translating. By drawing on the revised model of Bloom's taxonomy of educational objectives (Anderson, Krathwohl, & Bloom, Reference Anderson, Krathwohl and Bloom2001), the CSE has highlighted the development of learners' cognitive ability, including lower-level abilities such as identifying and retrieving, and higher-level abilities such as summarizing, analyzing, evaluating and critiquing. In addition, the pragmatic ability subscale, hopefully, will promote a better understanding of language in use as well as the teaching and learning of cross-cultural communication in China, while the subscales of interpreting and translating, the first of its kind, will make an original contribution to the language framework literature in the world.

A number of studies have recently been published, introducing various aspects of the development and validation of the CSE, including the theoretical framework of CSE (Liu & Han, Reference Liu and Han2018), the development of subscales of listening (He & Chen, Reference He and Chen2017; Min, He, & Luo, Reference Min, He and Luo2018), speaking (Jie & Jin, Reference Jie and Jin2017; Jin & Jie, Reference Jin and Jie2017), reading (Zeng & Fan, Reference Zeng and Fan2017), writing (Deng & Deng, Reference Deng and Deng2017; Pan, Reference Pan2017), translating (Bai, Feng, & Yan, Reference Bai, Feng and Yan2018), interpreting (Wang, Xu, & Mu, Reference Wang, Xu and Mu2018) and pragmatic competence (Han & Huang, Reference Han and Huang2018). The CSE was released in early 2018 and became effective on June 1, 2018, which is intended to guide or even regulate the English language teaching, learning and assessment for all educational stages in China.

Using the CSE as a reference, the NEEA has initiated the project of developing the National English Testing System (NETS). It is expected that the NETS will provide a full-skill testing system of English language proficiency, with the test results used to inform decisions in teaching, evaluation, admission and employment, etc. The NETS will be a series of five criterion-referenced English language tests, in alignment with Levels 3–7 depicted in the CSE, named NETS-3, 4, 5, 6 and 7. NETS-5 and NETS-6 are currently being developed, whose testing syllabus will hopefully be released in late 2020.

3.1 Test reliability and validity

One of the seminal issues in language assessment research is validity, which is defined as ‘the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests’ (AERA, APA, & NCME, 2014, p. 11). In other words, validity, including reliability, is grounded in score interpretation and use (Chalhoub-Deville, Reference Chalhoub-Deville2016), which has been a constant topic for Chinese researchers in the past eight years. Before discussing individual studies, we would like to underscore three general observations here.

1. 1. The studies selected (e.g. Peng & Zou, Reference Peng and Zou2012; Li, Reference Li2013; Tai, Reference Tai2015; Zhu, Reference Zhu2017) have documented evidence regarding the reliability and validity of the large-scale English tests mentioned above, namely, NMET, GSEEE, CET and PETS. However, the number of empirical studies on high-stakes English tests is far from enough, given the long history of these tests as well as their critical role in education in China. More efforts are needed to offer justification of these English tests such as NMET and GSEEE to contribute to more transparency of high-stakes tests in China.

2. 2. Most empirical studies on the validation of high-stakes tests were conducted by researchers in higher education institutions, with very few published by staff working at educational examinations authorities at provincial and national levels. This stands in sharp contrast with the large number of papers published in international journals like Language Testing and Language Assessment Quarterly by staff from international testing organizations such as the Educational Testing Service. This is understandable given the different foci and functions of the organizations, especially when considered within the different social, cultural and political contexts where these tests are developed and administered.

3. 3. Researchers favored quantitative approaches to study the validity and reliability of language tests, such as Many-facet Rasch Modeling (MFRM) and confirmatory factor analysis (CFA) (e.g. Sun, Reference Sun2011; Peng & Zou, Reference Peng and Zou2012; Li, Reference Li2013; Wu, Reference Wu2017), but there are also studies that have experimented with the use of other methodologies like conversation analysis.

General test. One important approach of validating score-based interpretations of a test is to examine whether the internal structure of the test matches the construct definition and whether the internal structure remains invariant across groups. Taking a longitudinal approach, Zhu (Reference Zhu2017) examined the construct assessed by five test formats of CET-4 over a span of 30 years using exploratory factor analysis (EFA). The different factor structures across years led him to reach a conclusion that the CET-4 is transitioning from measuring more distinct language abilities to integrated language ability, highly consistent with the change of philosophy of contemporary College English teaching in China. This study serves as one of the earliest attempts in China to track the shift of a criterion-related language test in orientation to the development of language teaching across three decades, although we believe CFA may constitute a better method to explore whether the internal structure is consistent with the construct definition of the test over long periods. Extending EFA to CFA, Fan, Ji, and Yu (Reference Fan, Ji and Yu2014) demonstrated how factor structure and factorial invariance evidence can be effectively utilized to support the construct validity of an in-house English proficiency test. This study represents an initial effort by Chinese scholars to provide empirical support for the current practice of reporting a single composite score as well as component scores for each section of the test, although attempts to justify such score reporting practices were made for some international EFL tests a decade ago.

Speaking. Apart from the validity research focused on large-scale tests as a whole, there are studies of speaking, writing, listening, and language elements in other settings. For speaking, Sun (Reference Sun2011) examined the reliability and validity of a trial speaking test for prospective secondary school English teachers by using generalizability (G-)theory and MFRM. The complementary use of G-theory and MFRM in examining performance assessment constitutes a general trend in the international language assessment community, as each method helps researchers gauge the consistency of score-based interpretations from a different perspective, the former at an aggregate group level and the latter at an individual level. The concern for high validity of speaking assessment was echoed by Liu (Reference Liu2015), which has provided empirical evidence to support the claim of consistency of assessment records and meaningfulness of score-based interpretations of the listening-based retelling task in the NMET within Bachman and Palmer's assessment use argument (AUA) framework. Considering the consensus of adopting the argument-based approach to validation (Kane, Reference Kane2013), studies of this nature have particular strength and make some methodological contribution to the language assessment literature. In a qualitative study, L. Tang (Reference Tang2014) employed conversation analysis to examine Chinese test takers' interaction characteristics while taking the Business English Certificate oral examination. Although failure to associate the interaction patterns and test scores was one of the limitations of the study, in-depth qualitative analyses of the test outcome in this study indeed contributed to a better understanding of Chinese test takers' communication strategies in the interaction process. More studies like this are needed to encourage language assessment researchers to adopt rigorous qualitative methods to complement the current dominant trend of using quantitative methods in validation studies.

Writing. Integrated writing tasks have been gaining popularity in high-stakes assessment programs for academic purposes, as well as in classroom contexts. The issue of construct validity of this task type is the focus of two studies. Li (Reference Li2013), adopting a mixed-method approach, did a validation study of the summarization task as a read-to-write integrated writing task. Li's (Reference Li2013) dissertation is outstanding in that it proposed a model of summary writing in assessment context, by drawing on participants' think-aloud protocol and interview data. Furthermore, his finding that genre exerted a significant influence on test takers' summarizing processes as well as products could shed light on future design of the summarization task in the NMET, an area that has been largely unexplored. Wu (Reference Wu2017), in a similar study, validated an integrated read-to-write task using MFRM and interview data. The author pointed out that source verbatim use was the most difficult for test takers, as compared to content and language. This bears some similarity with the general perspective that source verbatim use constitutes a major challenge for raters when scoring test takers' essays in integrated writing tasks.

Listening. What gap-filling or outline tasks in listening comprehension assessment truly measure has been a highly controversial research issue, due to the seemingly confounded construct of the ability to comprehend oral input and to use orthographic knowledge to spell the word. Tai (Reference Tai2015) revealed that note-taking quality accounted for more than 50% of the variance of the mini-lecture task score in TEM-8, indicating that note-taking is an important part of the construct measured by the mini-lecture task in academic setting. Wang (Reference Wang2014) took a different approach, collecting data on test takers' cognitive processes while listening to an academic mini-lecture mainly through think-aloud verbal protocols and questionnaires. The findings confirmed the current task format of a structured summary as a valid task to measure academic listening ability. Although we agree that tapping into test takers' cognitive process by using think-aloud protocols and questionnaires constitute important validity evidence for the task, listening while doing think-aloud may have altered the test-taking process itself. The recent application of non-invasive techniques, like functional near-infrared spectroscopy (Ansaldo, Kahlaoui, & Joanette, Reference Ansaldo, Kahlaoui and Joanette2012) in language studies, when used in conjunction with verbal protocol, may help us better explore what cognitive processes are measured by listening assessments.

Language elements. Tests of grammar and vocabulary knowledge have also drawn attention from researchers. Peng and Zou (Reference Peng and Zou2012) employed content analysis, as well as sophisticated quantitative analyses like MFRM and CFA, to investigate the construct validity of the Grammar and Vocabulary Section of TEM-4. The results not only pointed to the high construct representation and relevance of the test, but also revealed the high correlation and thus high degree of overlapping between grammar knowledge and vocabulary knowledge traits. The findings have not only consolidated our current understanding about different components of organizational competence but may also shape future theoretical conceptualizations. In a similar way, C. Zhang (Reference Zhang2015) investigated the construct validity of the Grammar Gap-filling Task in the Matriculation English Test in Guangdong using Rasch analysis. This study has reiterated how psychometric methods based on item response theory (IRT) can be incorporated in the investigation of validity of language tests.

To sum up, this stream of research has contributed to our understanding of the validity and reliability of EFL tests developed and used in China. There are, however, several issues that remain to be more systematically and rigorously researched.

1. 1. Researchers have been almost exclusively focusing on the score-based interpretations of the test, rather than the various decisions that are made based on the test, although it is generally agreed that it is not the test itself or the test scores that should be validated, but rather the proposed interpretation and use of the test that should be justified (Bachman & Palmer, Reference Bachman and Palmer2010; Kane, Reference Kane2013). Given that multiple uses, intended or unintended, are being made of some large-scale language tests in China, it is essential for test developers and researchers to work together to justify the uses of tests and test scores, so that more beneficial consequences of using the test can be achieved.

2. 2. The validation studies have been primarily conducted on the test instrument itself instead of its relation to real-world scenarios. While it is important to see whether test scores serve as good indicators of language ability, more research is needed to investigate whether scores can predict real-life language performance, or in Chapelle, Enright, and Jamieson's (Reference Chapelle, Enright and Jamieson2008) term, whether scores could be ‘extrapolated’ beyond the test itself to the real world.

3. 3. Applications of psychometric analyses should certainly continue to be encouraged to illuminate the score-based interpretations of language tests. More rigorous qualitative methods, however, need to be relied on to shed light on the cognitive validity of language tests in China.

3.2 Factors affecting test performance

This category covers factors affecting performance and scores on language tests, including test methods, test-taker characteristics, test-taking processes and test-taking strategies. The test methods, other than the ability being measured, that affect score interpretation and use have long been an issue of concern in the field of language assessment. In the selected studies, the effect of test methods on test performance and score interpretation has been investigated in a variety of ways, such as task type (He & Min, Reference He and Min2012; Zhang & Zhang, Reference Zhang and Zhang2014; Xu, Reference Xu2018), genre type (Li, Reference Li2014), prompt characteristics (He & Sun, Reference He and Sun2015), input material characteristics (Kong, Wang, Zhou, & Chen, Reference Kong, Wang, Zhou and Chen2011; Peng & Zhang, Reference Peng and Zhang2013; He, Chen, & Min, Reference He, Chen and Min2018) and item format (Gao, Reference Gao2014). We believe there are two broad patterns that are worthy of attention.

1. 1. Most of the studies selected (e.g. Gao, Reference Gao2014; Li, Reference Li2014; He et al., Reference He, Chen and Min2018; Xu, Reference Xu2018) concentrate on pinpointing the factors that influence test takers' performance as well as score-based interpretations of the test, while a few endorse an expanded view of validity (e.g., Huang, Reference Huang2012; Jin & Cheng, Reference Jin and Cheng2013), providing alternative ways of viewing the relationship between these factors and test constructs.

2. 2. While there is much research (e.g. Kong et al., Reference Kong, Wang, Zhou and Chen2011; Peng & Zhang, Reference Peng and Zhang2013; Li, Reference Li2014; He & Sun, Reference He and Sun2015; Xu, Reference Xu2018) on the effect of test method on performance and scores on language tests, comparatively little attention (e.g. Gao, Reference Gao2014) has been paid to the cognitive and metacognitive processes that test takers were engaged in when responding to test items and tasks.

Effect of task type. A plethora of studies have examined the effects of different task types on test performance. For instance, by employing a within-group design, He and Min (Reference He and Min2012) revealed that students of low- and high-proficiency levels performed significantly better in an integrated listening-reading-writing task than in an independent writing task, while no significant statistical difference was reported for the intermediate proficiency level. This was later echoed by Zhang and Zhang (Reference Zhang and Zhang2014), which provided further evidence that integrated writing tasks had a facilitating effect on high-level students in terms of language complexity, fluency, expression of thought, and organization. Although studies of this nature are particularly important in the Chinese educational context, where integrated writing tasks are increasingly used to replace independent writing tasks in high-stakes tests such as TEM-8, PETS-3 and PETS-4, these two studies limited their observations to score comparison and surface-level textural feature analyses, failing to examine test takers' output in relation to source materials at a deeper level to reveal the underlying reasons for the interaction between task type and proficiency level. In her dissertation on the oral part of the TEM-4, Xu (Reference Xu2018) showed that monologues and dialogues generally elicited similar utterance features, but the two tasks led to discrepancy in terms of fluency and pronunciation error frequency. The finding that the performance of peer interlocutor affected the dialogic fluency and coherence is of particular importance in light of the prevalent trend of using dialogues or group orals as a task type in high-stakes large-scale speaking tests in China; beyond China, it is generally considered that more empirical evidence needs to be established to justify the meaningfulness and impartiality of score-based interpretations of the paired or grouped oral test.

Effect of genre type. In addition to task types, there is an interest in the influence of genre types and prompt characteristics on test takers' performance. Li (Reference Li2014) examined whether genre as a characteristic of the input material affected test performance in summary writing using a within-group design. The results of MFRM analysis revealed that generally test takers performed better in expository than narrative text summary writing. We agree with the authors that MFRM can be employed to put the difficulty parameter of each task type on the same scale, but we doubt that expository summary writing is easier than narrative summary writing, as there were plenty of other factors influencing the difficulty of the task which were not controlled for, such as information density and organization of the source text. In a quantitative analysis of 508 essays sampled from the ‘FLTRP CUP’ National English Writing Contest in China, He and Sun (Reference He and Sun2015) explored the effect of prompt characteristics on test takers' performance in integrated writing. What makes this study stand out from other empirical studies on prompt effect is that the researchers used Coh-Metrix to analyze in detail the effect of two prompt characteristics on the surface and deep linguistic features of test takers' output in integrated writing. The linguistic analysis revealed that different topic domains highlighted different textual features and different task specifications led to different modes of argument.

Input material characteristics. When it comes to the input material of listening, however, Peng and Zhang (Reference Peng and Zhang2013) argued that the linguistic features of listening text could not be the most effective predictors of item difficulty. The strength of this study resides in the fact that its analysis was based on a corpus of real test data of TEM-4 over four consecutive years. Nonetheless, the finding was not corroborated by a recent study undertaken by He et al. (Reference He, Chen and Min2018), which reported that syntactical complexity of the input text, in particular left-embeddedness, following item type, had the second highest predicting power for item difficulty. What distinguishes this study from other similar studies is that the listening items used in this study were from five different test forms targeted at EFL learners at different educational stages in China. The item difficulty estimates were scaled vertically across different forms and used as the dependent variable, which represented a wider range of ability levels. Extending from audio to video, a more authentic task resembling real-world language use, Kong et al. (Reference Kong, Wang, Zhou and Chen2011), found that content videos seemed to facilitate test takers' listening more effectively than context video and lower achievers benefited more from videos than higher achievers. These studies have shown that performance on listening tasks has been affected by the language input at the micro lexical and syntactic level, as well as by the mode of input at a more macro level. Language test developers should therefore consider the potential influence of these factors when designing the test and be able to justify their influence in relation to the realities of the target language use domain.

Effect of item format. Another area of focus is the effect of item format. Using methodological triangulation, Gao (Reference Gao2014) probed into the interaction between item types and test-taker viewing behavior in video listening tests. Analysis of the videotaped data of test takers' eye movements revealed that test takers tended to watch video when answering multiple-choice-question items and not watch video when completing blank-filling items. What makes Gao's (Reference Gao2014) study exceptional is that, rather than relying on score comparisons between video listening and equivalent audio-only listening assessments, he videotaped test takers' viewing behavior to analyze their test-taking process when completing different items types, contributing to a better understanding of the construct implications of incorporating video in listening assessment. A more in-depth approach, we believe, might be to utilize eye-tracking technology to explore the interactions between test takers, item types and visual cues at a more fine-grained level. An expansion of knowledge on these interactions will help test developers and item writers in their use of video in listening assessment in a more theoretically defensible and practically feasible way.

Test-taker characteristics. Two studies have investigated the influence of test-taker characteristics. Jin and Cheng (Reference Jin and Cheng2013) were concerned with the effects of test takers' psychological factors on the validity of high-stakes tests. Based on the finding that test takers' motivation and test anxiety accounted for 18% of the variance of their score in CET-4, the authors called for more attention to the influence of non-language factors on test performance. The study is extremely important in that it situates motivation and test anxiety within test-taker characteristics and underscores the importance of understanding the influence of test takers' psychological factors on test performance from a test validity perspective. Huang's (Reference Huang2012) study also contributes to this broadened view of validity, assessing whether different interventions on background knowledge exerted different influences on the reading processes and products. Unlike most background knowledge effect studies which investigate the relationship between background knowledge and test construct, Huang's (Reference Huang2012) study advocated that background knowledge should be viewed as part of the construct in classroom-based assessment, since students in instructional practice are typically provided with relevant background knowledge as a lead-in. These two studies stressed the need for language assessment research to consider the central place of test takers in language tests, because it is the test taker that we are drawing inferences about based on his or her performance.

Test-taking strategies. Test takers' strategy use, whether metacognitive or cognitive, has been long recognized as an important factor affecting test takers' performance on language tests. Li and Zeng (Reference Li and Zeng2011) employed SEM to examine the relationship among metacognitive strategies, cognitive strategies and scores in reading comprehension tests. Different from most previous empirical studies on the influence of strategy use on test performance, this study is significant in that it delineated the influence of strategy use on the readers' construction process and integration process at a more theoretically oriented level using the construction-integration model. Wu (Reference Wu2014) adopted a mixed-method approach to examine test takers' source use strategies in the process of performing a reading-to-write task. Through think-aloud protocols and follow-up questionnaire and interview inquiries, this study has highlighted the role of the input text as an agent to stimulate thinking, provide supportive evidence and serve as a language repository, adding significantly to our understanding of the construct of integrated writing assessment.

To summarize, while this body of research has deepened our understanding of the factors that affect performance in language tests from various perspectives, it is also clear that there are several areas in need of more attention and further research.

1. 1. The somewhat divergent findings on the interaction between test-taker characteristics and task types point to the need for more rigorous research in an area that has important implications for test design. Systematic meta-analyses, for instance, can be conducted to consolidate our current knowledge and enlighten future assessment practices.

2. 2. Although some researchers (e.g. Huang, Reference Huang2012; Jin & Cheng, Reference Jin and Cheng2013) have touched upon a broadened view of the validity in approaching the interaction between these factors and test constructs, it would be useful for there to be more work enriching the construct through greater engagement with the realities of the particular target language use context.

3. 3. Given the advent of new technologies like eye-tracking and functional near-infrared spectroscopy, more direct evidence concerning the test-taking processes and strategies test takers employ to respond to tasks should be collected to shed light on the test construct. Research is also urgently needed to examine whether the test-taking processes bear any resemblance to the processes that are needed to perform such a task in the target language use domain.

4. 4. Researchers have primarily focused on EFL learners at tertiary levels, with only one study incorporating learners at secondary levels (e.g. He et al., Reference He, Chen and Min2018). This indicates that our knowledge about the language assessment situation at primary and secondary levels remains limited. More empirical studies are needed to examine factors that affect performance on language assessments at different educational stages in China.

3.3 Rating and rating scales

Performance assessment has gained wide popularity in recent years, accompanied by increased interest in raters' rating quality, a central issue in the interpretation of outcomes from performance assessment. Empirical studies on this theme have primarily focused on the development of rating scales (Jiang, Wang, & Wang, Reference Jiang, Wang and Wang2012; Sun & Wei, Reference Sun and Wei2012; Liu, Mak, & Jin, Reference Liu, Mak and Jin2013; Zou, Reference Zou2017) and automated scoring models (Jiang & Wen, Reference Jiang and Wen2012; J. Wang, Reference Wang2013), highlighting the important role of rating scales and models in operationalizing the construct of performance assessment. Researchers have also displayed a strong interest in rater behaviors, including rater-scale interaction (Li, Reference Li2012; Liu & Lü, Reference Liu and Lü2015), rater judgment as a function of factors such as experience (Zhang & Peng, Reference Zhang and Peng2012) and amount of prior training (Li & Guan, Reference Li and Guan2016), as well as how raters arrive at their ratings (Chen, Reference Chen2013).This reflects the general view that raters play a central role in mediating the interpretation of test takers' performance in performance assessment. Before delving into individual studies, we would like to make three general comments on the empirical studies concerning rating and rating scales published in China.

1. 1. The dominant method used in studies here has been MFRM (e.g. Liu & Lü, Reference Liu and Lü2015; Li & Guan, Reference Li and Guan2016). The inclination towards the use of MFRM in evaluating rating quality is not surprising, given the strong support for this approach by researchers in the national and international language assessment research community (e.g. McNamara & Knoch, Reference McNamara and Knoch2012; Liu & Lü, Reference Liu and Lü2015) as well as its practical utility for quality monitoring in language assessment practice. However, it should be noted that the information that can be yielded by G-theory about the overall quality of rating should not be underappreciated.

2. 2. Inter-rater and intra-rater consistency or reliability coefficients have been reported in most selected studies as sufficient evidence for high rating quality (e.g. Li, Reference Li2012; Liu & Lü, Reference Liu and Lü2015), which is in alignment with the current state of research on rating quality published in international journals (Wind & Peterson, Reference Wind and Peterson2018). Very few efforts, however, have been devoted to triangulating the findings from statistical analyses of rating results with textual features of essays at different levels to support the meaningfulness of the score-based interpretations.

3. 3. In spite of initial rejections and skepticism in early days, there is a growing trend of investigating how automated scoring systems can be developed and used in assessing translation and writing in China (e.g. He & Wang, Reference He and Wang2012; Jiang & Wen, Reference Jiang and Wen2012; J. Wang, Reference Wang2013), with little attention to speaking.

Development of rating scales and automated scoring models. In recognition of the key role that rating scales play in operationalizing the construct, Chinese scholars have attempted – with different degrees of success – to develop new, and refine existing, rating scales. For instance, Jiang et al. (Reference Jiang, Wang and Wang2012) developed an analytic rating scale to evaluate both the meaning and form of language in English-Chinese translation tasks. It was later extended by J. Wang (Reference Wang2013), who, by employing a corpus-based approach, constructed an automatic evaluation model for English-Chinese translation. This study has demonstrated how approaches in corpus linguistics, a vital and thriving discipline, can be utilized to increase research in language assessment. The high correlation coefficients between automatic ratings and human ratings reported in the two studies, however, are not unexpected, as translation calls for a more fixed type of response in terms of content as compared to writing and is therefore presumably easier to be scored by automated scoring systems. Targeting the assessment of content quality in writing, Liu et al. (Reference Liu, Mak and Jin2013) proposed using the hierarchical decision-tree approach as a new rating method. This method diverges from holistic or analytic rating scales used prevalently in language assessment literature in that it draws upon Toulmin's argument structure (Toulmin, Reference Toulmin2003). It was found that the proposed method could significantly differentiate test takers at three proficiency levels and enjoyed comparatively higher inter-rater and test-retest reliability, as compared with those of traditional scoring methods. This study signifies a welcome step towards rating essays using alternative approaches, which may eventually contribute to the measurement of content in automated essay scoring systems, a thorny issue that needs to be tackled.

Notwithstanding the difficulty, there has been some interest in developing rating scales and constructing automated scoring systems for writing. Zou's (Reference Zou2017) dissertation is one of the few representative studies on how to design an analytic rating scale for writing in a real testing context, adopting a multi-phase, multi-method approach. Xu (Reference Xu2015) attempted to develop a new rating scale for a diagnostic writing test, in order to better inform teachers of their students' strengths and weaknesses. In contrast, in He and Wang's (Reference He and Wang2012) study, an automated essay scoring system was developed through the statistical model of multiple linear regressions, using grammar and basic writing styles as predictors. One major concern of this study lies in its potential construct underrepresentation. Only grammar and basic writing styles were taken into consideration when building the model, although the scores derived from the model make a relatively reliable prediction on the writing scores. This study, therefore, to some extent, revealed that currently, in EFL writing assessment in China, the construct of language ability is targeted at more than writing ability itself.

Rater-scale interaction. Three studies investigated the complex interactions between raters and rating scales in EFL speaking and writing assessments, underscoring the need to explore rater behavior at a more fine-grained individual level. Liu and Lü (Reference Liu and Lü2015) compared the use of analytic and holistic rating scales for a computer-based speaking test. The results showed that the analytic rating scale outperformed the holistic one. Consistent with these findings, Li's (Reference Li2012) dissertation, by exploring the interaction between raters and rating scales in essay rating processes and outcomes in the context of CET-6, reported that as compared to the holistic scale, the use of an analytic scale led to finer distinctions among test takers' writing ability, higher proportion of test takers with acceptable fit, smaller influences of rater severity variability on final scores, as well as more consensus reached on the understanding and application of the scale. Li's dissertation is extraordinary, not only in its meticulous analysis of both qualitative data, elicited through think-aloud protocols, questionnaires and semi-structured interviews, and quantitative data, utilizing both G-theory and MFRM, but also in its emphasis on the use of multiple data collection and analytical tools in examining the complex interaction between raters and rating scales in an operational CET-6 rating session. Focusing on two analytic rating scales for two speaking tasks, namely, reading aloud and oral presentation, Sun and Wei (Reference Sun and Wei2012) reported that, while no obvious problem was detected for the reading aloud rating scale, the oral presentation rating scale did not function as well as expected, with inverse threshold estimates on the two highest score categories. We endorse the authors' view that it may be attributed to the fact that the descriptors for that dimension of the scale are somewhat vague. However, we believe it might also be caused by the small number of raters involved as well as the lack of rating experience of the raters.

Factors influencing rater behavior. Rater behavior may also differ due to the influence of factors such as amount of experience (Zhang & Peng, Reference Zhang and Peng2012) and prior training (Li & Guan, Reference Li and Guan2016). By examining the differences and similarities between expert raters and novice raters in rating TEM-8 writing, Zhang and Peng (Reference Zhang and Peng2012) revealed that novice raters yielded more lenient ratings than expert raters. Additionally, some of the novice raters displayed interactional bias with some dimensions of the rating criteria, which was expected, as most of the novice raters were still pre-service English teachers. However, by involving pre- and in-service teachers as raters, this study, in some sense, contributes to the improvement of the English assessment literacy of pre-service and in-service English teachers, a topic frequently discussed and strongly advocated by language assessment researchers in recent years. By giving sufficient training, novice raters can rate as reliably as experts. Li and Guan's (Reference Li and Guan2016) study serves as a very good example to illustrate the effect of training on raters' rating behavior. By comparing the rating quality before and after training with MFRM, Li and Guan (Reference Li and Guan2016) concluded that positive effects were achieved in PETS-1 speaking rater training, where the exact agreement percentage between novice raters and expert raters increased significantly after training. These two studies highlight the importance of rater experience and training in maintaining high rating quality in performance assessment.

Rating process. Chen (Reference Chen2013) viewed essay rating as a problem-solving process and delved into the underlying cognitive structures raters established in solving the problem. Based on both quantitative analyses, such as MFRM, EFA and CFA, and qualitative analysis of raters' think-aloud protocol and follow-up interview data, the study proposed a framework of the rating process. The main contribution of this study is that, in addition to the five factors that have been repeatedly mentioned as influencing essay rating processes, namely, rating scale, rater characteristics, rater training, rater context and textual feature, it explicitly outlined rating supervision as the sixth principal factor influencing raters' rating.

In summary, the articles have facilitated our systematic understanding of the various factors that may affect rater's rating processes and outcomes in performance assessments. There are two further comments and recommendations that we would like to make about this group of studies.

1. 1. It is important to investigate how manifest variables like raters' prior training (e.g. Li & Guan, Reference Li and Guan2016) and experience (e.g. Zhang & Peng, Reference Zhang and Peng2012) have exerted influences on raters' rating performance. More research, however, is needed to examine how latent factors, like psychological traits, cognitive styles and rater beliefs, could account for differences in raters' rating performance. By delving into more internal factors and the decisive internal features characterizing good raters, more effective training at a more in-depth level can be conducted to bring the raters into a common ‘judgment community’ where they would share similar core beliefs about rating in a given test context.

2. 2. The findings that analytic rating scales outperformed holistic rating scales (e.g. Li, Reference Li2012; Liu & Lü, Reference Liu and Lü2015) are not particularly striking, as the former could be used to target at a finer-grained level of information. Most studies of this kind, however, have been conducted within an experimental design, which does not fully represent real operational rating sessions. Considering the large number of essays that need to be marked for large-scale standardized tests in China within a short period of time, the adoption of analytic rating scales seems less feasible. More endeavors, therefore, are warranted to investigate how essays are rated in real rating sessions of high-stakes language tests in China and what suggestions can be given to improve raters' rating performances under such circumstances.

3.4 Technology and language testing

Alongside the unremitting efforts to develop automated scoring systems reviewed above in the category of rating and rating scales, a plethora of studies, and arguably the largest number of publications related to technology and language testing, are focused on the impact of the use of automated scoring systems on EFL teaching and learning (Tang & Wu, Reference Tang and Wu2012; J. Tang, Reference Tang2014; Hu, Reference Hu2015; Zhang & Sheng, Reference Zhang and Sheng2015; Zhou, Reference Zhou2015; Lu, Reference Lu2016) and test takers' perceptions and attitudes towards the use of automated scoring systems (Li & Ni, Reference Li and Ni2017). Researchers have also been concerned about the comparability between computerized tests and their traditional counterparts (Dai, Reference Dai2011), the construct measured by computerized tests (Jin & Zhang, Reference Jin and Zhang2013), and design and validation of innovative computerized tests such as cognitive diagnostic ones (Meng, Reference Meng2013; Du & Ma, Reference Du and Ma2018) and adaptive ones (Min, Reference Min2012). There are three general observations that we would like to make here before discussing individual studies.

1. 1. Language assessment researchers in China have explored a number of high-priority issues regarding the use of automated scoring, such as how test takers interact with the feedback generated by automated scoring systems (e.g. Hu, Reference Hu2015; Lu, Reference Lu2016) and how the use of automated scoring systems affect language teaching and learning (e.g. Tang & Wu, Reference Tang and Wu2012; Zhou, Reference Zhou2015).

2. 2. It is noteworthy that some of the studies reviewed in this theme have reconceptualized computer technology as an integrated part of the language construct (e.g. Min, Reference Min2012; Jin & Zhang, Reference Jin and Zhang2013), which is in alignment with the expanded view of the language assessment construct in the new era. Such conceptualization is indispensable for researchers who want to look beyond the traditional test format and embrace new ways of operationalizing constructs and validating tests.

3. 3. The selected studies include some rigorous empirical research on attempts to develop and use more individualized computerized tests, such as computerized cognitive diagnostic tests (e.g. Meng, Reference Meng2013; Du & Ma, Reference Du and Ma2018) and computerized adaptive tests (e.g. Min, Reference Min2012). Such endeavors are indications of a growing awareness of the importance of personalization in language assessment.

Use of automated scoring systems. The question of how the use of automated scoring systems, in particular, the use of feedback generated by the systems, impacts EFL teaching and learning has been addressed by Hu (Reference Hu2015), who investigated how the feedback yielded by Pigai, a widely used automated scoring system in China, affected students' self-corrections and hence their writing products. It was reported that although the overall quality of the essays improved significantly in terms of essay length, low-frequency word use, word variety, average word length and number of clauses, no significant difference was observed regarding average sentence length. Using the same system, Pigai, Lu (Reference Meng2016) probed into how students' EFL writing processes may be affected by the feedback, by drawing on think-aloud protocols and stimulated recalls. The study is important in that it not only puts forward 20 strategies students have generally utilized in their three-phase writing process, but also delineates the factors that have affected students' writing processes, such as perceptions of the system, and the goals and nature of revision tasks. Rather than focusing on the effect of feedback on students' writing processes or products, Zhang and Sheng (Reference Zhang and Sheng2015) concentrated on the feedback itself. Perhaps most importantly, this study, by comparing and analyzing the feedback generated by Pigai for high-score and low-score CET-4 benchmarking scripts, confirmed a general observation that there was still room for improvement in terms of accurate identification of errors and provision of appropriate suggestions, despite an overall high satisfaction rate of the feedback.

In addition to feedback, there is an interest in conducting longitudinal studies to investigate how the overall use of automated scoring systems affect EFL teaching and learning in the classroom. In a mixed-method, one-semester longitudinal study, Tang and Wu (Reference Tang and Wu2012) investigated how the integration of Writing Roadmap (WRM), a web-based online essay scoring system, in college EFL classroom teaching enhanced students' writing ability and promoted learner autonomy. Although we have reservations as to whether the use of the system could swiftly change the teaching mode from attention to language accuracy to content, and from writing product to writing process and writing pedagogy, we are impressed with the model the authors proposed to integrate automated scoring systems into the teaching of writing. This was later applied by J. Tang (Reference Tang2014) to the EFL classroom in senior high schools, where content was found to be the least improved dimension for the experimental group after the use of WRM for one year. This finding indirectly pointed to a common problem of most automated scoring systems, namely, the lack of individualized diagnostic feedback on content. Specifically focusing on syntactic ability, Zhou (Reference Zhou2015) reported that the use of Pigai over a period of one semester significantly enhanced the experimental group's language complexity and fluency, with no obvious differences in accuracy. This study highlights the importance of examining syntactic ability as a core component of writing quality, yet the conceptualization of syntactic ability is primarily confined to discrete surface-level features like word length, number of T-units, number of clauses, amount of subordination and sentence patterns. We believe that a more systematic conceptualization of syntactic ability with a more fine-grained set of dimensions should be incorporated to facilitate a more in-depth understanding of the effect of the use of automated scoring systems on syntactic development and hence writing ability.

Test takers’ perceptions and attitudes. Relatively few studies have investigated the stakeholders' perceptions and attitudes towards the use of technology in language testing. On the basis of the data collected from questionnaire surveys administered to non-English major undergraduates, Li and Ni (Reference Li and Ni2017) reported generally positive attitudes towards the use of automated scoring systems. The benefits of this study lie in its exploration and confirmation of the factors underlying EFL learners' acceptance of new technology, namely perceived usefulness, perceived uniqueness and behavioral intention, using EFA and SEM. Nonetheless, the findings are primarily based on the students' responses to questionnaire surveys. Further effort is needed to collect more in-depth data from a wider range of stakeholders, such as teachers and school administrators, in order to find out how we can make full use of the benefits technology has to offer to language assessment.

Reliability and validity issues concerning computerized tests. Two studies have explored the reliability and validity issues concerning computerized tests. Dai (Reference Dai2011) concluded that the Computerized Oral Proficiency Interview (COPI) could replace the traditional Oral Proficiency Interview (OPI), on the basis of the high correlation coefficients between COPI and OPI results. Although the psychometric qualities of the two tests are compared rigorously, the issue of the underlying construct is not addressed at all. In a comparative study of PTE-Academic and IB-CET-6, Jin and Zhang (Reference Jin and Zhang2013) examined whether integrated tasks typically used in computerized tests led to a muddied construct validity of the assessment, despite being more authentic to target language use domain tasks. The findings substantiated the comparability of two tests in rank-ordering test takers at the test level but not at the section level. Their study succeeds not only in examining the comparability between two large-scale computerized tests, but goes a step further and connects the differences of construct of the two tests with the type of tasks used, thus providing a novel suggestion on using task-based score reports instead of traditional skill-based score reports for tests using integrated tasks.

Development of individualized computerized tests. Three studies have reported local attempts to develop and use more individualized computerized tests, including computerized cognitive diagnostic tests and computerized adaptive tests. Meng's (Reference Meng2013) study serves as a good example of utilizing cognitive diagnostic approaches (CDA) to develop an EFL listening diagnostic test with fine-grained diagnostic reports and corresponding remedial guidance. This was later expanded by Du and Ma (Reference Du and Ma2018), who combined tree-based regression analysis with CDA when constructing an EFL reading diagnostic test. What makes these two studies stand out from similar studies in language assessment literature is that the researchers not only consulted a number of experts in Q-matrix construction, but also elicited a large amount of data through student think-aloud protocols. This arguably makes the Q-matrix a better reflection of the real cognitive processes students may undergo when taking the listening and reading test, thus contributing to a more accurate diagnosis of test takers' strengths and weaknesses. In a mixed-method study, Min (Reference Min2012) made an attempt to design a computerized adaptive language test (CALT) to assess listening and reading proficiency in English using mixed-format with dichotomous and polytomous IRT models and investigated validity issues of the CALT under the AUA framework. This is one of the few representative studies which fully investigate the construct validity of a CALT, as well as the impartiality of the inferences made from test takers' scores in the CALT across different subgroups．

In summary, research on technology and language testing has boomed in the past decade and will continue to flourish in the years to come, given the extensive use of computer technology in the target language use domain and the surge in artificial intelligence (AI) technology in the world. A number of areas remain which need to be further explored.

1. 1. Researchers in China have been showing considerable interest in the rating quality of Pigai and Bingo, two locally developed automated scoring systems. These two systems, as well as other automated scoring models constructed in the studies reviewed here, however, have mainly used writing samples to train the algorithm to build the model, which is a type of statistics-based program, and stands in contrast with rule-based or knowledge-based systems where machine-learning methods without trained sets are relied on (Hoang & Kunnan, Reference Hoang and Kunnan2016). More studies, therefore, are needed to look at how latent semantic analysis and neural networks can be employed to design more intelligent automated scoring systems.

2. 2. More research is needed to provide a more independent and transparent comparison of the quality of different automated scoring systems. The users of automated scoring systems need to be better informed about how each feedback feature is designed, how the feedback is connected to the target ability, how accurate the feedback is and how the feedback can be used in different contexts.

3. 3. Following the reconceptualization of the language construct in computerized language tests, another welcome step would be the promotion of innovations in task design and assessment criteria, as the main benefit of using technology in language assessment is the expansion of constructs that cannot be measured in traditional pen-and-pencil language tests.

4. 4. Given the great breakthrough of AI technologies in recent years, a promising avenue for further research would be the use of AI to aid test developers, such as facilitated creation of test materials and automated item generation.

3.5 Test washback and impact

The term washback, referring to the influence of testing on teaching and learning, is usually used alongside impact by many language assessment researchers (Green, Reference Green2013). While washback studies have typically concentrated on the effect of testing on teaching and learning in the classroom at a micro level, impact research has expanded it to a larger context such as the educational system and the society at a macro level (Tsagari & Cheng Reference Tsagari, Cheng and Shohamy2017). Compared to other categories reviewed above, studies on test washback and impact in China are much fewer in number, and the dominant trend is investigating washback of large-scale high-stakes tests on EFL teaching and learning, including two CET studies (Guo & Li, Reference Guo and Li2012; Xiao, Gu, & Ni, Reference Xiao, Gu and Ni2014), two TEM studies (Xu, Reference Xu2012, Q. Xu Reference Xu2014), two NMET studies (Dong, Reference Dong2014; F. Zhang, Reference Zhang2015) and one GSEEE study (S. Xu, Reference Xu2014). Before looking at these individual studies, there are three general observations to be made.

1. 1. The majority of research on this theme conducted by Chinese researchers can be considered traditional washback studies, where data are mainly collected from program administrators (e.g. Xu, Reference Xu2012, Q. Zu Reference Xu2014), teachers (e.g. Guo & Li, Reference Guo and Li2012) and students (e.g. Xiao et al., Reference Xiao, Gu and Ni2014; Q. Xu, Reference Xu2014), using questionnaire surveys, interviews and classroom observations as the main research instruments. Little research, however, has touched upon the impact of language tests in relation to the broader social context.

2. 2. It is noteworthy that some of the most rigorously conducted empirical studies have incorporated the multi-method, multi-phase approach (e.g. Q. Xu, Reference Xu2014) or longitudinal approach (e.g. Xiao et al., Reference Xiao, Gu and Ni2014) to examine the consequences of testing on teaching and learning.

3. 3. Most of these studies are exploratory in nature, trying to reveal the seemingly linear effects of testing on teaching and learning. However, as pointed out by Green (Reference Green2013), the issue of washback is more of an ‘interaction between tests, teaching and learning’ (p. 48) than a unidirectional effect of tests on teaching and learning.

CET washback. Guo and Li (Reference Guo and Li2012) examined, primarily using classroom observation, questionnaire and interview methods, the washback effect of the Internet-based CET reform on 50 college English teachers in three higher education institutions. By showing a large discrepancy in teachers' reactions to the reform in terms of beliefs, interpretations and practices, this study underscored the urgent need to enhance teachers' assessment and information literacy, with a view to bringing a more positive washback effect of the reform. Xiao, Gu, and Ni (Reference Xiao, Gu and Ni2014) provided an alternative way of investigating washback, employing a more rigorous quantitative rather than traditional qualitative approach. Using multi-group SEM, the study tracked the longitudinal changes of 11 factors such as attitudes and test preparation in the CET washback mechanism, based on three questionnaire datasets collected sequentially from college students in different time periods. The outcomes of this study are of value to the field because they have elucidated how the factors subsumed under the washback effect may change over time. However, further effort involving the same group of students is needed to validate the mechanism, as the results of this study are mainly based on a different group of students' own reports in different time periods, not a longitudinal study in a traditional sense.

TEM washback. When it comes to TEM, Q. Xu (Reference Xu2012) drew on the voices of stakeholders who make important instructional decisions about English majors, namely, five foreign language experts and 724 English program administrators across China, and explored their perceptions of TEM-8 and its washback through questionnaire surveys. The study found evidence that lack of transparency of the rating criteria and limited information provided in the score reports may have led to some negative washback, despite an overall positive attitude towards the test. Later in her dissertation Q. Xu (Reference Xu2014) adopted a much more complex multi-method and multi-phase approach to examine the washback effects of TEM-4 and TEM-8 on experts, program administrators, teachers and students. The multi-method and multi-phase approach, which remains quite rare in doctoral research in China, is one of the contributions of this dissertation. In addition, by involving a wider range of stakeholders, Q. Xu's (Reference Xu2014) dissertation takes her from a consideration of micro-level washback effect on classroom to discussions of impact on macro-level institutional contexts.

NMET washback. With a focus on NMET, the highest-stakes English test in China, Dong (Reference Dong2014), like Q. Xu (Reference Xu2014), weaved qualitative and quantitative approaches to examine the impact of the NMET in Beijing on English language teaching and learning in senior middle schools. Unlike the other studies reviewed here, which primarily focused on the positive washback of the test investigated, this study highlighted the need to include an oral English component in the NMET, as the absence of an oral test in the current NMET in Beijing, has, to a large extent, exerted a negative washback effect on students' oral English learning and teaching in the senior high schools. The Computer-based English Listening and Speaking Test, as a compulsory part of the NMET in Guangdong, is the focus of F. Zhang's (Reference Zhang2015) dissertation. The findings of her qualitative study showed that teachers' varied test preparation behaviors were primarily determined by their beliefs about test value and expectations of test success. F. Zhang's (Reference Zhang2015) study, by highlighting the major function of teacher beliefs and unveiling the interactive relationship between internal teacher factors and external contextual factors, has made a contribution to explore what is going on inside the ‘black box’ of washback.

GSEEE washback. Positioning learners as the key stakeholders, S. Xu (Reference Xu2014) combined qualitative and quantitative approaches in investigating how learners’ test preparation for the writing tasks of the GSEEE operated under the washback mechanism. The major contribution of this study is that it provides empirical evidence of the direct influence of testing on learners and on their learning processes and outcomes, an area which remains underresearched (Tsagari & Cheng, Reference Tsagari, Cheng and Shohamy2017). In addition, by involving stakeholders outside the conventional EFL classroom, namely, instructors and students from coaching schools, this study sheds light on the washback effect on local educational context as well as the larger social context.

It seems clear that some interesting work is being done in the area of washback of language tests on language teaching and learning in China. Two areas remain where we would like to call for further research.

1. 1. Washback and impact of testing are by nature complex, dynamic and ever-changing. To address the non-linear and complex impact of testing on teaching and learning, Wang and Qi (Reference Wang and Qi2016) advocated the application of the Dynamic Systems Theory to washback studies, which may help contribute to a more in-depth understanding of the washback mechanism and thus constitutes an interesting future direction for washback studies.

2. 2. A topic hardly touched upon by studies reviewed here as well as in the international language assessment community is the washback of low-stakes assessment such as self-assessment and peer assessment. The lack of relevant research might be associated with the long-standing hypothesis that assessments without important consequences have no washback. Nonetheless, the impact of self- assessment and peer assessment on learner autonomy and learning as a whole may be an interesting further avenue of research.

3.6 Classroom-based assessment

Apart from large-scale standardized language tests, the past decade has also witnessed a growing interest in classroom-based assessment, or alternatively, instructionally relevant assessment in China (Pellegrino, DiBello, & Goldman, Reference Pellegrino, DiBello and Goldman2016). Ten studies fall into this category, ranging from teachers' conceptions of classroom-based assessment (Xu, Reference Xu2011) to teachers' assessment practice, which encompasses complexities of teacher assessment practices (Yang & Wen, Reference Yang and Wen2014a, Reference Yang and Wen2014b; Xu & Zhang, Reference Xu and Zhang2017), involvement of students in classroom-based assessments (Wang, Reference Wang2011, W. Wang, Reference Wang2013; Kong, Li, & Yu, Reference Kong, Li and Yu2013), and use of portfolio assessment (Wang, Reference Wang2012; Cao & Chen, Reference Cao and Chen2013). We would like to point out three broad patterns that deserve attention before delving into individual studies.

1. 1. Studies reviewed in this category include rigorously conducted empirical research using a broader range of purely qualitative methods, such as narrative inquiry (e.g., Xu, Reference Xu2011). Such methods are extremely valuable in illuminating why people perform in the way they do in language assessments in their local contexts, providing additional sources of evidence for validation of the assessment.

2. 2. It is encouraging to see that some attempt has been made to address the reliability and validity of classroom-based assessment (e.g., Xu & Zhang, Reference Xu and Zhang2017), which is a key issue of concern, as on-going evidence always needs to be collected to ensure that our interpretations of student performance in the classroom and decision making about student learning are reliable and valid.

3. 3. Several studies (e.g., Wang, Reference Wang2011, Reference Wang2013; Kong et al., Reference Kong, Li and Yu2013) on the comparison of learning through self-assessment, peer assessment and teacher assessment have yielded conflicting findings. One challenge facing the effective implementation of peer or self- assessment is intimately associated with Chinese students' cultural values that see teachers as a more authoritative voice than students. The large class sizes in Chinese contexts, however, have inevitably prevented teachers from giving individualized comments to all learners.

Conceptions of classroom-based assessment. Regarding conceptions of classroom-based assessment, Xu (Reference Xu2011) takes a narrative inquiry approach into EFL teachers' identity development in assessment reform. Xu (Reference Xu2011) started with an elaborate account of a female college English teacher's biography, and then proceeded to her teacher identity construction throughout her ten years' assessment practice. By probing into the gap between her ‘actual identity’ and ‘designated identity’, the author concluded that teachers' identities, highly embedded in socio-cultural and historical contexts, evolved in the process of struggling between the teacher's dual identities, the ‘professional self’ and ‘personal self’. Although it may be difficult to generalize the findings of this case study, it underscores the need to situate such inquiry within the wider social, cultural and historical frameworks to understand why practices have continued the way they do.

Complexities of teacher assessment practices. Moving on from the intricacies of identity construction, another line of research has revealed the complexities of teacher assessment practices. In a classroom observation study, Yang and Wen (Reference Yang and Wen2014a) present an excellent account of the dynamic changes in the goals of real-time formative assessment by integrating grounded theory and classroom discourse analysis. Using the same datasets, Yang and Wen (Reference Yang and Wen2014b) attempted to build a contingency-centered theory for real-time formative assessment in EFL classrooms. The importance of Yang and Wen's (Reference Yang and Wen2014a, Reference Yang and Wen2014b) studies is that they have provided valuable insights into the dynamics of real-time formative assessment and moved forward Black and Wiliam's (Reference Black and Wiliam2009) argument of capitalization upon ‘moments of contingency’ in the formative assessment theory. In contrast to these two qualitative studies on the complexity of assessment as interaction, Xu and Zhang (Reference Xu and Zhang2017) in a mixed method study attempted to address the complex challenges that language assessors face in ensuring effective and fair implementation of group work assessment. The results showed that individual scores were highly discriminating, and students and teachers generally held positive attitudes towards individual scores. Based on the findings, the authors highlighted the necessity and feasibility of assessing individual contributions in group work. Studies of this kind are extremely important not only for Chinese educational context where group work has gained prominence in classrooms, but also for the international language assessment community where the reliability and validity of classroom-based assessment remains underexplored and underappreciated.

Students’ involvement. Another theme that emerges as pertinent to teachers' classroom-based assessment practices is the involvement of students in assessment, in particular, investigation of the potential for learning through self-assessment and peer assessment. Situated in dynamic assessment, Kong, Li, and Yu (Reference Kong, Li and Yu2013) made a systematic comparison of peer-mediated intervention and teacher-mediated intervention in EFL writing. A generally more effective function of teacher-mediated intervention is reported, evidenced in the finding that teachers focused more on the content and ideas than language use and had a more significant effect on students' learning in terms of lexical complexity. In contrast, Wang (Reference Wang2011)'s study showed that the integration of self- and peer assessment functioned best in promoting students' learning in writing, followed by self-assessment and then teacher assessment. Complementing these quantitative findings on learning gains, W. Wang's (Reference Wang2013) longitudinal qualitative inquiry into students' perceptions of self- and peer assessment in EFL writing revealed that students perceived self-assessment as helpful for the improvement of content, organization and accuracy, and peer assessment useful for the boost of reader awareness, organization and accuracy. This dissertation is particularly important in the Chinese educational context where peer and self-assessment seem to be both underappreciated and underresearched, and traditional teacher-centered pedagogy is preferred. However, this study would have made more contribution to the field if students' positive perceptions could have been somehow related to their learning gains.

Portfolio assessment. Portfolio assessment, as an alternative to traditional tests in high-stakes contexts, is another important theme in classroom-based assessments. Wang's (Reference Wang2012) study in a junior high school attested to the effectiveness of portfolio-based learning assessment system in fostering learner autonomy, by drawing upon data from questionnaires and case studies. When portfolio assessment was applied in the teaching of interpreting, it was also reported to be conducive to the promotion of learners' motivation and autonomy (Cao & Chen, Reference Cao and Chen2013). Notwithstanding, given the current tide of big data and convenient sharing, a promising research field would be the integration of technology in portfolio assessment, namely, the use of e-portfolios in classroom-based assessments, through which students can keep ongoing records of their language learning experiences inside and outside of the classroom.

It can be seen that studies on classroom-based assessment have, though not many in number, generally yielded satisfying results. There are three further issues that we would like to point out regarding this group of empirical studies.

1. 1. To address the dilemma of peer assessment, self-assessment and teacher assessment, a teacher-student collaborative assessment approach was proposed by Wen (Reference Wen2016), which integrated the three types of assessment in pre-class, in-class and post-class sessions to form a coherent mechanism. More empirical efforts are warranted to see whether this approach indeed provides more favorable conditions for teacher and student dialogues and thus leads to more learning gains.

2. 2. In spite of evidence of teachers' identity construction in classroom-based assessment (Xu, Reference Xu2011), there are few empirical studies on teachers' language assessment literacy. The development of teachers' knowledge about, and strategies and skills in, language assessment are essential for the effective implementation of classroom-based assessment. It would thus be helpful to devote more empirical efforts to the investigation of teacher assessment literacy and recommend actions on how to promote it nationwide.

3. 3. It seems that language assessment researchers in China have as yet failed to recognize the potential value of alternative assessment approaches, as evidenced in the limited amount of work on portfolio assessment and dynamic assessment. More empirical research is needed to explore the use of alternative assessments to supplement current standardized tests with a view to maximizing learning gains through assessment.