Three cases of failed policy attempts

South Korea

In 2012, the Ministry of Education and the Korea Institute of Curriculum and Evaluation (KICE) launched the NEAT, an exam designed to replace the existing Korean College Scholastic Ability Test (KCSAT) – a paper-based indirect standardized test used for college admission. Unlike KCSAT, NEAT was an internet-based English test intended to directly assess test-takers’ production skills (speaking and writing), as well as receptive skills (listening and reading) with a strong emphasis on communicative competence (Ahn, Reference Ahn2015). The government anticipated that a more practical communicative English test would help Korean students develop production ability, hoping for a positive washback effect (Shin & Cho, Reference Shin and Cho2020; Whitehead, Reference Whitehead2014). NEAT consisted of three levels: Level 1 for adult learners, Level 2 and Level 3 for high school students. Level 2 and Level 3 targeted academic English abilities and basic social English abilities respectively. Students in principle could choose a level according to their future plan (Shin & Cho, Reference Shin and Cho2020). As test-takers were allowed to take the exam twice and submit the higher score for college admission, it was expected that the test could reduce stress among students. It was also intended to decrease the astronomical expenditure on private-sector education (e.g., tutoring and cram schools) for test preparation by reducing the exam's difficulty level. As opposed to KCSAT, NEAT was designed to be a criterion-referenced test, not a norm-referenced test. The idea was that students should be able to obtain a high score without receiving additional private instruction (Lee & Lee, Reference Lee and Lee2016).

In spite of the government's intention, the plan to implement NEAT was abandoned in 2015. The implementation faced both logistical and technical challenges. It required resources to establish a countrywide testing infrastructure and hire well trained graders who could evaluate students’ speaking and writing performance. Since universities and colleges did not want to be labeled as lower-tiered schools, only a handful of schools accepted Level 3 scores when a trial NEAT was implemented in 2013 (27 of 222 four-year universities, and nine of 149 two-year colleges, accepted the Level 3 test scores) (Shin & Cho, Reference Shin and Cho2020); in practice, students would have no choice but to take the Level 2 test to gain college admission. Tied with the launch of NEAT, the Ministry of Education also revised the National English Curriculum in 2009 so as to promote communicative competence with a focus on speaking and writing. High school English teachers, however, expressed concerns that schools would not be able to help students prepare for the test because of limited time and experience in teaching and assessing speaking. Dissatisfied with current school practices, students would turn to the private sectors for the NEAT preparation (Lee & Lee, Reference Lee and Lee2016).

Finally, because NEAT lowered the targeted difficulty level, the exam could not differentiate among test-takers well enough for college admission. As a result, many universities, especially Korea's most prestigious ones, opted to deemphasize the NEAT and to implement their own admission criteria, such as TOEIC and TOEFL, English writing samples, intensive interviews, and extra-curricular activities. In contrast to the government's hope, the multi-level admission criteria further widened the college accessibility among students according to their socio-economic status (SES). Students with higher SES have access to more diverse experiences of learning English, including international travel and study as well as academically intensive experiences based on their parents’ social networks (Shin, Jahng & Kim, Reference Shin, Jahng and Kim2019).

China

China also attempted to reform its English education policies. Two policy attempts are directly relevant to the present discussion; one is to reduce the English portion in China's high-stake National College Entrance Exam (Gaokao), and the other is to incorporate speaking assessment in high-stakes tests, including the Gaokao.

Since the early 1990s, English has held equal weight with two additional main subjects in the Gaokao: Chinese and Mathematics. However, in late 2013, China's Ministry of Education announced a tentative plan to reduce the weight of English in the Gaokao. Beijing simultaneously proposed to reduce the score of the English test from 150 to 100 points, while other provinces considered eliminating English completely from the Gaokao. This policy initiative provoked fierce debate among parents and scholars over the role of English and the future of English education in China. Some argued that lowering the weight of English in the Gaokao may help change exam-oriented English education in China and thus reduce students’ burden in learning English both in and outside of school (Zhao, Reference Zhao2016). Through reducing the importance of English in the Gaokao, more educational resources and students’ attention would be focused on learning Chinese language and culture, which in turn was considered favorably (Guo & Du, Reference Guo and Du2013). Others, however, concerned that devaluing or eliminating English from the Gaokao may have negative washback effective on primary- and secondary-school English education, particularly in regions with limited resources; English education may be deemphasized in those regions (Zhou, Reference Zhou2014). When the official Gaokao reform plan was revealed in 2014, however, the scoring weight of English remained the same, as if the various public voices unheard (Shao & Gao, Reference Shao and Gao2017). The only change made was that from year 2017 students can take the English exam twice and select the higher score.

Another important governmental policy attempt is to incorporate examination of spoken English into high-stakes English tests, including the Gaokao. Despite the promotion of communicative English language teaching as national policy since the early 21^st century, local implementation has varied tremendously by region. This gap is even greater when it comes to the inclusion of speaking in tests because they are both labor- and resource-intensive. While wealthy areas, such as Beijing and Shanghai, have started to implement computer-moderated spoken English tests, other provinces are still working to ensure that all schools have qualified English teachers for their students. Some have cautioned that if spoken English becomes a mandated part of the Gaokao, the inequality of English education across Chinese provinces and between high and low SES families would be further widened (Qi, Reference Qi2018). Observing substantial regional and SES disparities, officials have only gradually incorporated a spoken English exam in the Gaokao on a trial basis. Only a handful of provinces, such as Guangdong and Shanghai, have included spoken English test as part of the Gaokao. In most other provinces, spoken English remains either optional or a mere ‘formality’, not counting towards the Gaokao English results (Fan & Tong, Reference Fan and Tong2014).

Japan

Instead of developing an original speaking test, Japan's Ministry of Education, Culture, Sports, Science and Technology (MEXT) opted to use external standardized English proficiency tests which included a speaking element as part of the university entrance exam (MEXT, 2018). Eight external tests were chosen originally to serve this purpose, starting from 2020. MEXT received strong pressure from business and political communities to take radical action to improve Japanese English-speaking skills, believed to be indispensable for the nation to remain competitive globally (Abe, Reference Abe2017). Utilizing external tests served to remedy the logistical difficulties of administering a speaking test to a large number of students in a single day as part of the existing Common Test. It also became impossible to evaluate student speaking performance validly and reliably in this high-stakes examination. After receiving substantial stakeholder criticism, in late 2019, MEXT announced that they would postpone the implementation (MEXT, 2019).

This plan created tremendous confusion among test-users before its implementation. Japan's public and many private universities following this policy were asked to decide which external tests should be accepted, how to determine the cut-off score for each designated test, and how to use the test results. Universities found these decisions challenging and made decisions almost arbitrarily (Kim & Mizuto, Reference Kim and Mizuto2019). It proved next to impossible to compare scores of multiple external tests with differing target domains, abilities, proficiency levels, and formats. Social network sites filled with both student and teacher comments expressing their confusion and anxiety due to insufficient information about the consequence of this new policy. MEXT's intention to ease student anxiety by allowing them to take external tests multiple times, as opposed to having only one chance under the current system, proved more stressful for students because they felt the need to begin preparation for these tests even earlier. Since designated external tests also differ in accessibility and cost, students, depending on their SES and prefecture of residence, may face limits in which test to take and the number of times they can take them, including practice examinations.

As we have discussed elsewhere (Butler & Iino, Reference Butler, Iino, Lanteigne, Coombe and Brownin press), the most critical problem with this policy is that, in all likelihood, students’ SES would be the best predictor of success on these tests. In a society where Japanese is almost exclusively used, students have to make special effort to create opportunities to practice their English-speaking skills, requiring financial and local resources beyond regular schoolwork. Foreign language teaching, especially for children, is a substantial business in Japan, with a U.S. $7.8 billion market in 2017, according to the Yano Economic Research Institute (2018). Parents with higher educational backgrounds who reside in large cities tend to invest substantially more in their children's English-speaking practice at an early stage in their lives. Meanwhile, among Organization for Economic Cooperation and Development (OECD) member states, Japan's relative poverty rate ranks among the ten highest (OECD, 2015). Making it harder for otherwise qualified students from low SES backgrounds to get into college will only increase social disparity and block opportunities for advancement.

Trapped between beliefs and reality

All three governments identified two common problems related to English education that they felt could be solved through college entrance exam reforms: (a) students’ oral communicative skills in English are weak; and (b) the heavy financial and emotional burden on students in countries with widening SES and regional disparities to prepare for English examinations. Indeed, we have some evidence for supporting the government concerns. For example, while expenditures in South Korea on private English education at the K–12 totaled $5.1 billion in 2019, students’ English proficiency did not rank highly in international proficiency tests (EF institute, 2020).

The governments asserted that changes in entrance exams should have a substantial impact on English education in and outside of formal education. This assertion was easy to understand, given the socio-historical contexts in which exams have been used in these countries; highly competitive exam systems have played a significant role as a selection method of individual talents for higher education and career options and advancement (e.g., Butler & Iino, Reference Butler, Iino, Lanteigne, Coombe and Brownin press; Kwon, Lee & Shin, Reference Kwon, Lee and Shin2017; Qian & Cumming, Reference Qian and Cumming2017). The centuries-old tradition of selecting individuals based on merits was apparently associated with a widely subscribed notion in East Asia that one's achievement should be attributed mainly to effort rather than innate talents. Effort is considered a universal human trait, regardless of upbringing in the society (Phillipson, Reference Phillipson2013; Stevenson & Stigler, Reference Stevenson and Stigler1992).

For concrete policy solutions, the three governments took a two-pronged approach: (i) to introduce speaking assessment as part of the high-stakes college entrance exams; and (ii) to offer increased flexibility in the exam system (e.g., permitting multiple attempts to takes examinations or allowing exam choice), as well as lowering minimum standards for English (e.g., lowering the level of difficulty and reducing the English exam's weight). These proposed solutions, however, appeared to rest on beliefs that diverge from reality, which seems to be a major source of the policy failure.

The first problem can be attributed to the governments’ monolithic view towards English. Government documents all portray English as a symbol of global competency and elite communication, thereby making it easily justifiable to hold English proficiency an ideal qualification for admission to higher education. Policy makers assume the existence of a single, correct, and measurable (or testable) standard versions of spoken English. This monolithic view coincides with the nature of standardized testing. College entrance exams in the three countries have been primarily composed of multiple choice questions. In principle, each test item should be only one ‘correct’ answer. Any test items with multiple acceptable answers are treated as ‘flawed’ and become the target of severe criticism (e.g., Kwon et al., Reference Kwon, Lee and Shin2017).

In reality, however, English is far from monolithic both in terms of actual usage and utility in the societies in question. Newly emphasized ‘communicative competence’ – prioritizing actual language use and communicativeness – does not always mesh with prescriptive grammar and normative language use that national curricula and standardized tests have been relied on. Speaking is particularly dynamic due to its individual, regional, and contextual variability and spontaneity (Luoma, Reference Luoma2004). A hypothetical ‘standard’ speaking ability would be based on an imagined ‘native speaker,’ or ‘global elite’ model (Ahn, Reference Ahn2015; Jeon & Lee, Reference Jeon and Lee2006). Due to logistical and technical constraints, the current policy proposals for speaking tests in China, Japan and South Korea are largely monologue-based; candidates speak to a computer on a set topic with no interlocutors. The monologue format is easier to standardize than dialogical or group discussion formats because various potential interlocutor effects can be ruled out. However, it is also easier for test takers to come up with test-wise strategies (e.g., memorizing monologues) that would be a serious potential threat to validity of the test (Al Fraidan, Reference Al Fraidan2019). Indeed, it is often reported that existing standardized speaking tests do not reflect the actual communicative abilities that students may exhibit outside of school and assessment contexts. Curran (Reference Curran2018), for example, beautifully illustrated four working-class individuals in South Korea who did not perform well in standardized tests and yet obtained sufficient communicative English-speaking abilities in light of their own goals and job requirements. Moreover, while English teachers in East Asia are often accused of having insufficient English oral skills (Butler, Reference Butler2014; Freeman et al, Reference Freeman, Katz, Gomez and Burns2015), given that the target model itself is imagined, it is not surprising that English teachers ‘fail’ to offer ‘model’ oral input to their students.

Second, the unsuccessful policy attempt was clearly reflected in the tug of war between equality and equity concerns. Equality ensures that every student has the same support in order to access to the given opportunity, whereas equity offers differentiated supports to students, depending on their needs in order to access to the given opportunity (Levitan, Reference Levitan2015). By providing options for test takers and/or by designing the test as a criterion reference test, the government attempted equality in English education. It was hoped that these attempts would reduce the burden of students’ English learning and ultimately reduce the high cost of private education. As long as students make effort to study the content covered at school, they can have an equal chance to be successful in the entrance exam.

However, the remedy for equality would not necessarily yield equity in testing, given the substantial gaps in accessibility and availability of English language test preparation resources. These choices are often constrained by the parents' SES. While higher SES students can enjoy many options to maximize their chances to get into college of their choice, lower SES students, in practice, may have fewer options; their choice is often determined by their ability to pay for additional resources. Empirical evidence indicates that speaking abilities are particularly strongly influenced by the students’ regional and SES background (e.g., Butler & Iino, Reference Butler, Iino, Lanteigne, Coombe and Brownin press; Marchant & Finch, Reference Marchant and Finch2016). In linguistic environments where local languages are almost exclusively spoken, students need to create opportunities to practice English in order to develop proficiency. It is hard to develop fluency within limited time allocated for English speaking in school curriculum without external practice opportunities. Creating such opportunities, however, requires financial resources and cultural capital which are unevenly distributed. Moreover, evidence shows that one's ability or opportunities to demonstrate effort – believed in East Asia to be equally given to everybody – is indeed influenced by SES (Kariya, Reference Kariya2008).

Recommendations and conclusions

Considering the incompatibility of dynamic and multifaceted nature of speaking abilities and standardized tests as well as structural difficulties in ensuring equity for assessing speaking abilities in English, it may not have been a good policy solution to incorporate English speaking as part of standardized college entrance exams in China, Japan and South Korea. This recommendation does not mean that speaking is unimportant in English education in these countries, but rather that the policy most likely would not have yielded the intended consequences: namely, improving students’ speaking abilities in English. First, we have plenty of evidence indicating that washback effects of tests are complicated; we cannot simply expect that learning and pedagogies would change as expected with changes in assessment (Cheng, Sun & Ma, Reference Cheng, Sun and Ma2015).

Second, speaking abilities measured through standardized tests would not necessarily meet the real needs of students. Perhaps we need to reconceptualize the construct of speaking abilities. For example, due to COVID–19, online interactions (classes and business meetings) have become increasingly common, and anecdotal evidence has suggested that oral communication may require different communicative strategies as part of communicative competence (e.g., relying less on gestures and other non-verbal information including back-channel behaviors, understanding different meaning of pauses and pace, taking overt strategies to take a conversation floor, and developing higher meta-awareness of one's own language use). Although empirical research is required to confirm such initial observations, a more flexible and fluid view of speaking abilities than the current one may be necessary in our rapidly changing world. It is advisable for policy makers to seek solutions other than standardized testing to assist students’ oral communicative competence in such a way that better suits their real needs.