1. Introduction and selection protocol

In terms of area, Canada is the second-largest country in the world, with a relatively small population of approximately 35 million people. According to the Association of Universities and Colleges of Canada (AUCC), there are 83 universities in Canada that are independent post-secondary education institutions with degree-granting authority. At the time of this survey, 19 of these offered doctoral degree programs in the broader applied linguistics area within a range of faculties, schools and departments. Canada has two official languages, French and English (although Inuktitut is an official language in Nunavut, a federal territory of Canada). Doctoral research is undertaken in both languages, but given the scope of the present survey, we have focused exclusively on English-medium universities. Further, the numbers of doctoral dissertations in applied linguisticsFootnote ¹ (broadly defined) across Canadian English medium universities are impressively large, with three dominant areas of research: 1) second language acquisition, 2) bilingual and immersion education, and 3) language teaching and learning.

A search of the above three areas with Theses Canada Portal (www.collectionscanada.gc.ca/thesescanada/index-e.html) identified more than 250 dissertationsFootnote ² which had been completed over the last five years (between 2006–2011). We therefore had to narrow our review of Canadian dissertation research and have chosen to focus on research related to language assessmentFootnote ³. Our justifications for this are, first, that the three dominant areas listed above are not mutually exclusive. Research related to language assessment and testing is not only embedded within each of these areas, but is also becoming an increasingly important link between them. In fact, many of the questions addressed in doctoral research are directed at the central role of assessment and testing in teaching and learning, within a range of curricular contexts and at different instructional levels and stages of language development. Second, language assessment has gained in both prominence and importance within Canadian applied linguistics research, as evidenced by the increasing numbers of dissertations on the subject, as well as papers presented by Canadian graduate students at the annual Language Testing Research Colloquium (LTRC) of the International Language Testing Association (ILTA). We have also seen a dramatic increase in job openings for language assessment specialists at the faculty level and in testing agencies within the broader Canadian and international contexts. Increased interest in issues related to language assessment in Canada has also resulted in the formation of a national association, The Canadian Association of Language Assessment/L'association canadienne pour l’évaluation des langues (CALA/ACEL), which was officially established in 2009. Third, both authors are language assessment experts with the expertise necessary to judge the merits of dissertations in this area. There is no doubt that language assessment will continue to be one of the key focuses of future research in language teaching, as broadly defined by this journal, so we believe a review of Canadian doctoral research on language assessment is both useful and timely.

A number of different search methods were used to identify dissertations in language assessment and testing. First, data were collected from the Library and Archives Canada – Theses Canada Portal. Under the advanced search option, keywords such as ‘language test’, ‘language testing’, ‘language test performance’ and ‘classroom assessment’ were used, specifically for dissertations at the Ph.D. level. This search identified six dissertations written within the search time framework. Broadening the search keywords to ‘ESL language assessment’, ‘ESL and assessment’, ‘ESL and assessing’, ‘language and scoring’ and ‘language and feedback’, ten more dissertations were identified. One of these was an Ed.D. thesis (Wakamoto Reference Wakamoto2007), but because only a few universities in Canada offer such professionally-oriented degrees, we decided to restrict our review of doctoral research to Ph.D. dissertations. After careful reading, three more dissertations, those by Ishii (Reference Ishii2009), Suzuki (Reference Suzuki2009) and Yang (Reference Yang2008), were excluded from the review due to the narrow linguistic focus of their studies, that is, they did not show the relationship between teaching, learning and assessment, or have implications for language teaching as defined by the journal (see also footnote 1, in which we clarify our definition of applied linguistics). Thus, as a result of our initial search, 12 out of the 16 Ph.D. dissertations that we had identified were included at this stage.

We then contacted 15 professors who worked in the broader area of applied linguistics within 11 universities offering doctoral programs that included a focus on language assessment and testing, namely, the University of British Columbia, University of Alberta, University of Calgary, University of Toronto, Brock University, University of Ottawa, McGill University, University of Montreal, Concordia University, University of Sherbrook and Laval University. We located a further 12 dissertations as a result. This paper thus reviews a total of 24 Ph.D. dissertations in language assessment and testing.

Each of us then independently examined the abstracts of the 24 dissertations in order to identify recurring topics. We then met and compared our analysis, reducing the number of topics and refining our definitions. At this point, we consulted the International Language Testing Association (ILTA) bibliographical categorization index (www.iltaonline.com) to further refine and validate the topics. Using the ILTA classification index, we reduced the number of topics to five: 1) reliability, validity and factors affecting test performance; 2) washback (impact) and ethics; 3) raters, rating and rating scales; 4) classroom-based research: teaching, learning and assessment; and 5) vocabulary learning, lexical proficiency and lexical richness. We have situated our review within the field of language assessment by discussing the impact and/or contributions of these dissertations. Due to the considerable variety of dissertation topics and the varied methodology/methods employed in these 24 doctoral research studies, our review is at times uneven. Dissertations which demonstrated methodological strength are given greater attention because of the greater validity of their findings and their original contribution to scholarship within the field of language assessment. We also made an effort to point out some of the unique features of certain dissertations, even if they are of single-method design. Later in our review we have highlighted the interdisciplinary nature of applied linguistics. The language assessment research considered here, for example, was influenced by educational measurement, educational psychology, linguistics, sociology and philosophy, along with other humanities and social science disciplines. Our review ends with some concluding remarks and a consideration of future directions.

2. Reliability, validity and factors affecting test performance

At the core of any testing inquiry is the question of validity. Test developers and test users want to be confident in the inferences drawn from testing results, and the outcomes that are a consequence of those results. Validity, including reliability, is a way to view and interpret test information (Moss, Girard & Haniford Reference Moss, Girard and Haniford2006). More specifically, establishing the validity of a test involves determining the appropriateness and accuracy associated with assessing student ability (Messick Reference Messick and Linn1989). Test validity should be approached longitudinally from multiple dimensions, because interpretations of validity concepts such as accuracy and appropriateness are influenced by the assumptions made about student learning, the purpose of the test, and the importance that is assigned to the testing context. The more traditional types of language testing research typically involve issues of both reliability and validity, which are, therefore, at the heart of language testing research (Bachman Reference Bachman2000). Research into factors affecting test performance also falls into the broad category of test reliability and validity research, because it adds to our understanding of the kinds of variables (beyond language ability itself) that contribute to students’ test scores, thus helping to measure the accuracy and appropriateness of our tests.

Five doctoral dissertations considered in this review fall into this category: Abbott (Reference Abbott2005), Farnia (Reference Farnia2006), Gao (Reference Gao2007), Limbos (Reference Limbos2005) and Zheng (Reference Zheng2010). The first four were concerned with test reliability and validity, and all focus on reading. The final study, Zheng (Reference Zheng2010), investigated factors influencing test performance of language achievement and proficiency. Abbott (Reference Abbott2005) and Gao (Reference Gao2007) dealt with adults’ reading in large-scale testing, while Farnia (Reference Farnia2006) and Limbos (Reference Limbos2005) investigated reading achievement and reading difficulties in children. Abbott (Reference Abbott2005) and Gao (Reference Gao2007) are guided by the educational measurement and cognition literature while Farnia (Reference Farnia2006) and Limbos (Reference Limbos2005) were informed by literatures drawn from educational and clinical psychology. Zheng's (Reference Zheng2010) study explored factors contributing to the test performance (in English) of Chinese learners at university level. Three of these studies – Abbott (Reference Abbott2005), Gao (Reference Gao2007) and Zheng (Reference Zheng2010) – were aligned with the core research tradition in language assessment, while those of Farnia (Reference Farnia2006) and Limbos (Reference Limbos2005) belonged more to the reading psychology tradition, which indicates the theoretical and interdisciplinary diversity of these studies.

Abbott (Reference Abbott2005) tested the hypothesis that some of the test items included in the Canadian Language Benchmarks Assessment (CLBA) reading subtest favor certain groups whose first language orthographies differ markedly from that of English. The study concerned Mandarin speakers, who have a tendency to use bottom-up local reading strategies, and Arabic speakers, who tend to have top-down global reading strategies. The study was in two parts and used a mixed method design (Creswell & Plano-Clark Reference Creswell and Plano-Clark2007; Teddlie & Tashakkori Reference Teddlie and Tashakkori2009). Verbal report data were first collected from Mandarin and Arabic learners to identify, clarify and elaborate on the reading strategies engaged by the CLBA reading comprehension tasks. In part two of the study, Abbott collected data from 250 Arabic and 250 Mandarin examinees. Three reading experts assigned each of the 32 CLBA reading items to one of the seven bottom-up or top-down reading strategy categories identified in part one of the study. SIBTEST analysis revealed systematic group differences in both bottom-up and top-down reading strategy categories. Items involving breaking words into smaller parts, scanning for details, identifying synonyms or paraphrases and matching key vocabulary in the text were found to favor Mandarin speakers, while items involving skimming for the gist, connecting or relating information presented in different parts of the tests and drawing an inference favored Arabic speakers. These results provide evidence for the validity of the bottom-up and top-down reading strategy framework (Grabe Reference Grabe2009; Grabe & Stoller Reference Grabe and Stoller2011) and a substantive method for interpreting group differences on the CLBA reading assessment. This study employed verbal protocol data and sophisticated statistical approaches such as differential item function analysis, which are common to research methodologies and analytical approaches in language testing (Bachman Reference Bachman2000). This mixed method study was elegantly designed and executed and has led to a number of publications (e.g. Abbott Reference Abbott2007). The same characteristics and contribution are also seen in the study by Gao (Reference Gao2007) discussed below, again in the area of reading assessment.

Gao's (Reference Gao2007) study addressed the call to integrate cognitive psychology with assessment to inform test design and validation and to provide detailed diagnostic feedback. One approach to this is to model item statistics, in particular item difficulty. This study used a cognitive-psychometric approach to model the reading items included in the Michigan English Language Assessment Battery (MELAB) and tested the model through four stages:

1. a. An initial cognitive model was hypothesized to underlie the MELAB reading performance based on the review of the process associated with reading and reading test taking by ESL learners. The model was then validated by

2. b. having the cognitive demands made by the test items analyzed by raters,

3. c. collecting students’ verbal reports of the processes they used to arrive at the correct responses, and

4. d. examining the relationship between the proposed cognitive processes and item difficulty estimates using a tree-based regression.

The study demonstrated the value of using multiple sources of evidence, using a mixed method design and employing different participants (e.g. raters and test-takers) to evaluate a cognitive model and the relationship between cognitive psychology and language assessment. Considering the interdisciplinary nature of language testing (Bachman Reference Bachman2000), studies of this nature have particular strength and make an original contribution to our understanding of test validation.

Zheng (Reference Zheng2010) examined motivation, anxiety, global awareness and linguistic confidence and their relationship to language test performance within the context of Chinese university students taking the College English Test Band 4 (CET-4) in China. Using a mixed method approach, through concurrent survey and interview inquiries, this study explored whether and how the selected psychological factors contributed to students’ CET performance. Results from exploratory factor analysis revealed that Chinese university students displayed three types of instrumental motivation (mark orientation, further-education orientation and job orientation), two types of anxiety (language anxiety and test anxiety) and two types of confidence (linguistic confidence and test confidence). The results of confirmatory factor analysis led to a modified socio-educational model of motivation with some context-specific concepts (new instrumental orientations, global awareness and linguistic confidence) that more accurately represented the characteristics of the Chinese university students. The results of structural equation modeling confirmed that attitude toward the learning situation and integrative orientation were two strong indicators of motivation, which in turn influenced language achievement and confidence. The negative impact of anxiety on language achievement was confirmed. Certain group differences were found between male and female students, high and low achievers, students from the arts programs and those from the science programs, and students who started to learn English before Grade 7 and those who did so after Grade 7. The interview findings further confirmed stronger instrumental than integrative orientations within the study context. This study tested well-developed motivation and anxiety models in the Chinese context and expanded our knowledge of theory development in English language education. The implications of this study point to the importance of understanding language test-takers’ characteristics in their macro- and micro-learning contexts, thus providing evidence for the meaning of test scores and informing test score use.

What impressed us is that all the three studies above 1) used multiple sources of evidence to support their findings; 2) collected data concurrently or sequentially at different stages; 3) conducted data analysis using advanced statistical procedures alongside sophisticated interpretative or qualitative analysis. All three are of high quality and have been published (cf. Abbott Reference Abbott2007; Gao & Rogers Reference Gao and Rogers2011).

Farnia (Reference Farnia2006) and Limbos (Reference Limbos2005) took a different approach, collecting data through individual testing of reading with younger students. Both studies therefore addressed the nature of reading development at the school level and involved students learning English as a first and a second language. Farnia (Reference Farnia2006) was concerned with uncovering the patterns of growth in text-reading fluency and reading comprehension in two groups of students: children with English as a second language (ESL) and those with English as a first language (EL1). The study explored the role of linguistic and cognitive predictors in understanding this growth. The sample consisted of 50 EL1 and 107 ESL children whose performance on the potential predictors of memory, phonological processing, spelling, word level reading skills and oral language proficiency (vocabulary and grammar) was assessed yearly from Grade 1 to Grade 6. Text-reading fluency and reading comprehension were tracked from Grade 2 to Grade 6.

Three interrelated studies were conducted. Study 1 focused on growth trajectories in lower, word-level and high-level (comprehension and fluency) reading skills in Grades 1–3 and Grades 4–6. Hierarchical Linear Modeling indicated that ESL children developed faster than EL1 children in word-level reading skills, aspects of phonological processing, memory and oral language. Spelling and language proficiency skills were significant concurrent and longitudinal predictors of reading comprehension in Grades 4–6. Study 2 revealed similar patterns of text-reading fluency trajectories for the EL1 and ESL children. Regardless of initial text-reading fluency level and language group, students who were able to read words fluently at the outset demonstrated a faster growth rate from Grades 2 to 6. Study 3 explored further facilitation between text-reading fluency and reading comprehension at Grades 4–6, and results showed that such growth among the EL1 and ESL students contributed to their later performance in reading comprehension. This study considered implications for assessment, the identification of developmental risk factors, practical means to improve text-reading fluency and reading comprehension for ESL children in comparison to EL1 students. Studies of this nature are extremely important for countries like Canada where schooling involves both groups of students with reading development at various levels.

Limbos (Reference Limbos2005), in a similar study, explored the common misconception that assessment of reading disabilities in ESL students must be delayed for two or three years until oral language skills have developed. Using a battery of measures assessing various domains of reading, memory and cognitive and oral language, the study collected data from 339 Grade 1 students (107 EL1 students and 232 ESL) and 253 Grade 3 students (80 EL1 and 173 ESL). Confirmatory factor analysis corroborated the theoretical constructs of Phonology, Oral Language, Verbal Memory and Reading at Grades 1 and 3. Structural equation modeling was used to examine the relationship of these constructs to Grades 1 and 3 reading criteria and supported the phonological-core deficit model of reading. Logistic regressions showed that at both Grades 1 and 3, a composite Phonology score consistently made a significant, unique contribution to the predictor of at-risk EL1 and ESL students. The data suggested that it is possible to identify EL1 and ESL students who are highly or moderately at risk at Grade 1. As mentioned above, the Limbos (Reference Limbos2005) and Farnia (Reference Farnia2006) studies are of critical importance to Canada because both EL1 and ESL students were studied and compared within the same classroom context. Such studies have important implications for teaching and assessment in multilingual and multicultural classrooms and important impacts on policy within the Canadian education context.

3. Washback (impact) and ethics

Research into washback and the impact and consequences of testing has flourished for the past 25 years and has become a prominent area of research in language testing. While traditional washback research focused on the teaching and learning context, impact studies have expanded it to a broader range of test stakeholders in ever-larger contexts. The issue of ethics in language testing is also relevant here, since, whatever the scope of the study, it is at the core of language testing research relating to test validation and the concept of consequential validity (Messick Reference Messick and Linn1989). Messick and others have argued that test consequences lead to inferences about test performance and thus influence test validation (Messick Reference Messick1996; Kane Reference Kane2002; Mislevy, Steinberg & Almond Reference Mislevy, Steinberg and Almond2003; Moss et al. Reference Moss, Girard and Haniford2006). McNamara & Roever (Reference McNamara and Roever2006) argue that one of the greatest challenges facing these researchers is the clear separation between the psychometric, positivist (or post-positivist) orientations to interpretation about test-taker performance and the messy, socially constructed nature of consequences. Addressing this relationship tends to improve the quality of the research, a fact that is demonstrated in the six studies reviewed below.

The six studies that expand these discussions on washback and impact in language testing are those by Fleming (Reference Fleming2007), Sterzuk (Reference Sterzuk2007), Mullen (Reference Mullen2009), Baker (Reference Baker2010), Shih (Reference Shih2006), Tan (Reference Tan2009) and Wang (Reference Wang2010). Of these, four might be considered traditional washback research, collecting data from teachers and students, while two (Fleming Reference Fleming2007; Sterzuk Reference Sterzuk2007) employed a more philosophical framework to investigate power and identity issues that arise from testing.

Fleming (Reference Fleming2007) drew on the voices of Punjabi-speaking immigrants enrolled in a government-sponsored ESL program and explored how these adult newcomers to Canada constructed new national identities. The study, using survey methods of questionnaires and interviews, traced how the common threads in their conceptions of citizenship compared to those embedded within the national ESL assessment – the Canadian Language Benchmarks Assessment (CLBA) and its associated curriculum documents. It further illustrated how these documents construct and position idealized conceptions of second language (L2) learners. Assessment and curriculum have the power to construct identity, especially in immigrant countries like Canada, and also shape teaching and learning within their context (Shohamy & McNamara Reference Shohamy and McNamara2009). The study showed that there were significant gaps between the national assessment of the CLBA (and the associated curriculum documents used in this context) and the range of views expressed by the learners in this study. Based on this research, Fleming (Reference Fleming2007) outlined some implications associated with L2 citizenship education in terms of research priorities, national curriculum development and pedagogical treatment options. In addition, three specific recommendations are made with regard to curriculum content: that citizenship content be made more explicit within the Canadian national assessment program and curriculum documents; that this content emphasize positive representations of learners in documents as active and socially integrated; and that this content be centered on the legalistic aspects of citizenship and avoid the use of singular normative cultural standards.

Sterzuk (Reference Sterzuk2007), like Fleming (Reference Fleming2007), provides an alternative way of conducting washback- and impact-related studies, employing a philosophical rather than a validity framework (see the discussion in Cheng Reference Cheng, Shohamy and Hornberger2008). This study, using a postmodernist and critical inquiry approach, focused on the negotiation of power in schools and the social and academic experiences of First NationsFootnote ⁴ and Métis children in Saskatchewan, who speak a non-standard variety of English called Indigenous English. The study explored these learners’ experience in a class of 25 and also interviewed 11 educators to explore their perceptions of the learners’ literacy and language ability. The results of the study indicate that the First Nations children speak a dialect of English that differs phonologically, morphologically, syntactically and lexically from Standard English. They also differ in discourse behaviors from their White settle classmates. The study examined the children's speech and classroom behaviors in order to identify important characteristics of their discourse. The findings derived from the educators' perceptions have implications for teaching, literacy development, classroom management, evaluation and, in particular, for the referral of these students for speech and language assessment. The context of the study is extremely important in that it was the only one over this five-year period that researched Aboriginal students in Canada; the rest of the doctoral research reviewed here involved ESL and international students (test-takers).

Baker (Reference Baker2010) explored one case of high-stakes language assessment – English proficiency assessment for teacher certification – in Quebec, Canada. She conducted three interrelated studies as a manuscript-based thesis (thesis by publication). The first examined the final administration of a writing test, the second, the pilot administration of a new replacement test, and the third, the first official administration of the new test. This sequential and mixed method design allowed the researcher to examine different aspects of the test's impact on stakeholders and raters. This program of research is critical in that

• • It integrated social and political values into the test validation process suggested by Cheng (Reference Cheng, Shohamy and Hornberger2008) and Moss et al. (Reference Moss, Girard and Haniford2006);

• • Test stakeholders, including the test-takers (teachers applying for certification) themselves, were not only consulted but also determined to a great extent the direction of the research program. Test-takers are increasingly being studied as they bear the highest stakes of all (Fox & Cheng Reference Fox and Cheng2007; Cheng & Deluca Reference Cheng and DeLuca2011);

• • The conflicting views and competing interests of the stakeholders were embraced. Similar results were found by previous studies (e.g. Fox Reference Fox2003; Qi Reference Qi2007).

Baker's three interrelated studies (Reference Baker2010) demonstrate a responsible and progressive approach to researching the assessment of language proficiency for professional certification, an area of language assessment that has been relatively less researched.

Shih (Reference Shih2006), Tan (Reference Tan2009) and Wang (Reference Wang2010) examined, primarily using survey methods (questionnaires and interviews), the impact and washback of testing programs elsewhere in the world. Shih (Reference Shih2006) used a case study approach and investigated a range of stakeholders’ perceptions of the Taiwanese General English Proficiency Test (GEPT) and its washback on schools’ policies, teaching and learning of English. The GEPT is a test of English language proficiency for Chinese learners of English at all levels and has gained societal acceptance for educational and employment purposes in Taiwan. The study was conducted at applied foreign language departments in two technological (vocational) institutes. The principal research methods used were interviews with the departmental chair, teachers, students and their immediate family members, and observations of English language courses. Such research methods are often used in traditional washback studies. However, Shih's inclusion of family members as stakeholders is unusual in studies of this nature. His findings revealed several features of the GEPT's washback on teaching and learning within the research context. The dissertation concluded by proposing a washback model of students’ learning; this will be very useful for future research into washback, as this area of empirical research is relatively new (20 years) and is in need of both theoretical and methodological guidance.

Wang (Reference Wang2010) explored the washback effects of the CET (College English Test) on teacher beliefs, interpretations and practices, in particular how the ‘teacher factor’ is manifested in the washback phenomenon as proposed by Watanabe (Reference Watanabe, Cheng, Watanabe and Curtis2004). Wang (Reference Wang2010) also investigated the pedagogical as well as the social and personal influences of the test on teachers’ beliefs and interpretations and practices within the Chinese tertiary context. Participants were 195 tertiary-level EFL teachers in the non-English programs. The study asked whether tests are a major constraint on College English instructional innovation in China and what aspects are relevant to this factor (e.g. teacher beliefs, knowledge or experiences) and present a barrier to the implementation of instructional change. A mixed methods approach of a teacher survey and in-depth case studies was used to collect data. Qualitative analysis involved the use of the constant comparative method, while quantitative analysis involved descriptive and inferential statistics (e.g. exploratory factor analysis, confirmatory factor analysis and structural equation modeling). The findings from this study suggest that the CET, coupled with various interrelated components of the ‘teacher factor’, does indeed create a washback effect. Given the complexities underlying the washback phenomenon, the educational change in curriculum and assessment is not sufficient on its own to entail teacher change in terms of pedagogical strategies – as previously suggested by Cheng (Reference Cheng2005) and Wall (Reference Wall2005). Wang (Reference Wang2010) emphasized that for fundamental changes in teacher practice to occur, they must be accompanied by other changes in teachers’ knowledge, beliefs, attitudes and thinking that inform such practice.

Tan's (Reference Tan2009) study, in a secondary setting in Malaysia, examined a change in the language of instruction for mathematics and science subjects from Bahasa Malaysia to English in Malaysia. This policy has two objectives: to promote student learning of mathematics and science, and to increase student proficiency in English. The Education Ministry also chose to create a washback effect by introducing a bilingual high-stakes secondary exit examination. As is common in applied linguistics, a new test (a revised testing system) is introduced to bring about changes: what is assessed becomes what is taught. This is an area that has been studied by many researchers (e.g. Cheng Reference Cheng2005; Wall Reference Wall2005). To examine the policy, the study drew on the insights and perspectives offered by the literature on educational change, content-based instruction and washback in language testing. This was a mixed methods study involving a longitudinal and cross-sectional research design. The results point to the complexity of educational change processes and indicate that the classroom implementation of this policy is affected by multiple factors.

Mullen (Reference Mullen2009) studied the impact of using a proficiency test as a placement tool. This is not an impact and washback study in the traditional sense, in that the study focused on the impact (area, extent and direction) of using a test for a purpose for which it was not originally designed. In particular, the research questions asked if a standardized proficiency test, the Test of English for International Communication (TOEIC), when used for placement purposes, had an impact, defined as one of the six qualities in the model of Test Usefulness (Bachman & Palmer Reference Bachman and Palmer1996). Increasing numbers of such studies of language testing are being carried out, as tests designed with one purpose are retrofitted for another (e.g. Fox Reference Fox2009; Doe Reference Doe, Tsagari and Csepes2011; Fox & Hartwick Reference Fox, Tsagari and Csepes2011). In Phase I of this sequential mixed methods evaluative study, 15 teachers and 677 North American university students who were studying L2 English answered questionnaires about their experiences of taking the TOEIC. First, the results showed that the group of 126 misplaced student participants differed significantly from the 551 correctly placed students on the dimensions of college English experience, age and first and second languages. The first of these was correlated most strongly with misplacement. These differences were found to relate to the TOEIC and its practice of predicting productive skill ability from formally measured receptive skill ability. Second, teachers who held Master's degrees had significantly fewer student failures in their classrooms than teachers with Bachelor's degrees. In response to the study's finding that impact exists in the form of student misplacement, in Phase II, teacher and student interviews focused on their experiences related to misplacement. Five teachers, who held a Bachelor's or Master's degree, and 13 students, eight misplaced and five correctly placed, supported the findings of Phase 1, showing that the practice of predicting productive ability from receptive ability resulted in misplacement for certain groups of students. In turn, this misplacement resulted in three consequences for stakeholders, involving the apparent validity of the TOEIC for making course-level changes, the misplaced students’ willingness to learn, and the teacher's decision to award pass grades to the misplaced. There is evidence in this study that when a proficiency test is used for placement purposes, misplacement occurs with identifiable groups of students and impacts all stakeholders. Studies of this nature are extremely important not only for Canada but also for English-medium universities around the world where a fast increasing number of international students choose to pursue their degrees in English. One of their first steps is to take an entrance English test and be placed into English programs using such test scores.

4. Raters, rating and scales

The doctoral dissertations discussed in this section, by Barkaoui (Reference Barkaoui2008), Isaacs (Reference Isaacs2010) and Kim (Reference Kim2010), are representative of a dominant trend in language testing and assessment research, namely, the investigation of the complex interactions between raters, ratings and scales in the evaluation of L2 spoken and written performances. All three researchers use mixed method approaches (Creswell & Plano-Clark Reference Creswell and Plano-Clark2007; Teddlie & Tashakkori Reference Teddlie and Tashakkori2009), explicitly acknowledging their perspective as pragmatic (Greene, Caracelli & Graham Reference Greene, Caracelli and Graham1989), and arguing that mixed method research inevitably triangulates findings by drawing from different sources of evidence about the same phenomena, thus increasing the potential validity of the findings. Taken together, these three dissertations extend an important line of validity inquiry, which has risen in prominence over the past decade due to the landmark work of such researchers as Fulcher (Reference Fulcher1996) and Turner & Upshur (Reference Turner and Upshur2002), who argued for empirically-derived scales in the measurement of written and spoken performance.

Barkaoui (Reference Barkaoui2008) explored the effects of scoring method (holistic or multiple-trait) and rater experience on the outcomes of ESL essay-rating performance. Holistic scales direct the raters’ attention to a general, overall impression of the quality of a written text, whereas multiple-trait scales direct their attention to specific features of writing (e.g. vocabulary, verb tense agreement and use of transitions). Interestingly, he found that both methods were measuring the same or similar constructs in his study; however, holistic methods appeared to support greater inter-rater reliability, whereas multi-trait methods allowed for finer grain evaluations of writing and supported higher rater self-consistency, especially for novice raters. His findings suggest that focusing on the criteria in a multi-trait scale helped to encourage more self-monitoring strategies, and required raters to attend to a wider range of features in the writing and use all of the rating criteria in the scale.

Barkaoui reports that novice raters differed from experienced raters in his study in that they varied more in their judgments, focused more intently on specific features of the scale, and took more time. None of these findings are particularly surprising, as they are consistent with those reported in the research literature on rater-scale interactions and the role of experience (and training) in rater response (see, for example, Cumming Reference Cumming1990; Hamp-Lyons Reference Hamp-Lyons and Kroll1990, Reference Hamp-Lyons1995; Weigle Reference Weigle1994; Barkaoui Reference Barkaoui2007). Barkaoui's dissertation is remarkable, however, in its meticulous analysis of both qualitative data, elicited through interviews and think-aloud protocols, and quantitative data, utilizing ‘a multi-faceted Rasch approach (FACETS) to examine the effects of scoring method on (a) estimates of examinee ability, (b) rater severity, (c) rater self-consistency, and (d) biased interactions between facets in the rating context’ (Barkaoui Reference Barkaoui2008: 100). He not only combines qualitative and quantitative approaches in examining rater-scale interactions, but also demonstrates understanding of the implications of mixed method research, which other researchers might choose to ignore or underplay. For example, in drawing on think-aloud protocols and interviews, he builds into his study procedures to assess the effects of thinking aloud, in order to examine the impact of these research techniques on his findings, inferences and conclusions. In so doing, he points out, ‘there are real advantages for combining methods in test validation research provided that one does not adopt a naive view that such approaches will be “unproblematic”’ (Barkaoui Reference Barkaoui2008: 226).

Kim (Reference Kim2010) also engaged in a complex, two-phase, mixed method validation study of a criterion-referenced scale or checklist. She investigated scale development and validation within the under-researched (Fox Reference Fox2009; Fox & Hartwick Reference Fox, Tsagari and Csepes2011) and still under-theorized (Alderson Reference Alderson, Fox, Wesche, Bayliss, Cheng, Turner and Doe2007) area of diagnostic assessment of L2 academic writing development in the context of classroom-based assessment. In phase one, the scale criteria for the Empirically-derived Descriptor-based Diagnostic (EDD) Assessment – the focus of her research – were elicited through think-aloud protocols provided by nine ESL teachers. The teachers rated and suggested the diagnostic feedback they would provide to students based on their evaluation of ten L2 essays, written in response to prompts drawn from the Test of English as a Foreign Language (TOEFL). As a result, criterion-referenced descriptors of academic writing were identified as the basis of the EDD checklist (39 descriptors were initially identified, further analysis reducing the number to 35).

In phase two, the EDD checklist was piloted, modeled and evaluated. In order to determine its generalizability across raters and essay prompts, seven ESL teachers used the EDD checklist to assess 80 TOEFL essays. They also completed a questionnaire in which they accounted for their scores and scoring process, and were subsequently interviewed about their use of the checklist. Scores were analyzed for their generalizability using FACETS. Kim also investigated the relationship between the EDD checklist and performance on other measures of ESL academic writing, comparing her EDD results with those of professional, Educational Testing Service (ETS) raters, who are trained and calibrated to mark TOEFL essays consistently in order to provide reliable (and defensible) scores on this high-stakes proficiency test. In her main study, ten ESL teachers assessed 480 TOEFL essays using the EDD checklist, and the scores were further analyzed to examine the dimensionality of the construct of L2 or ESL academic writing. Finally, classroom teachers’ questionnaire and interview responses were analyzed to evaluate positive and negative washback (Cheng, Watanabe & Curtis Reference Cheng, Watanabe and Curtis2004) at the classroom level as a result of the use of the EDD checklist.

Like Barkaoui (Reference Barkaoui2007), Kim applied FACETS to examine marking consistency and found evidence that supported Barkaoui's (Reference Barkaoui2008) findings, namely that it is hard to achieve a high degree of agreement between raters who are ‘not professional assessment raters’ (Kim Reference Kim2010: 137). She notes that the patterns in her own study are very similar to those reported by Barkaoui (Reference Barkaoui2008), who reported that teachers reached a level of agreement of 22.4% when using a nine-point holistic scale, which increased to 23.1% with a nine-point multi-trait or analytic scale; and that experienced raters exhibited a higher degree of agreement (26.1%) than novice raters (20%).

Kim's (Reference Kim2010) study is extremely complex methodologically, weaving qualitative and quantitative approaches together across two phases with multiple embedded studies and five complex research questions. Like Barkaoui, she defines her approach as ‘grounded on pragmatism’ (p. 57) and explicitly acknowledges that mixed methods research can ‘reduce biases and limitations inherent in a single method, while strengthening the validity of inquiry’ (ibid.). Of course, one of the dangers of such complex studies is that they are more difficult to control for coherence. In this regard, Barkaoui's study is more effective, but Isaacs (Reference Isaacs2010, see below), who also uses a mixed method approach in her consideration of pronunciation in speaking tests, utilizes the increasingly popular manuscript-based thesis option (Baker Reference Baker2010), which is particularly suited to complex, multi-step, mixed methods research, but remains quite rare in Canadian doctoral research.

At the beginning of her dissertation, Isaacs explains that she has chosen to prepare a ‘manuscript-based thesis, comprised of a collection of three studies that are part of the same overall program of research, in lieu of a “traditional” thesis’ (Isaacs Reference Isaacs2010: iv). All the studies were co-authored, with Isaacs as first author in each case, and one of the studies (Isaacs & Trofimovich Reference Isaacs and Trofimovich2011) had been published by the time she completed her dissertation. The advantages of such a manuscript approach in doctoral dissertation research are clear not only in terms of the inherent coherence of the dissertation itself, but also with regard to the next steps for doctoral graduates, who are enabled to apply for positions following their degree completion with a clear program of research and a list of publications in peer-reviewed journals.

Isaacs’ (Reference Isaacs and Trofimovich2011) program of research examined variation in raters’ judgments of L2 pronunciation, an under-researched construct, which ‘continues to be the EFL/ESL orphan’ (Gilbert, cited in Isaacs Reference Isaacs2010: 184). Specifically, she states that her purpose was to ‘better understand major constructs in L2 pronunciation research, improve current measurement practice, and, ultimately, reinvigorate the conversation on L2 pronunciation assessment, which has been virtually absent from the research agenda since the time of Lado (Reference Lado1961)’, (Isaacs Reference Isaacs2010: 24).

In study one, with her co-researcher (Isaacs & Trofimovich Reference Isaacs and Trofimovich2011), Isaacs takes a particularly novel approach, recruiting 30 music majors and 30 non-music majors to rate 40 L2 speech samples for three key features of pronunciation: comprehensibility, accentedness and fluency. With their musical experience, music majors are more attuned to sound qualities, have a heightened sense of the musicality, rhythm, flow, cadence and clarity of speech. Isaacs reports that music majors assigned significantly lower ratings than non-music majors for accentedness, particularly for low ability learners, but phonological memory and attention control did not influence their ratings. In study two, Isaacs and her co-researcher investigate the effects of scale length and rater experience on raters’ judgments of the same three features. The results showed that experienced and novice raters achieved a high degree of consensus about the highest and lowest scoring L2 speakers, but had difficulty differentiating between scale levels in the absence of guidance from the rating instrument. Study three is the direct outcome of studies one and two, as Isaacs confronts the key issue: ‘what is the construct?’ (Bachman Reference Bachman, Fox, Wesche, Bayliss, Cheng, Turner and Doe2007: 41), recognizing that the numerical scales used in studies one and two did not sufficiently articulate what separated one scale point from another in terms of specific performance features that raters should attend to. Like Barkaoui (Reference Barkaoui2008) and Kim (Reference Kim2010), Isaacs attempts to construct an empirically-driven L2 scale that ‘builds into the scoring rubrics, the qualities of the L2 performance that appear to be most salient to raters’ (Isaacs Reference Isaacs2010: 127).

5. Classroom-based research: Teaching, learning and assessment

Six of the dissertations considered in this review focused on teaching, learning and assessment in language classrooms across a range of program types, language learners and purposes for learning. We divide our discussion of these dissertations into two sub-sections: those investigating adult language learners and those investigating young language learners (YLLs). This approach is consistent with the research literature, which typically recognizes that the learning issues and language needs of YLLs differ considerably from those of older or adult language learners (McKay Reference McKay2006).

6. Vocabulary learning, lexical proficiency and lexical richness

Baba (Reference Baba2007), Cervatiuc (Reference Cervatiuc2007) and Douglas (Reference Douglas2010) investigated features of vocabulary acquisition and development, lexical proficiency and/or lexical richness and their impact on academic outcomes in adult, non-native English speakers (NNES). While Douglas (Reference Douglas2010) and Baba (Reference Baba2007) researched novice undergraduate writers engaging with the demands of university study in a second language, Cervatiuc (Reference Cervatiuc2007) explored the factors that 20 highly proficient NNES attributed to their success in vocabulary acquisition.

All of these researchers investigated vocabulary size and profiled the English lexical proficiency of their participants in order to address their research questions. Baba (Reference Baba2007) used a mixed method approach to assess the English summary writing of 68 Japanese undergraduate EFL students in Japan. After each student had written summaries of two topics, she interviewed them using a stimulated-recall approach to elicit the types of strategies the students reported using. She analyzed four features of lexical proficiency in their writing: vocabulary size, depth of vocabulary knowledge, word definition ability and lexical diversity. She also scored the summaries for their overall quality, ideational quality, use of academic sources and extent of copying from source texts. Using the scores assigned to the summaries as a guide, she then selected ten texts (five that were more effective, five less) for additional textual analysis, paying particular attention to issues of lexical repetition and topical structures. Discriminant analysis suggests that key differences between more or less successful writers (n = 33/group) were associated with productive English abilities (scores on summary writing, copying and word-definition ability), and both word definition and summary writing performance were strong predictors of higher or lower levels of proficiency.

Baba (Reference Baba2007) argues that much more work is needed to fully develop the construct of vocabulary knowledge as it is operationalized in tests, and to better understand the relationship between vocabulary knowledge and writing performance. She found evidence in her study that ‘lexical proficiency was crucial’ (p. 157) for the students in her study, although not every dimension of lexical proficiency was equally related to writing performance. She argues for continued research on the role of tasks in relation to contexts in eliciting writing, more probing interviews with writers to look in greater depth at their accounts of how and why they use the words they do, and longitudinal research to trace the development of their vocabulary knowledge over time.

Her calls for more longitudinal research into lexical richness in writing are answered in part by the work of Douglas (Reference Douglas2010), who investigated vocabulary knowledge as it was evidenced in the lexical richness of novice native speaker (NS) and non-native English speaker (NNES) undergraduate writers on a test of writing, and related initial measures of lexical richness to academic outcomes. He found that ‘vocabulary knowledge appears as an underlying variable related to students’ academic outcomes at university’ (p. 206).

Douglas’ quantitative study drew on a sample of 745 students (561 NS students and 184 NNES students) who were required to take a writing test upon entry to undergraduate programs at the university where the study took place. These students allowed their tests to be used anonymously for research purposes and filled in an information form, which provided the researcher with demographic information and a means of linking their initial test performance to university course outcomes. He used a corpus linguistic approach and lexical analysis tools available online to investigate the degree of lexical richness in the students’ written tests. A number of tools were used to analyze the test material for indices of lexical richness, based on a corpus developed by the researcher on the written entry test and responses to a provincial test written at the end of grade 12 as a high school matriculation requirement. Academic outcome measures were derived through detailed analysis of university transcripts.

Like Baba (Reference Baba2007), Douglas found evidence of the importance of vocabulary knowledge, lexical depth and breadth in eventual academic success at university. He found that NNESs who do not have the depth or breadth of vocabulary knowledge of their NS peers often persist (Fox: Reference Fox2005) in their academic study in spite of many failures and setbacks. As Douglas points out, however, ‘While [these] NNES students eventually graduated from university. . .they were faced with diminished academic outcomes in terms of Grade Point Averages, Length of Program, Courses Attempted and Not Earned, and Academic Standing’ (Douglas Reference Douglas2010: ii). Hierarchical regression analysis found that lexical richness upon entry to university was a significant predictor of academic outcomes. In this regard, Cervatiuc's (Reference Cervatiuc2007) research is also particularly important. Her research was aimed at obtaining insight into the factors that account for high levels of lexical proficiency amongst highly successful academic or professional adult NNES.

The 20 participants in Cervatiuc's study had arrived in Canada after the age of 18, scored at native-like levels on tests of vocabulary size (i.e. from 13,500 to 20,000 base words) and had vocabulary profiles consistent with native-like writing standards (Morris & Cobb Reference Morris and Cobb2004). She found that both situational and linguistic factors had an impact on their acquisition of vocabulary, but ‘the two underlying forces that drive their success and activate a unique combination of situational factors, L2 input, individual differences, and learning strategies are awareness of inner and outer resources and willpower to consistently employ these resources in order to make language and vocabulary gains’ (Cervatiuc Reference Cervatiuc2007: iv).

Baba's (Reference Baba2007) focus on metacognitive strategy use in her Japanese EFL writers and Douglas's (Reference Douglas2010) findings regarding persistence support Cervatiuc's (Reference Cervatiuc2007) conclusions regarding her 20 highly proficient NNESs. Drawing on a vocabulary size test, language proficiency assessments, interviews, questionnaires and samples of her participants’ writing, she cross-analyzes or triangulates findings, guided by principles of grounded theory (Glaser & Strauss Reference Glaser and Strauss1967; Hutchinson Reference Hutchinson, Sherman and Webb1997).

Although she does not describe it as such, Cervatiuc's study employs a mixed method approach with both quantitative and qualitative phases in her research design. In her own view, her study is mainly interpretative (qualitative dominant) because ‘meaning is quintessential in interpretative research (Bogdan & Bicklen Reference Bogdan and Bicklen1998) which focuses on participants’ perspectives or perceptions, values and attitudes’ (Cervatiuc Reference Cervatiuc2007: 73). Taking into account her participants’ perceptions of how they acquired, used and developed their lexical proficiency allows her to develop rich profiles of the 20 exceptionally successful L2 learners in her study and provides the initial groundwork for a theory of successful L2 vocabulary acquisition.

7. Conclusions and future directions

Examining such a large number of doctoral dissertations produced during the five-year period 2006–2011 in Canada was a daunting task. Guided by the ILTA citation framework, we have shown how these research studies are situated within the field of language assessment (and more broadly within applied linguistics) and outlined their scholarly contribution. To conclude, we would like to make two further points based on this review, and consider future directions for doctoral research in language assessment, both within Canada and internationally.

First, we can see the increasing use of mixed methods in comparison with single method studies. Indeed, 16 of the 24 dissertations considered here used mixed methods in their research, taking a pragmatic view in combining qualitative and quantitative research methods (see Table 1), although the strength of the mixed methods across these dissertations varies: some designs are driven by research questions and some by the researcher's choice of methods (Teddlie & Tashakkori Reference Teddlie and Tashakkori2009). When research methods are driven by research questions and the link between questions and methods is stronger, so are the results. When, on the other hand, the use of a method is method-driven (in that the researcher clearly chose a certain method due to a research tradition, or personal preference) the overall impact of the research may simply be technical, and the results weaker. All of these studies employed multiple study phases – sometimes conducted concurrently, but mainly consecutively. A majority of the researchers adopted a ‘program of research’ approach, reflecting the research direction of the Social Sciences and Humanities Research Council (SSHRC)Footnote ⁵ of Canada – the federal funding agency. These 16 dissertations demonstrate superior methodological strength and have made original contributions to the field of language assessment. Only four dissertations were purely qualitative, and four quantitative. Some studies (Fleming Reference Fleming2007) used more than one type of method, but both were survey methods drawn from the same research tradition. We recognize the distinct strength of the increased use of mixed methods and multiple stage studies as they naturally lead to a program of research rather than single studies. Although only two of the dissertations considered here (Baker Reference Baker2010 and Isaacs Reference Isaacs2010) were prepared as manuscript Ph.D.s, the advantages of this type of dissertation were evident. We predict an increasing number of both mixed methods and manuscript dissertations in Canada in the future.

Table 1

Canadian doctoral dissertations (2005–2011) in language assessment and testing, by methodological approach

Second, we observed some trends in study participants. Participants in the doctoral dissertations we reviewed tended to be L2 students (both immigrants and international students) who were in school (K-12) and enrolled in ESL programs in Canada. The majority were adult learners; only four studies dealt with young learners. Only one of the studies considered Aboriginal language learners (Sterzuk Reference Sterzuk2007), an indication that this is an under-researched area in the literature. This may be due to the nature of doctoral programs in Canada. Most of the dissertations reviewed were produced within applied linguistics departments rather than faculties of education. Some studies were devoted to the teacher participants alone (e.g. Baker Reference Baker2010, Wang Reference Wang2010) and some to student participants alone (e.g. Limbos Reference Limbos2005; Farnia Reference Farnia2006), but most dealt with both teachers and students (e.g. Shih 2007; Sterzuk Reference Sterzuk2007; Tan Reference Tan2009). There was also a balance between studies conducted in Canada (e.g. Abbott Reference Abbott2007 and Mullen Reference Mullen2009) and elsewhere in the world (e.g. China and Malaysia). Indeed, these studies reflect the social composition of Canadian society and two trends: the increasing number of international students on university campuses and English language learners at the school level.

The breadth of the literature that these researchers drew on to inform their work was particularly impressive, crossing disciplinary, epistemological and methodological boundaries. The depth of their inquiry was evident in their management of complex research designs and analyses. As a collection, breadth lent vigor and strength to the doctoral research considered here, while depth insured its quality. Taken together, these researchers have made an important contribution to scholarship in language testing and assessment in Canada and internationally.

What are the directions of future doctoral research on language assessment in Canada and internationally? First of all, we expect more research to be conducted at the intersection of assessment, teaching and learning, where testing is not viewed or researched in isolation, but rather embedded within the context where it occurs. We also predict that an increasing number of studies will explore the central role that assessment plays in teaching and learning, as in classroom-based assessment, for example. This type of research will be more interdisciplinary, contextually based, and involve multiple stakeholders. Methodologically, we will continue to see more studies with mixed methods, multi-methods and multi-phases: employing, in a sense, a ‘program of research’ approach. We will see more use of a (multiple) manuscript format which enables doctoral researchers to firmly establish a program of research, become actively involved with the dissemination of research findings, and publish during the course of their doctoral study. This format allows for a more mentored, collaborative, and team-based approach to doctoral study and research, and a more comprehensive preparation of doctoral researchers for a wide range of academic and research careers. It is in these doctoral research studies that we see the future of our field.