Introduction

Testing second language (L2) learners' knowledge of speech acts productively through discourse completion tests (DCTs) is complicated by the need for rating responses. Not only does this require a significant outlay of resources, but the rating process itself introduces additional variance into scores as the measurement of learners' ability and of the difficulty of items is mediated by the raters. It is therefore desirable in practice that ratings are consistent, but it is just as important to understand inconsistencies and improve rater training to preempt them.

A useful tool for understanding the relationship between test-taker ability, rater judgments and item difficulty is many-facet Rasch measurement (MFRM), implemented in the program FACETS. This paper applies MFRM to data from a productive speech act test.

Testing Knowledge of L2 Speech Acts

Work on speech acts has been a central feature of acquisition and assessment research in interlanguage pragmatics. Assessment studies are more recent, with the first large-scale test development and validation study by Hudson et al. (1995) actually being a method comparison study, which employed traditional written DCTs, oral DCTs, multiple-choice DCTs, roleplays and self-assessments with Japanese ESL learners in Hawai'i. Their study was replicated by Yoshitake (1997) and adapted for English-speaking learners of Japanese by Yamashita (1996). Brown (2001) shows that the reliability of all the instruments was acceptable except the multiple-choice DCT, which had been especially difficult to construct due to the challenges of writing distractors that were clearly unacceptable but still attractive. This is a somewhat unfortunate finding, as the multiple-choice DCT was the instrument with the greatest practicality. Written and oral DCTs were less practical because they required human raters but could at least be administered to groups of test takers.