Introduction
Although task-based language teaching (TBLT) and its companion, task-based language assessment (TBLA), have gained in prominence in research and teaching, research projects have been limited due to small sample sizes and a paucity of sharable data. In addition, although comparisons of learner development over time provide critical information that can inform second language acquisition (SLA), such longitudinal data are scarce, both in general and in TBLA/T. Moreover, data in languages other than English are equally difficult to access. Thus, this resource development project focuses on oral proficiency assessment data and reports on the design and plans for implementing an internationally available, multilingual corpus of speaking test data. The potential for the corpus is reviewed in light of preliminary research that explored lexical complexity and lexical diversity that employed parts of the corpus as a resource in language assessment research. The project will eventually result in a multilingual, sharable database of student oral language samples pre and post study abroad (SA) on the same tasks in each language that will allow for a variety of projects including but not limited to longitudinal comparisons of proficiency gains, task analyses, and complexity, accuracy and fluency (CAF) analyses.
Literature review
To develop this database, we will examine the differences between test taker performance on the same tasks, based on the test taker's holistic proficiency rating and the alignment of the task to a proficiency rating on the same scale.
‘Communicative task’ has long been a focus of research in SLA and applied linguistics (e.g., Gass, Mackey, & Ross-Feldman, Reference Gass, Mackey and Ross-Feldman2005; Taguchi, Reference Taguchi2007). Long and Norris (Reference Long, Norris and Byram2000) point out that in TBLA the task is the ‘primary unit of analysis’ (p. 600). Thus, the definition of ‘task’ is central to the study reported on in this paper, the development of the database, and the eventual possibilities for research that will emerge from this resource.
Although many conceptualizations of communicative task appear in the literature, this research relies on Luoma (Reference Luoma2004, p. 31), ‘Speaking tasks can be seen as activities that involve speakers using language for the purpose of achieving a particular goal or objective in a particular speaking situation.’
Methods
Instrument
The Simulated Oral Proficiency Interview (SOPI) is a tape-mediated test of speaking proficiency that relies on a test booklet and two tape recorders (or a language lab) to elicit language in place of a human tester (Malone, Reference Malone2000). The SOPI has demonstrated high interrater reliability with the American Council on the Teaching of Foreign Languages (ACTFL) OPI. One advantage of such an approach in language research is that test takers will take the same or similar tasks both within and across the languages of the test. All participants took the SOPI within the first weeks of arriving at their SA program and before departing. Beyond the sample reported here, it is planned to digitize all SOPI responses for the corpus.
Data
The backbone of the new corpus is based on the data collected in SA research reported in Vande Berg, Connor-Linton, and Paige (Reference Vande Berg, Connor-Linton and Paige2009). The original research questions centered on developmental gains in SA programs using a pre- and posttest design, including a student questionnaire, intercultural assessment data and information on the influence of SA on student professional gains. This corpus focuses on the language data.
Audio processing
The methods used to demonstrate the potential of the corpus are open-source audio analysis and natural language processing (NLP) tools. The audio data analyzed here was extracted using the Pratt software tool, specifically, the Syllable Nuclei v2 Praat script (De Jong & Wempe, Reference De Jong and Wempe2008). This allowed for a detailed operationalization of the CAF measures proposed for analysis.
Natural language processing
Transcribed data was analyzed using the Spanish-language NLP module from the Natural Language Toolkit (NLTK) package for Python, version 3.7, and was implemented using PyCharm, version 2020.1. The NLTK module allowed for word and bi-gram identification, part-of-speech tagging, and visualization of frequencies relevant to research (Bird, Klein, & Loper, Reference Bird, Klein and Loper2009).
Results
CAF measures
The initial research using the corpus investigated the relationship between rating and CAF measures in a sample of L2 Spanish learner data (Myers, Reference Myers2020). The first research question explored the possible differences in CAF produced by SA learners rated at different levels. The second research question explored how CAF measures vary depending on the proficiency level (ACTFL Intermediate, Advanced, or Superior) with which the task is aligned. The analysis used six measures of production as dependent variables (Myers, Reference Myers2020). The variables of interest were distilled from a battery of measures previously featured in the CAF assessment literature (Myers, Reference Myers2020) and follow the recommendation that speech acts are a realistic measure of proficiency sufficient to capture the dynamic nature of language production under assessment (Norris & Ortega, Reference Norris and Ortega2009).
The results from the CAF analysis (n = 14) reveal that the linguistic output of CAF measures predicated reliability between levels more so than between sub-levels within a proficiency band. The most interesting result was reported between the Advanced and Superior levels. Statistical analyses using independent samples t-tests found significant results in three CAF measures: proportion of error-free AS-units, dysfluencies, and dysfluency time proportion. Given the small samples size, results are encouraging because they suggest that continued research utilizing CAF measures should be fruitful in identifying which linguistic features can reliably differentiate between levels and sub-levels on the ACTFL rating scale.
Lexical diversity
The motivation for using NLP tools is to better understand the relationship between lexical frequency and rating on the ACTFL proficiency scale. Lexical diversity has been shown to correlate with better ratings in both written and spoken tasks (e.g., Yu, Reference Yu2010). Although more sophisticated statistical modeling of lexical diversity is available to researchers (e.g., Jarvis, Reference Jarvis2013), the use of frequency counts as type/token ratios (TTR) is still a valid way to reliably differentiate speaker proficiency level as a first-pass metric, in order to rank an individual's word use in a more informative frequency band compared against frequencies in a large corpus (e.g., Amoroso, Reference Amoroso2015) like the Corpus of Contemporary American English or El Corpus del Español (Davies, Reference Davies2008). Preliminary results suggest that lexical diversity and TTR will be a reliable predictor of rating in the high frequency band. The sample (n = 3) compared three TTR ratios within one task (Task 5) and found varying distributions in lexical choice in the ten most frequent tokens in the task. Although not statistically significant, X2 (df = 2, N = 3) = 5.6445, p = .227, due to the single task analysis with a small dataset, the results demonstrate the potential for research at the lexical level.
Conclusion
The initial research using the corpus explored differences in CAF features by rating level and by communicative task. At a minimum, the results provide a window into the wide range of linguistic research that the corpus can inform. Additionally, post-hoc research focused on lexical diversity as a predictor of SOPI rating and demonstrates the potential for targeted comparison of lexical distribution across tasks and proficiencies. The results reported here demonstrate the potential for research to investigate the relationship between linguistic features (CAF measures) and proficiency rating. At the lexical level, patterns in lexical distribution over proficiency level and task can be investigated in order to answer research questions related to oral proficiency rating.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/S0261444820000567
Bradford Salen joined the Georgetown Linguistics Ph.D. program in 2018. He received his B.A. in Linguistics and his M.A. in Second Language Acquisition from the University of Maryland. His research interests include computational approaches to second language processing, language acquisition, language assessment, and psycholinguistics.
Margaret E. Malone (Ph.D., Georgetown University) is Director of the Assessment and Evaluation Language Resource Center and Research Professor at Georgetown. She is also Director of the Center for Assessment, Research and Development at ACTFL. She has three decades of experience in language test development, materials development, delivery of professional development and teacher training through both online and face-to-face methods, data collection and survey research, and program evaluation. Her current research focuses on language assessment literacy, oral proficiency assessment, the influences of the Seal of Biliteracy on language teaching and learning and the development of short-cut measures of proficiency. She teaches courses on language testing, language acquisition, and SA.
