INTRODUCTION
Analyses of learner language have always been central to second language acquisition (SLA) research. Indeed, Corder's (Reference Corder1967) and Selinker's (Reference Selinker1972) seminal works, argued by many scholars to symbolize the genesis of the field (e.g., Ortega, Reference Ortega2009; but cf. Thomas, Reference Thomas, Herschensohn and Young-Scholten2013), both regard learner production as fundamental. In the last two decades, tasks have been used by numerous researchers to elicit learner production. The interests of such studies range from testing theoretical models (e.g., the cognition hypothesis) to exploring instructional treatments and developing novel assessments.
The goal of the current article is to review studies that analyze task-based learner language. We present a systematic review of the substantive foci addressed in this line of research. Specifically, among other issues and practices, we consider the extent to which different linguistic and interactional features of task-based learner production have been examined. We also assess a number of methodological features (i.e., research designs, data analyses, and reporting practices) found in the domain. In doing so, we connect our work to the growing body of methodological syntheses (e.g., Plonsky, Reference Plonsky2013) and to the broader methodological reform movement taking place in applied linguistics (Byrnes, Reference Byrnes2013).
LITERATURE REVIEW
Since Long (Reference Long, Hyltenstam and Pienemann1985), tasks have been used in diverse empirical and pedagogical domains, serving two main purposes: as (a) a primary unit of instruction and (b) a tool for eliciting language production. The growing interest in this area has largely been motivated by the fact that tasks provide contexts for meaningful second language (L2) learning and use. Accordingly, tasks have become essential to both L2 research and pedagogy, and have linked the two domains (Mackey, Reference Mackey and Mackey2007).
Tasks have been used as a tool to elicit learner production in two major types of designs. In some instances, the tasks and the language produced in carrying them out function as a treatment or intervention, the effects of which (i.e., L2 development) are made known on a posttest. We refer in the present study to this major design type as “task-based learning outcome” or “task-as-treatment” studies. Oftentimes, however, researchers collect samples of task-based learner production in order to understand the impact of task design or condition on individual features or more global assessments of learner language. We refer to this type of research as “task-based learner performance” studies.
These two major design types serve as the organizing principle for the first two of three sections of the literature review that follows. We focus throughout on the learner language induced in the context of tasks as defined by Bygate, Skehan, and Swain (Reference Bygate, Skehan, Swain, Bygate, Skehan and Swain2001): “A task is an activity which requires learners to use language, with emphasis on meaning, to attain an objective (p. 11)”. In the third section, we describe previous reviews studies that, like ours, have engaged in methodological synthesis.
Task-Based Learner Performance Studies
Many studies of task-based production are interested in testing theoretical claims or hypotheses related to task design (e.g., task complexity, task types, task modality) and implementation (e.g., task planning, task repetition) as well as learner characteristics (e.g., proficiency, anxiety, task familiarity) during task performance. Such studies employ both monologic and dialogic tasks, and many different features of learner language and interaction are analyzed. For instance, a number of studies have tested Robinson's (Reference Robinson2001) cognition hypothesis and Skehan's (Reference Skehan2009) trade-off hypothesis, both of which focus on the effect of task complexity on learners’ linguistic performance and interactional competence (see Jackson & Suethanapornkul, Reference Jackson and Suethanapornkul2013, for a review).
Whether they involve monologic or dialogic performance, most task-based learner production research has examined linguistic features of learner language. Some studies analyze individual grammatical and/or lexical items. For example, Michel (Reference Michel2013) was interested in the use of conjunctions by students who performed tasks at two levels of complexity. Other studies analyze broader performance-based measures within the complexity, accuracy, and fluency (CAF) framework (Housen, Kuiken, & Vedder, Reference Housen, Kuiken and Vedder2012; Michel, Kuiken, & Vedder, Reference Michel, Kuiken and Vedder2012). Complexity traditionally indicates L2 learners’ ability to use a range of sophisticated structures and vocabulary in the L2 (Bulté & Housen, Reference Bulté, Housen, Housen, Kuiken and Vedder2012; Ellis & Barkhuizen, Reference Ellis and Barkhuizen2005). Accuracy refers to learners’ ability to produce target-like language (see Foster & Wigglesworth, this volume). And fluency is characterized by the rate and/or amount of production in a given time (Segalowitz, Reference Segalowitz2010). According to Housen et al. (Reference Housen, Kuiken and Vedder2012), “CAF emerge as the primary epiphenomena of the psycholinguistic processes and mechanisms underlying the acquisition, representation and processing of L2 system” (p. 2). It is not surprising then that these constructs appear frequently in SLA, language assessment, and other areas of applied linguistics.
Despite their frequent application and intuitive appeal, there is little consensus surrounding measures of CAF. In line with recent discussions of the importance of including various types and levels of these linguistic constructs (Norris & Ortega, Reference Norris and Ortega2009), numerous submeasures have been developed for CAF. Michel, Kuiken, and Vedder (Reference Michel, Kuiken and Vedder2007), for example, included five measures of accuracy (total number of errors per AS-unit, number of lexical errors per AS-unit, number of omissions per AS-unit, percentage of self-repairs, and ratio of self-repairs to errors), four measures of complexity (total number of clauses per AS-unit, subordination index, percentage of lexical words, and Guirad's Index), and three fluency measures (the ratio of syllables per minute in unpruned speech, the ratio of syllables per minute in pruned speech, and number of filled pauses per 100 words). Their study is not atypical. Both within and across studies, many definitions and measures of CAF and of other features observed in task-based production can be found. One of the primary goals of the present study is therefore to arrive at a better understanding of the extent to which different features of learner production have been examined in the context of task performance.
Looking across one subset of task-based learner performance studies, Jackson and Suethanapornkul (Reference Jackson and Suethanapornkul2013) meta-analyzed investigations into Robinson's cognition hypothesis. Their findings indicate that resource-directing tasks are associated with (slightly) higher accuracy and (slightly) less fluency (see Sasayama, Malicka, & Norris, forthcoming, for a book-length synthesis and meta-analysis on the relationship between task complexity and learner output). However, little is known about precisely which kinds of learner production data have been examined in recent task-based studies.
Task performance in both written and oral modes has also been compared in terms of interactional features (Payant & Kim, Reference Payant and Kim2015) such as feedback moves. Interaction-based studies of task-induced production often examine the presence of interactional and nonlinguistic features believed to promote L2 development. For instance, the occurrence of learners’ attention to linguistic forms such as corrective feedback, negotiation of meaning, language-related episodes (LREs, i.e., when learners talk about, question, and/or self-or-other correct language use; see Swain & Lapkin, Reference Swain and Lapkin1998), and form-focused episodes (FFEs, i.e., learners’ attention to linguistic forms during meaning-oriented tasks) have all been identified. Such studies exhibit methodological and theoretical parallels to the first phase of research in the interactionist tradition of SLA; see commentaries in the introduction to Mackey's (Reference Mackey and Mackey2007) volume and in Plonsky and Gass's (Reference Plonsky and Gass2011) methodological synthesis, both in the domain of interaction. Both task-based and interactionist researchers have examined in some depth the features of learner dialog (e.g., clarification requests, repairs) in different contexts (e.g., lab vs. classroom) and participant configurations (e.g., equal vs. unequal proficiency, number of interlocutors) (e.g., see Gass & Varonis, Reference Gass, Varonis, Gass and Madden1985).
Some scholars have gone beyond linguistic- and interaction-based targets to study the content of learner production as a function of task features or conditions. Among other dependent variables, Sample and Michel (Reference Sample and Michel2014) examined participants’ ability to complete a task, which the authors argued to be “more important than any other feature of task performance” (p. 30). Such efforts address the necessity to confirm the quality or adequacy of task performance (De Jong, Steinel, Florijn, Schoonen, & Hulstijn, Reference De Jong, Steinel, Florijn, Schoonen, Hulstijn, Housen, Kuiken and Vedder2012; Ortega, Reference Ortega and Ellis2005) as well as the assumptions researchers make about task features. A given task may be complex from researcher's or teacher's perspective, but not from that of the participants (see Révész & Gurzynski-Weiss, this volume). More fundamentally, examinations of task-induced learner language that fail to consider the semantic content therein run counter to many of the theoretical and practical principles underlying a task-based approach to L2 learning and teaching (Pallotti, Reference Pallotti2009).
Situational features of tasks have been manipulated as well to examine their impact on linguistic and interactional features of learner production (e.g., Philp, Adams, & Iwashita, Reference Philp, Adams and Iwashita2014). Conditions vary, for example, as to whether the task is completed in the oral or written mode (e.g., Révész, Reference Révész2009) and whether the task is completed face-to-face with an interlocutor or via computer-mediated communication (CMC) (e.g., Gurzynski-Weiss & Baralt, Reference Gurzynski-Weiss and Baralt2014; see Ziegler, this volume, for a review). Although lab vs. classroom and foreign- vs. second language effects are rarely examined in primary studies, there is theoretical and empirical evidence to support the idea that these settings may moderate certain types of findings (Fujii & Mackey, Reference Fujii and Mackey2009; Plonsky & Oswald, Reference Plonsky and Oswald2014; Ziegler, in press; Ziegler, this volume).
The present study brings together these various strands of research to understand the extent to which and means by which different designs, task conditions, and features of learner language have been examined.
Task-Based Learning Outcome Studies
The second group of studies we introduce focuses on the role of tasks in L2 development. This line of research generally employs a pretest–posttest (task-as-treatment) design. Similar to the studies described thus far, developmental or interventionist studies often manipulate task features, situational/contextual, and/or participant variables. Though not mutually exclusive, the two types of designs differ in that this line of task-based research also examines how different task features and conditions impact subsequent language learning. This type of research runs parallel in many ways to the second phase in the interactionist tradition of SLA (Plonsky & Gass, Reference Plonsky and Gass2011).
By implementing an interventionist approach, researchers are able to estimate the effects of different task types or conditions as well as task-based production on subsequent development in L2 lexis and grammatical forms (e.g., Kim, Reference Kim2012). The treatment groups in Révész (Reference Révész2009), for example, completed a picture description task in one of four conditions: +/−task support (access to photos while describing them), +/−interactional feedback in the form of recasts. The results of the study presented both the number of recasts provided during the tasks (i.e., an interactional feature) and a comparison of the gains in performance on the targeted linguistic feature (past progressive) across the four task conditions. Similarly, Kim and Tracy-Ventura (Reference Kim and Tracy-Ventura2013) compared the effects of different task repetition conditions (task vs. procedural repetition) on learners’ ability to produce language that is complex, accurate, and fluent. Among other results, the study found that both conditions led to a small increase in fluency on posttests, but that procedural repetition (and not task repetition) induced a boost in syntactic complexity.
Recently, the development of pragmatic features in a task-as-treatment approach has begun to receive attention as well. For instance, Kim and Taguchi (Reference Kim and Taguchi2015) examined the role of task complexity in the learning of request-making expressions by using collaborative writing tasks with Korean junior high school learners. Their findings indicated that both simple and complex tasks were equally beneficial for immediate learning outcomes, but only learners who performed more complex tasks that required reasoning processes maintained their learning four weeks after task performance.
Several meta-analyses have addressed the effects of tasks on L2 development. Keck, Iberri-Shea, Tracy-Ventura, and Wa-Mbaleka (Reference Keck, Iberri-Shea, Tracy-Ventura, Wa-Mbaleka, Norris and Ortega2006) found strong evidence of learning resulting from task-based interaction, especially when tasks were designed in a way that required use of the target feature(s). Cobb (Reference Cobb2010) replicated and expanded on Keck et al. by including more recent studies, but also restricted her domain to studies focused on gains in L2 grammar only. Both of these meta-analyses could be considered to have addressed subdomains within the scopes of Mackey and Goo's (Reference Mackey, Goo and Mackey2007) meta-analysis on the effects of L2 interaction and Plonsky and Gass's (Reference Plonsky and Gass2011) synthesis of interactionist research methods and outcomes.
Complementary to these meta-analyses, which have largely examined the effects of task-based instruction and interaction, the current study reviews the linguistic and interaction features analyzed in this body of research.
Methodological Synthesis in L2 Research
As we state in the title and introduction to this article, we are not only interested in the target features and analyses of task-based learner production but in the methodological practices of this domain as well. Syntheses of research practices, part of the broader conceptual and methodological tradition of meta-research across the sciences (see Ioannidis, Fanelli, Dunne, & Goodman, Reference Ioannidis, Fanelli, Dunne and Goodman2015), seek to both describe and evaluate individual domains of inquiry. Certain methodological features are examined that are not necessarily associated with higher or lower quality but that help illuminate the domain and perhaps point to areas in need of further investigation (e.g., the change over time in lab- vs. classroom-based research found in Plonsky & Gass, Reference Plonsky and Gass2011). Other features are indeed concerned with methodological rigor, validity, and transparency (i.e., study quality; see Plonsky, Reference Plonsky2013) such as the use of pretests in quasi-experiments and full reporting of data.
Some methodological syntheses review a particular feature or technique across subdomains. Derrick (Reference Derrick2015), for example, synthesized reporting practices related to the development and use of data collection instruments in L2 research. Plonsky and Gonulal (Reference Plonsky and Gonulal2015) reviewed the use of factor analysis in applied linguistics. Other methodological syntheses, including the present study, consist of methodological reviews of a particular substantive domain. For example, Liu and Brown (Reference Liu and Brown2015) synthesized research and reporting practices in studies of written corrective feedback. Even closer to the focus of the present study, Ziegler (Reference Ziegler2016) reviewed methodological features of interaction-based research comparing synchronous CMC (SCMC) and face-to-face contexts.
Results in this small but growing body of research are mixed but do not generally paint a very favorable picture of L2 research methods. A thorough review of findings would not be appropriate here. However, one trend observed across several domains is a lack of transparency. L2 researchers often fail to report critical data such as standard deviations (along with their means), test statistics, reliability estimates, and so forth. The field of L2 research has also been weakened by dogmatic adherence to and overuse of null hypothesis significance testing (see Norris, Reference Norris2015; Plonsky, Reference Plonsky and Plonsky2015b), a problem compounded by the small samples common to the field (Plonsky, Reference Plonsky2014).
The Study
There are two motives behind the current study. The first is an interest in providing the field of applied linguistics with a better understanding of the features of learner production that are examined in research on task-based teaching and learning. Our second motive is an interest in describing and evaluating the research and reporting practices in this domain. One motive, corresponding to Research Question 1, is chiefly substantive in nature, and the other, which corresponds to Research Questions 2 and 3, is methodological. Our research questions are as follows:
-
1. To what extent and by which means have different features of learner production been analyzed in task-based L2 research?
-
2. To what extent have different designs, analyses, and reporting practices been employed in task-based L2 research?
-
3. How reliable (psychometrically speaking) are researchers’ analyses of task-based production?
METHOD
The procedures used to carry out this study follow what is considered best practices in the realm of synthetic or meta-analytic research in applied linguistics (see Plonsky & Oswald, Reference Plonsky, Oswald and Plonsky2015). The focus of the current study, however, required a somewhat unique approach. Whereas meta-analyses of L2 research examine the effect sizes resulting from a treatment or intervention, the current study addressed substantive and methodological features (not findings) in studies of task-based learner production.
Inclusion Criteria and Search Techniques
The first step in this study involved defining the parameters by which primary studies would be selected. The resulting set of inclusion criteria were developed and applied, leading to a total sample of 85 unique reports.
The first and main criterion was an analysis of task-induced learner production. Within this broad parameter, all theoretical approaches, language/interactional features, modes, designs, and contexts were considered. Some studies identified in the search included treatments involving task-based activities and/or production but did not analyze what the learners produced during the tasks. These studies were excluded (e.g., de la Fuente, Reference de la Fuente2006). Studies that only examined teacher or learner perceptions of task-based teaching and learning were also excluded (e.g., Chen & Brown, Reference Chen and Brown2012; Zheng & Borg, Reference Zheng and Borg2014). In the interest of synthesizing and assessing current research in this area, only studies published in the last 10 years, 2006–2015, were eligible.
Although an inclusive approach is generally preferred (Plonsky & Brown, Reference Plonsky and Brown2015; Plonsky & Oswald, Reference Plonsky, Oswald and Plonsky2015), the current study included only published studies. This choice was informed by three considerations. First, the study was not necessarily interested in the entire population of research on task-based learner production but, rather, those that have likely had an influence on theory and practice in the domain. Second, published studies, which have gone through peer and/or editorial review, are likely to be of higher quality than unpublished ones. Third, excluding unpublished studies often leads to an overestimate of meta-analytic effects (i.e., publication bias; see Oswald & Plonsky, Reference Oswald and Plonsky2010), which were not considered in the present study. Although dissertations and conference presentations may be of high quality, access to these types of outlets is often restricted, thus reducing their influence on theory and practice.
A number of search techniques were applied to identify primary studies that met the criteria just described. The first and most productive searches were conducted via three databases: Linguistics and Language Behavior Abstracts (LLBA), Google Scholar, and Google (keywords: task-based, TBLT, second language, L2). In addition, the samples of two meta-analyses with scopes closely related to that of the current study were reviewed to identify additional studies (Cobb, Reference Cobb2010; Jackson & Suethanapornkul, Reference Jackson and Suethanapornkul2013). Two special issues, one in Applied Linguistics (Housen & Kuiken, Reference Housen and Kuiken2009b) and one in International Review of Applied Linguistics (Robinson & Gilabert, Reference Robinson and Gilabert2007) were also examined for studies that might meet the inclusion criteria.
The final sample consisted of 85 unique reports, published from 2006 to 2015 (see Table 1). Full references for the sample will be made available by the authors upon request. Further searches could have been applied using other techniques (e.g., forward citations, email list requests) and extensions on those already employed (e.g., additional database searches). The sample should not be considered exhaustive. Nevertheless, we have no reason to believe that the sample obtained for the present study is not representative of the domain of interest.
Studies of Task-Based Production From 2006 to 2015
Note. k = number of studies.
In addition to chronological dispersion, a range of different journals are represented in the sample. As shown in Table 2, a small number of reports were published in a variety of prominent applied linguistics journals with both general and specialized orientations. And although all studies were published in English, the journals represented are quite international in scope. This domain is clearly of interest to scholars from a broad range of countries and contexts around the world.
Studies of Task-Based Production Across Journals
Note. The remaining journals each contributed one study to the sample: Annual Review of Cognitive Linguistics; ELT Journal; EUROSLA Yearbook; Heritage Language Journal; International Journal of Applied Linguistics; International Journal of Research Studies in Language Learning; Journal of Applied Linguistics; Journal of Asia TEFL; Language Assessment Quarterly; Language Learning Journal; Latin American Journal of Content & Language Integrated Learning; Pazhuhesh-e Zabanha-ye Khareji; Southeast Asian Journal of English Language Studies; Teachers College, Columbia University Working Papers in TESOL & Applied Linguistics; ReCALL; RESLA; and TESL Canada.
Data Collection
A coding scheme was developed to extract substantive and methodological information from each study in the sample. Six categories of data were collected: (a) study identification (e.g., author, journal), (b) study design and setting (e.g., class vs. lab, target language), (c) production targets and features (e.g., mode, grammar, complexity), (d) coding process (e.g., corpus-based, by hand), (e) statistical analyses, and (f) reporting practices associated with transparency (e.g., reliability, effect sizes, confidence intervals). The complete set of items appears in Table 3, and our data collection instrument will be made available on the IRIS (data collection instruments for research into second language learning and teaching) database (Marsden, Mackey, & Plonsky, Reference Marsden, Mackey, Plonsky, Mackey and Marsden2015).
Coding Scheme Categories and Items
Note. All items coded dichotomously (i.e., 0 = absent, 1 = present) unless marked with * to indicate items with an open-ended response.
After an initial version of the coding scheme was developed, Plonsky coded three studies. In doing so, the operational definitions of a small number of items were refined. He then trained a second rater, his research assistant (RA), who has taken graduate courses in both task-based teaching and learning and in meta-analysis. After discussing the goals of the study and each item in the coding scheme, a step that led to the refinement of several operational definitions, the RA and Plonsky independently coded the remainder of the sample. Plonsky coded a total of 44 studies, and the RA coded 41.
Once all 85 studies were coded, Plonsky, recoded 10% of the sample (nine studies) that had been coded previously by the RA to estimate the interrater reliability of our data collection process. The overall agreement rate was 95%. Because our instrument was composed almost entirely of categorical items, which are susceptible to chance agreement and inflated reliability estimates (see Plonsky & Derrick, in press), we also calculated Cohen's kappa (κ = .91). Both indices demonstrated an overall high level of interrater agreement. However, upon closer inspection, we were concerned about four individual items. The two raters disagreed four out of nine times on the analyses of learners’ grammar and vocabulary, respectively, three times on the total number of statistical tests conducted, and three times on study design. Plonsky then recoded these four variables for the 41 studies that had been coded by the RA. The revised values were used in all subsequent analyses.
Analysis
The analyses involved in the current study were very straightforward. Research Questions 1 and 2 were addressed by calculating frequencies and percentages of the (categorical) substantive and methodological features found in the sample. Measures of central tendency were calculated for the few continuously measured variables such as sample size. Research Question 3 was addressed by calculating the mean, standard deviation, and 95% confidence intervals of the reliability coefficients associated with coding of task-based production as reported in individual studies. A small number of studies reported reliability coefficients for multiple target features (e.g., number of LREs + accuracy). In such cases, the estimates were combined (averaged) in order to avoid data dependencies at the meta-analytic level (Lipsey & Wilson, Reference Lipsey and Wilson2001). Reliability estimates that are reported may be higher than those that go unreported or that are never calculated in the first place. Consequently, the meta-analytic estimate provided in our results may be somewhat inflated.
RESULTS
Demographics, Contexts, and Research Designs
Frequencies and percentages of different demographics, research designs, and contexts were calculated for the sample of studies examining task-based learner production. Table 4 presents the results for two demographic features: learner proficiency and age. The majority of the studies in this area have collected data from what authors have referred to as intermediate learners. In nearly all cases, this label should be considered impressionistic; very few authors provide any justification (e.g., TOEFL score) for their assessment of proficiency. The data also reveal a very strong tendency toward research on adult learners.
Learner Demographics
*Totals do not equal 100 because some studies included participants at multiple levels.
Several design features in this domain were examined. The first, sample size, is closely related to the demographics just described. Samples ranged from 1 to 968. Most, however, were closer to the median of 36 (interquartile range = 28). Studies were also coded for two major design types: (a) task-based learner performance (k = 57, 67%) vs. (b) task-as-treatment (k = 28, 33%).
The study included three variables related to research context or setting, the results of which are found in Table 5. Again, several clear patterns emerge. Studies of task-based learner production are more classroom- than lab-based, and the vast majority of research in the domain is conducted in foreign-language rather than L2 settings. Most research in this area is also conducted with university learners, a finding that reflects the result above related to participant age.
Research Contexts, Settings, and Institution Types
This review also examined the different first languages (L1s) and target languages (TLs) being studied. As shown in Figures 1 and 2, this domain has relied on very few L1s and TLs, with English being the clear frontrunner in both categories. In fact, 86% of the studies in the sample are based on data from users of English as an L1 or TL.
Participants’ First Languages (%)
Participants’ Target Languages (%)
Analyses of Learner Production Data
In order to first understand the data collection procedures of research on task-based production, studies were coded for task modality (oral, written), as well as whether tasks were carried out via face-to-face or computer-mediated communication. The results for this aspect of the analyses appear in Table 6.
Modality of Task-Based Learner Production
*Categories are not mutually exclusive.
One of the main goals of this study was to better understand the features that have been studied in task-based production. Each study in the sample was coded for the different targets of analysis. These included linguistic domains such as grammar and pronunciation, the well-known CAF framework, use of participant L1, interaction features (e.g., LREs), and the quality of their task completion (i.e., was the task completed successfully? see Sample & Michel, Reference Sample and Michel2014).
The results for this phase of the analysis appear in Figures 3 and 4 and are presented as the percentage of studies in the sample that analyzed different features of learner production and interaction. We see first of all a variety of target features. It is also clear from the data that interest in learner production resulting from task-based research has focused most heavily on grammar, interactional features, and vocabulary; among CAF, the most often studied feature is accuracy. Considerably less attention has also been paid to pronunciation, fluency, and complexity, and very few studies have examined pragmatics and use of the L1. The quality of the task content produced by the participants (e.g., task completion) has been rarely studied.
Target Features in Task-Based Production (%)
Interactional Features Examined (%)
In order to better understand this domain, we also coded for subtypes of interactional features, a broad and widely studied aspect of this domain. Figure 4 presents the results for this phase of the analysis. Approximately a quarter of the entire sample studied LREs. Learner repair, feedback types, and the presence of negotiation for meaning have also been studied with some regularity. A variety of other interactional features were observed but in less than 5% of the sample. These include humor (Collentine, Reference Collentine2009), communication strategies (Khamis, Reference Khamis2010), and turn taking (Butler & Zeng, Reference Butler and Zeng2014).
In addition to the features of L2 production, we were also interested in the tools and techniques employed to analyze them. This phase of the study involved coding primary studies, first, for whether learner production was analyzed by hand or using automatic, computer-based tools such as those developed by corpus linguists. We also recorded the statistical analyses applied by researchers in this area. These results appear in Table 7 and Figure 5, respectively. Nearly all coding of target features in learner production was carried out manually. Only six studies applied automatic, corpus-based computer techniques whether alone or in coordination with hand coding.
Techniques for Coding Learner Production
Statistical Analyses in Studies of Task-Based Production (%)
There are relatively few unique statistics observed in this domain. As in other areas of applied linguistics, most studies rely on tests of mean differences between or among groups. Not only are the highest percentages found for ANOVA and t test, but also all post hoc tests found in the sample were associated with ANOVAs, and all nonparametric tests were conceptually (if not statistically) addressing differences between or within groups. The following analyses were found in a single study within the sample: ANCOVA, factor analysis, and Rasch analysis.
In addition to understanding the variety of unique analyses being carried out in our sample, we were also interested in how they are being used in combination as well as the total number of statistical tests applied per study. The latter issue is a concern due to the small samples typical in the field and their debilitating effect on statistical power. Whereas a small number of studies (k = 15, 18%) presented only descriptive statistics and ran no statistical tests, twice as many (k = 36, 42%) conducted a single type of statistical analysis (e.g., one or more t tests), and just about as many (k = 34, 40%) conducted multiple unique analyses (e.g., one or more t tests and correlations). Of the studies that conducted one or more statistical tests, the number of tests of statistical significance per study ranged from 2 to 109 (median = 12.5; interquartile range = 27.25).
Transparency
Eight features associated with researcher transparency were coded for, the results of which are presented in Figure 6. Research questions are consistently found in primary reports of research in this domain. Other practices are found only sporadically. Interrater reliability, for example, which is necessary given the judgment required to code for features of learner production (e.g., errors, T-units, LREs), was found in just over half of the sample. A somewhat smaller portion of the studies in our sample provided the tasks used in their studies. One quarter of the sample also failed to report a statistic—usually one or more standard deviations—required of the type of data being presented.
Features Associated with Researcher Transparency (%)*Calculated by dividing the number of studies that reported to have checked the assumption of their statistical tests (21) by the number of studies that conducted one or more statistical tests (70)
Reliability
One finding in the previous section pertains to the presence of reliability estimates in studies of task-based production. The results shown in Table 8 are concerned with the actual (aggregated) reliability as observed in the sample. Reliability estimates in this domain are generally quite high and consistent.
Descriptive Statistics for Meta-Analysis of Reliability Coefficients
DISCUSSION
This study applied a synthetic approach to examine research on task-based L2 learner production. Our interests were both substantive and methodological. We were concerned not only with what constructs and features had been studied but also with the methods that had been used to study them. It is somewhat artificial to consider these issues in isolation as “study results are determined conjointly by the nature of the substantive phenomenon under investigation and the nature of the methods used to study it” (Lipsey, Reference Lipsey, Cooper, Hedges and Valentine2009, p. 150). In our discussion, we therefore consider the relationships between these two types of findings when and as appropriate. We also use our results to make empirically grounded suggestions for building on and improving primary research in this domain of applied linguistics.
The current study revealed that interactional features are second only to grammar in the frequency with which they are examined. This pattern may be attributed to two reasons. First, many interactionist SLA researchers who use tasks to elicit participant interaction have shown that task features can also affect the presence and amount of different types of interactional features (Kim, Reference Kim2009; Révész, Reference Révész2011). Second, as the educational value of learner-learner interaction has been studied extensively in recent years, researchers have sought to understand how learning opportunities during interaction might be facilitated (Philp et al., Reference Philp, Adams and Iwashita2014).
We also analyzed the range of interactional features examined in the sample, the most frequent of which was LREs. Studies in this domain have investigated several other moves as well including feedback types (e.g., recasts), modified output, and uptake. These results as well as those discussed in the previous paragraph provide evidence of the strong theoretical and methodological connection between research on L2 interaction and task-based teaching and learning.
In terms of linguistic features that were analyzed in recent task-based studies, our results show a heavy focus on learners’ grammar and vocabulary production (69% and 53% of the sample, respectively). Moreover, it was common to observe studies interested in LREs, the most frequently analyzed interactional features, subdividing them further as grammatical or lexical (Kim, Reference Kim2009; Swain & Lapkin, Reference Swain and Lapkin1998). This finding is not particularly surprising given the frequent interest in encouraging learners’ attention to linguistic forms during meaning-oriented tasks. This result also mirrors classroom-based research indicating the frequency of teacher feedback in response to learners’ grammar and vocabulary errors (Brown, in press).
Although there are certainly many compelling reasons for the focus on learner-produced grammar and vocabulary, many of the theoretical and practical rationales for studying these targets would also apply to pronunciation and pragmatics. Outside of the realm of task-based teaching and learning, empirical efforts on L2 pronunciation and pragmatics are thriving (e.g., Culpeper, Taguchi, & Mackey, in preparation; Lee, Jang, & Plonsky, Reference Lee, Jang and Plonsky2015). We would encourage task-oriented researchers to collaborate with colleagues who work in L2 pronunciation and/or pragmatics, and vice versa, to add to this body of research and to build on the momentum in these areas (e.g., Gurzynski-Weiss, Long, & Solon, in preparation; Taguchi & Kim, under contract).
The CAF framework has received substantial theoretical and empirical attention in applied linguistics (Housen & Kuiken, Reference Housen and Kuiken2009a; Housen et al., Reference Housen, Kuiken and Vedder2012; Norris & Ortega, Reference Norris and Ortega2009; Pallotti, Reference Pallotti2009). As attested by the results of the current study, task-based researchers have certainly benefited from and contributed to such discussions. A substantial portion of the studies in our sample examined one or more of the CAF features. A number of these studies stem from early task-based research that predominantly adopted monologic tasks (e.g., picture description). For instance, early studies of task complexity testing the cognition hypothesis or trade-off hypothesis (see Jackson & Suethanapornkul, Reference Jackson and Suethanapornkul2013) and task planning time studies (Ellis, Reference Ellis2005, Reference Ellis2009) often focused their analyses on CAF. These and other efforts have led to significant developments in terms of both methods and theoretical underpinnings of various submeasures of each area (e.g., clausal complexity, phrasal complexity). However, in coding our sample of studies we observed a great deal of within- and between-study variety in terms of how these constructs are defined and measured.
The heterogeneity that results from the number and variety of unique operationalizations of CAF yields problems at two levels of analysis. First, a lack of consistency across studies impedes efforts such as the present study and Sasayama et al. (forthcoming) to synthesize findings and draw stable conclusions (see related comments in Foster & Wigglesworth, this volume). Theoretical plurality is certainly valuable, but there is a need for greater definitional and operational clarity of CAF (Norris & Ortega, Reference Norris and Ortega2009; Pallotti, Reference Pallotti2009). Based on a synthesis of accuracy measures and their associated weaknesses, Foster and Wigglesworth (this volume) present and illustrate a well-reasoned, alternative index. Their arguments are both substantively and methodological informed. We cannot predict whether their index will be adopted, but we applaud their effort toward consolidation. Similarly, Bulté and Housen's (Reference Bulté, Housen, Housen, Kuiken and Vedder2012) review found myriad definitions, operationalizations, and measures associated with linguistic complexity. The different types of CAF measures used in task-based studies need to be theoretically motivated and used in a way that allows for comparability across studies.
The second problem resulting from a lack of operational consistency is more statistical in nature. It is not uncommon to find studies of task-based teaching and learning with multiple independent and dependent variables. Furthermore, many designs include multiple measures of a single dependent variable such as linguistic complexity. Task-based researchers have occasionally applied multivariate techniques such as MANOVA in such situations (e.g., Ishikawa, Reference Ishikawa2008). As shown in our results, however, more often than not, numerous univariate and bivariate statistical tests are conducted. This practice, which is particularly problematic in light of the small samples typical in this body of research and throughout applied linguistics, is irresponsible and poses a serious threat to internal validity at the primary and secondary levels. A combination of (a) theoretical and operational consolidation and (b) a reduction in statistical testing would likely lead to marked improvements in our understanding of task-based learner language. In some situations, given the inherently multivariate nature of language learning and use, multivariate statistics can and should be utilized (Brown, Reference Brown and Plonsky2015; Plonsky & Oswald, under review).
Designs, Analyses, and Reporting Practices
Many of our results for research and reporting practices in this domain reflect those observed elsewhere in recent years in quantitative applied linguistics (e.g., Plonsky, Reference Plonsky2013). Some of these findings are associated with higher or lower quality. Others, however, simply provide a better understanding of this body of research.
With respect to designs, our results indicate that the majority of the sample was interested in describing linguistic and interactional features as a function of task and/or situational variables; only 33% of our sample tested the effects of a task-based treatment on subsequent learning. This might be due to the nature of traditional task-based studies, which compared learners’ task performance (e.g., CAF) between tasks with different design and implementation conditions. Over time the link between task-based research and SLA has grown, and more recent studies have adopted a pretest–posttest design to shed light on task-based language learning.
There are several findings related to sampling, another aspect of study design, worth highlighting as well. Samples in task-based research are limited both in size and diversity. Almost everything we know about tasks and learner language is based on highly educated young adult users of English. The main cause for such a restricted range of sample demographics is clear and well known: convenience. But the consequence of systematic oversight is potentially severe: massive limitations to both theoretical advancement and to the potential of our research to inform practice.
The analyses in this domain were also found to be less than optimal, though not in any immediate sense. The main issue is one that has surfaced several times in recent years in broader methodological and statistical discussions: an overreliance on null hypothesis significance testing (Norris, Reference Norris2015; Plonsky, Reference Plonsky and Plonsky2015b). Statistical tests in research on task-based learner production are simply overused and underpowered (i.e., lacking in the ability to detect true statistical relationships). Although we did not include this feature in our data collection instrument, with two notable exceptions (Alwi, Adams, & Newton, Reference Alwi, Adams and Newton2012; Baralt, Reference Baralt2013), we observed little to no evidence of concern for power. A related issue, noted by Pallotti (Reference Pallotti2009), is the practice of equating (statistically significant) mean differences with importance and, conversely, a lack of such differences with unimportance. Assuming principled design and data collection, the outcome of a theoretically motivated comparison of mean scores should be of interest regardless of whether or not it yields a statistically significant or large difference. In fact, when a difference is predicted by theory, some might argue a lack of evidence for that difference to be even more worthy of attention (see Godfroid & Spino, Reference Godfroid and Spino2015; Plonsky, Reference Plonsky2015a).
Looking ahead, we recommend a move toward larger samples and fewer contrasts. This approach will lead to a smaller but more stable set of results, which can then be aggregated at the synthetic/meta-analytic level. More generally, we would also suggest that researchers divest their attention away from statistical significance and toward a closer consideration and interpretation of descriptive statistics. Standard deviations are a prime example. At the primary study level, these values provide critical information regarding the extent to which participants perform uniformly (or not) on a given task or in response to a given treatment (see Pallotti, Reference Pallotti2009). At the meta-analytic level, they are used to calculate effect sizes. Nevertheless, standard deviations are almost never interpreted and they are often left out of primary reports.
Our results also show that automatized (corpus-based) analyses of learner language are almost never employed. Corpus techniques are not appropriate in all situations, and they can certainly present challenges. For example, corpus tools can be less accurate when used to analyze productive data that contain errors or other irregularities. Another challenge is that not all applied linguists have programming skills. We have found in our experience, however, that corpus researchers are often willing to engage with research problems outside their typical areas of study. We have also found that programs needed to assist with text analysis can often be developed in a reasonable amount of time, particularly in the case of lexical targets. In addition to greater efficiency and the obvious logistical benefits, there are at least two main advantages of corpus techniques over traditional, hand-coded analyses. First, corpus analyses can vastly improve replicability. The culture of openness and transparency currently developing in applied linguistics through IRIS and other initiatives has long since been present among corpus linguists, and both corpora and corpus tools are often shared freely (see, e.g., Kristopher Kyle and Scott Crossley's tools for assessing lexical sophistication and syntactic complexity: http://www.kristopherkyle.com/). This approach would not only allow for others to reexamine an original set of linguistic analyses, but also to replicate the analyses and generalize the results to other learners, tasks, and so forth (Kyle & Crossley, Reference Kyle and Crossley2015). Another potential advantage of corpus analyses for task-based learner production is the reliability inherent in the technique. Despite the generally high overall (meta-analytic) reliability reported above (M = .94, SD = .05), human rating is always susceptible to error. By contrast, computer programs always follow the rules written into them.
Related to the notion of transparency, the present study also coded for a number of reporting practices. Research questions were found in almost every study in the sample. Most studies (59%) reported an estimate of (interrater) reliability, which is higher than in most domains of L2 research (see Plonsky & Derrick, in press), but still less than ideal. Also concerning was the lack of interpretation surrounding reliability coefficients. As with effect sizes (42%), authors appear to treat the reporting of reliability estimates as a perfunctory act. They are almost never considered beyond an over simplified dichotomy of acceptable/unacceptable. Visual displays of data are found regularly in this domain (38% of the sample). However, the types of graphic techniques employed were rarely effective (see Hudson, Reference Hudson and Plonsky2015; Larson-Hall & Plonsky, Reference Larson-Hall and Plonsky2015). For example, most were simple bar graphs with no indication of dispersion. Finally, we note that 45% of the sample made their tasks available to readers and other researchers. This figure could and should be higher, and we would like to see more researchers upload their materials and instruments to the IRIS database, but we view this result as generally positive and indicative of a degree of transparency on the part of the authors in our sample.
Recommendations for Future Research
Our focus throughout much of this synthesis has been retrospective. However, we are not only interested in describing and evaluating previous research in this domain. We are quite concerned in fact about future analyses of task-based production. Therefore, based on the findings of the current study, we present in this section a set of recommendations for areas in need of attention in future efforts. Many of these comments echo our colleagues’ and collaborators’ suggestions made as part of broader methodological discussions taking place in the field (see Norris, Plonsky, Ross, & Schoonen, Reference Norris, Plonsky, Ross and Schoonen2015). For the sake of clarity and economy, our recommendations appear in bulleted form.
-
1. Substantive areas in need of further attention
-
• Production targets and features: pronunciation, pragmatics, L1 use, and quality of task performance
-
• Nonlinguistic and interactional features: modified output, uptake, and feedback types
-
• The role of learner background (e.g., individual differences, L1) as a moderator of production
-
• Face validity of task types
-
-
2. Designs
-
• Larger samples
-
• Greater variety of learner demographics (L1s, TL, ages, and proficiency levels)
-
• Aptitude-treatment interaction (ATI) studies, to investigate the role of individual differences as a moderator of the learning that results from different task conditions
-
-
3. Analyses
-
• Greater use of corpus techniques for analyzing learner production
-
• Fewer tests of statistical significance
-
• Greater use and interpretation of descriptive statistics, including effect sizes and confidence intervals
-
• Use of multivariate statistics when appropriate
-
-
4. Data reporting and transparency
-
• More thorough reporting of descriptive statistics including standard deviations, effect sizes, and confidence intervals
-
• Greater reporting and interpretations of reliability estimates
-
• Greater task and instrument availability (e.g., via IRIS)
-
• Improved techniques for visual displays of data
-
CONCLUSION
This study, a review of research on task-based learner production, has found empirical efforts in this area to be both abundant and diverse in their interests. Pursuing a wide range of linguistic and interactional features of learner language is certainly beneficial for advancing our theoretical understanding of tasks and of L2 development. However, the combination of a wide range of constructs and of definitions and measures of those constructs has led to a necessarily uneven body of results. In response to this issue and based on the results of our study, we suggest a number of substantive features that merit additional attention. We also join others in recommending an effort toward enhanced theoretical integration and consistency. Complementary to the findings concerning the linguistic and interactional targets, we have also examined a range of methodological practices in this domain. We are concerned about several of our findings including sampling practices, data analyses, and reporting practices. However, we are also encouraged by the productivity and persistence of scholars in this area who will no doubt continue their efforts to advance our understanding of the role of tasks in language learning and teaching.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit http://dx.doi.org/10.1017/S0267190516000015.
