1. Introduction
Between 1998 and 2004 the UK's education department injected £1.8 billion into schools to boost the use of computers and ICT (Ofsted 2002). Singapore launched its ‘Masterplan for IT in Education’ in 1997; by 2002 all schools were equipped with computers and internet access, and second and third Masterplans have been implemented (Ministry of Education, Singapore 2008). In the USA, following the Telecommunications Act 1996, between 1998 and 2001, over $1.7 billion per year was invested in ICT in schools (Goolsbee & Guryan Reference Goolsbee and Guryan2006).
The Spanish government launched ‘Escuela 2.0’ (Presidencia del Gobierno 2009), aimed at providing over 14,000 classrooms in both the primary and secondary sector with interactive whiteboards (IWBs), wireless internet, and student and teacher PCs. In Italy, the ‘Piano e-Government 2012’ (Governo Italiano 2009) aims to provide every school with broadband and at least three ICT suites equipped with IWBs and PCs. China aimed to connect 90% of primary and secondary schools to the internet by 2010 (UNESCO 2009).
It is likely that language learning classrooms will be part of this move to increased technology use. It is therefore crucial to review research on digital technology in language teaching, or Computer-Assisted Language Learning (CALL), including mobile technologies (Levy & Hubbard 2006), in primary and secondary sectors, in order to inform teachers, policy makers and materials designers. Previous CALL reviews have generally been of two kinds. Some have identified various historical phases in CALL pedagogy (Warschauer Reference Warschauer and Fotos1996; Delcloque Reference Delcloque2000; Chapelle Reference Chapelle2001; Bax Reference Bax2003; Jung Reference Jung2005). Some of these authors feel that CALL has been technology-driven rather than serving pedagogical needs (Delcloque Reference Delcloque2000). From a pedagogical perspective, some argue that CALL has never been communicative in the sense intended by most theorists (Bax Reference Bax2003), and others that there is an unfortunate top-down tendency, whereby CALL in higher education percolates down to younger learners (Jung Reference Jung2005), without reflection on their different needs. Other reviews of CALL have provided summaries and syntheses of research using traditional (narrative) methods (Levy Reference Levy2009), or attempts at more comprehensive methods with less than exhaustive search strategies (Beatty Reference Beatty2010)Footnote 1 or restricted to CALL-related journals (Stockwell Reference Stockwell2007)Footnote 2 and without differentiating educational sectors; or have not restricted themselves to empirical studies, but have included, for example, straightforward descriptions of software. In both these types of reviews Pederson noted, as early as 1987, the prevalence of CALL versus non-CALL comparisons and their limitations, notably their limited ability to recommend future design features for CALL software, the difficulties of replicating them, and their limited ecological validity.
While such comparative research has continued to proliferate, other approaches have emerged. In response to her criticisms, Pederson advocated more narrowly targeted studies focusing on differential effects of attributes of technologies, controlling for other features, and ensuring an investigation not only of the medium itself, but also of the way it is manipulated in the learning context. Levy (Reference Levy1997) agreed, but emphasized that isolating a key variable should not sacrifice ecological validity. Similarly, Chapelle (Reference Chapelle2001) proposed that any product-oriented investigation be complemented by process-oriented observation, a proposal supported by Allum (Reference Allum2002). Dunkel (Reference Dunkel and Dunkel1991) also advocated the use of ethnographic methods to investigate the impact of social and attitudinal variables on CALL effectiveness (see also Chapelle & Jamieson Reference Chapelle and Jamieson1986). In other words, the question is not only ‘Is the technology effective?’ but also ‘Why is it effective, and how does this contribute to a theory of language learning or of education more generally?’. Stockwell (Reference Stockwell2007: 115), reviewing studies from 2001 to 2005, concluded that ‘there still remains an element of failure to stipulate why a given technology was used in achieving learning objectives’, and noted a need for research investigating how technologies are used. Lastly, many reviewers (e.g. Felix Reference Felix2005; Hubbard Reference Hubbard2005) comment on the poor quality of research and reporting in CALL.
We have attempted to take account of these issues in the review questions and methodology of the current review.
2. General aim of the review
This systematic review aims to achieve the broadest possible coverage of empirical research evidence, published in English, for the use and effectiveness of CALL in primary and secondary second language (L2) education. To achieve this we followed the EPPI reviewing system (EPPI-Centre March 2007) which proposes, first, a keyword map (overview) to categorize studies, and then an in-depth review of a subset of studies selected according to specified criteria.
It is important to note five features that differentiate systematic reviewing from more traditional narrative reviewing. A systematic review:
1. is transparent in its procedures, from conception to final conclusions; an initial protocol specifies how the review is to be conducted;
2. includes studies based on exhaustive and reliable searching;
3. aims to reduce reviewer bias as much as possible;
4. attempts to produce syntheses containing clear messages about the reliability of the evidence reviewed;
5. attempts to ensure that the research is relevant and accessible to end-users.
3. Review questions
To avoid overlooking technologies investigated with other languages, and older technologies still being introduced in developing countries or continuing to be used in more developed countries, the first phase of the review maps all empirical research published between 1991 and 2010 on CALL in primary and secondary L2 education, including its use in Modern Foreign Language (MFL), English as a Foreign Language (EFL) and English as a Second Language (ESL) classrooms. The overarching question for the keyword map was therefore:
What empirical research was undertaken on the use of CALL in language learning and teaching in primary and secondary schools between 1991 and 2010?
We have justified our focus on primary and secondary language education. We justify limiting our search to post-1990 on grounds of technological advancement – what Jung (Reference Jung2005: 10) calls ‘the online/offline divide’.
The second phase of the review, the in-depth review, focuses in on EFL. It asks:
1. What technologies are reported as being used in L2 English teaching in the first decade of the 21st century, and why?
2. What evidence is there that technology facilitates the acquisition of linguistic knowledge and/or the development of language skills?
3. What insights, other than linguistic, are revealed by a review of research reports?
We justify narrowing our focus to post-2000 research on the basis of advancements in technology and its use in educational contexts; we justify focusing exclusively on target language English because of its role as the L2 most studied globally in primary and secondary contexts. It is, however, anticipated that the findings may apply to other MFL contexts.
4. Method
Once the general aims of the review and review questions were confirmed, we established a review protocol comprising the search strategy, the inclusion/exclusion criteria, the mapping procedure, the in-depth review procedure, and the synthesizing procedure.
4.1 Search strategy
Databases used for electronic searching were: for education, Education Resources Information Center; for linguistics, Language and Linguistics Behavior Abstracts; for psychology, PsycINFO; for computer science, INSPEC.
Different search strategies were tested, including free-text search and thesaurus. To ensure reliability and for optimum precision and recall, the search strategies were refined by inspecting the abstracts of retrieved articles and adding/deleting search terms (and their variants, synonyms, hyponyms and other related terms). To increase reliability, selective manual searching was additionally carried out on CALL Journal, ReCALL, Language Learning & Technology, and CALICO Journal.
4.2 Inclusion/exclusion criteria
On completing the search, we identified from abstracts (and sometimes from full articles) the studies to be included in the keyword map, according to these criteria:
1. They reported on the use of CALL (broadly conceived) . . .
2. . . . in foreign or second language learning. . ..
3. . . . with school age (primary and secondary) learners.
4. They focused on learners.
5. They reported on empirical studies carried out by the author(s) . . .
6. . . . which focused on the effects of interventions on the acquisition of linguistic knowledge and skills.
7. The studies were reported between 1991 and 2010.
8. They were published in peer-reviewed journals or in doctoral dissertations . . .
9. . . . in English.
Studies were excluded if they focused on:
1. corpus technology, e.g. concordancers
2. testing
3. teaching Classics (Latin and Ancient Greek)
4. teachers rather than learners.
This yielded 117 studies for inclusion in the systematic keyword map. For the in-depth review we applied additional criteria. Studies were only included:
1. if they reported on the effect of the use of technology . . .
2. . . . on specific language skills . . .
3. . . . of learners of EFL or ESL. . .
4. . . . and were published between 2001 and 2010.
The application of these additional criteria yielded a total of 47 studies for the in-depth review.
4.3 In-depth review: Characterizing studies
At least two reviewers independently read each paper in depth, using a question grid. They then met to compare answers and resolve any differences. It is particularly important in a systematic review to resolve differences in the quality assurance evaluations (High, Medium, or Low) regarding each study's contribution to the in-depth review questions:
1. Relevance of the focus of the study;
2. Appropriateness of the study's research design for addressing the review questions;
3. Trustworthiness of the study's overall methodology;
4. Contribution of the study (as a result of 1–3 above) to the review questions.
5. Review findings 1: The overview or keyword map (Appendix A)
Keywording and mapping the 117 studies revealed the following:
1. There was a significant increase over the period 1991 to 2010 in the number of studies conducted (Table 1).
2. The most frequently studied technologies were multimedia (22%), computer-mediated communication (CMC) technologies (22%) and the internet (15%). Among CMC technologies, e-mail was the most popular (12%; Table 4, page 11).
3. There was some interest in the use of text reconstruction, an early CALL technology, and in early general purpose technologies such as multimedia. Interest in the use of the internet increased; there are no studies focusing on the internet before 1995. Similarly, there are no studies of chat or discussion forums before 2000, and no studies of web 2.0 technologies before 2005 (Table 4).
4. More studies were conducted with secondary school students (60%) than with primary school students (40%).
5. English was a target language in 77% of the studies. European languages other than English were the next most popular target language (24%)Footnote 3.
6. The largest number of studies focused on vocabulary (24%) and writing (24%). Next came reading (22%). Half as many studies focused on speaking (11%) and listening (10%). Then came grammar (7%) and, finally, pronunciation (3%; Table 1).
7. Technology for vocabulary teaching was investigated equally with primary and secondary students. Studies of reading more frequently involved primary students than secondary students. Studies of grammar and writing more frequently involved secondary students than primary students (Table 1).
8. Word processors, the internet, CMC and Web 2.0 were more frequently studied with secondary than primary students. Multimedia was investigated more frequently with primary than secondary students.
9. Multimedia studies are particularly associated with vocabulary studies. Word processors are associated with writing studies, as are CMC and Web 2.0 technologies.
10. Regarding research methodology, the largest number of studies employed mixed methods (44%), and slightly more employed qualitative (31%) than quantitative methods (23%).Footnote 4
11. A wide variety of sample sizes were used, from fewer than ten participants to over 200. The most popular sample size was 21–30 participants (14%).
12. Most studies did not indicate whether the participants were male or female (52%). In the majority of the remaining studies the sample was described as ‘mixed’ (39%).
Studies by linguistic knowledge and skills, study date and education sector (P = Primary, S = Secondary; n = 117; studies not mutually exclusive by linguistic knowledge and skills)
Note: In two studies the sector of education was not indicated.
These data give rise to the following observations:
1. The fact that 71 of the studies included in the map did not indicate the sex/gender of participants was interesting, given the evidence that boys and girls are attracted to different aspects of ICT and appreciate different educational applications of technology, boys being more interested in technical aspects and problem-solving activities, and girls being more interested in creative aspects and social activities (Volman, van Eck, Heemskerk & Kuiper 2005).
2. Especially interesting is the finding that, although more studies were implemented in the secondary than the primary sector, the difference was not as large as might be expected, given earlier observations of a top-down effect from higher education (Jung Reference Jung2005).
3. The comparative lack of interest in grammar differed from Stockwell's (2007) review, a difference probably resulting from his inclusion of research in higher education.
4. The differences in the match between language skills and learner age might be attributed to: (1) vocabulary and receptive language skills being more privileged in primary education; (2) primary learners being typically less able to type well in order to use word processing facilities.
Note, however, that as well as linguistic knowledge and skills, these studies reported on a whole range of other outcomes: intercultural competence, narrative comprehension, stance -taking, phonological awareness, critical thinking, inter-subjectivity, deep vs. surface learning and knowledge construction.
Perhaps the greatest difficulty we encountered in keywording the map was identifying from abstracts and introductions precisely what technology was being used; we often had to consult the whole paper for this information.
Having provided an overview of CALL in primary and secondary language education between 1991 and 2010, we now turn to the in-depth review.
6. Review findings 2: The in-depth review
From 2001 to 2010 we identified 70 studies relating to EFL or ESL teaching in primary and secondary schools. Of these, 23 were not specifically concerned with the acquisition of linguistic knowledge or skills and were therefore eliminated from the in-depth review, leaving 47 studies (Table 2).
In-depth review studies by country and year (n = 47, not mutually exclusive by country)
In the primary sector, 19 studies were conducted, and in the secondary sector, 28. Again, few studies indicated the sex/gender of participants (n = 20). In the majority (n = 22) of the remaining studies, gender was indicated as mixed and not used as a variable.
Most of the 47 studies investigated vocabulary (n = 15), reading (n = 17) or writing (n = 17). There was much less interest in grammar (n = 3), speaking (n = 9), listening (n = 8) and pronunciation (n = 3). Importantly, Table 3 shows that CMC (and chat, in particular) features much more in the secondary than in the primary sector.
In-depth review studies by technology and linguistic knowledge and skills (n = 47, not mutually exclusive by technology or linguistic knowledge and skills)
Note: NLP = Natural Language Processing.
6.1 Review question 1: What technologies are used for English language learning, and why?
The reviewers attempted to determine from each research report when the study was carried out, why it was carried out then, and why it was carried out with that particular population. Unless CALL studies are contextualized in this way, it is difficult to discern a relationship between the use of technology and the pedagogical need for it. There might also be a considerable time-lapse between data collection and publication, with obvious implications for conclusions.
Taking writing research as an example, of the 17 studies reviewed, only nine provided the date of data collection, and reasons for the research are often vague. In Zhang et al. (Reference ‡Zhang, Gao, Ring and Zhang2007), the authors assert a dearth of research regarding online discussion in China at an unspecified time and state that, in high schools, technology is in place but is not used, primarily because instruction is textbook- and test-driven. Somewhat more informative are Lund & Rasmussen (Reference ‡Lund and Rasmussen2008), who report that the Norwegian government introduced knowledge promotion reform in 2006/2007 and that this, combined with an increase in digital and networked tools, led to their study of wikis for improving writing. As motivation for their study, Lin & Wu (Reference ‡Lin and Wu2010) mention the falling cost of laptops, one-computer-per-child initiatives, and the inception of netbooks, and Suh, Kim & Kim (2010) note the increasing popularity of gaming and potential educational benefits, including engagement, motivation, critical thinking, problem solving and experiential learning. Other reasons for carrying out the research were often specific to a particular context.
This pattern is repeated for the other language skills: the impetus for a study, although pedagogic, is often closely linked to very local concerns. For example, Coniam & Wong (Reference ‡Coniam and Wong2004) imply that they investigated the use of internet relay chat with grade 3 to 10 Hong Kong learners because of these learners’ errors in finite verb use in main clauses. Johns, Hsingchin & Lixun (2008) imply that they adopted, with Taiwanese secondary students, a technology based on gap-fill, putting sentences in the right order, and concordance-based exercises, because the students normally receive limited EFL instruction aimed almost entirely at passing university entrance exams. These statements reveal that the impetus driving research is often policy rather than education theory. This is not to imply that policy-driven research should be shunned, but one might have expected a stronger link to theories of L2 acquisition and learning.
6.2 Review question 2: What are the benefits of CALL?
We now review the evidence for whether technology facilitates the acquisition of linguistic knowledge and the development of language skills. We discuss in detail only studies rated as making a medium or high contribution to this review question (Tables 5–11).
Studies by technology and study date (N = 117, not mutually exclusive by technology)
Weight of evidence for studies focusing on vocabulary
Weight of evidence for studies focusing on grammar
Weight of evidence for studies focusing on pronunciation
Weight of evidence for studies focusing on reading
Weight of evidence for studies focusing on writing
Weight of evidence for studies focusing on speaking
Weight of evidence for studies focusing on listening
6.2.1 Evidence from vocabulary studies
One problem encountered was the considerable variety in outcome measures. Whereas one might have expected these to be (as in non-CALL vocabulary acquisition literature) vocabulary breadth or depth, receptive or productive vocabulary growth, there was instead an interest in ‘certainty’ of vocabulary knowledge (Kim & Gilman Reference ‡Kim and Gilman2008), the production of correct ‘index cards’ (Nakata Reference ‡Nakata2008), general vocabulary gains with no indication of whether this was receptive or productive (e.g. Kim & Gilman Reference ‡Kim and Gilman2008; Chen & Li Reference ‡Chen and Li2010), ‘content’ knowledge (Silverman & Hines Reference ‡Silverman and Hines2009), or knowledge of function words (Tsou, Wang & Li 2002).
Four studies investigated the effects of multimedia presentation on vocabulary acquisition. All four present positive findings. The first two (Tsou et al. Reference ‡Tsou, Wang and Li2002; O'Hara & Pritchard Reference ‡O'Hara and Pritchard2008) are broad media comparisons, where a software program featuring a range of media is compared with traditional classroom instruction. The third study (Silverman & Hines Reference ‡Silverman and Hines2009) focused on just one medium, video, comparing traditional instruction plus video with traditional instruction alone. The fourth (Kim & Gilman Reference ‡Kim and Gilman2008) evaluated differential effects of a number of combinations of media (text; text and speech; text and graphics; text, speech and graphics; reduced text and speech; reduced text, speech and graphics) for CALL vocabulary presentation.
Tsou et al. (Reference ‡Tsou, Wang and Li2002) report that students receiving multimedia instruction improved their knowledge of words (mostly function words) more than those receiving traditional classroom instruction. Similarly, O'Hara & Pritchard (Reference ‡O'Hara and Pritchard2008) compared the effects of completing a hypermedia project in PowerPoint with a pen-and-paper project on students’ acquisition of topic-related vocabulary; students in the hypermedia projects scored higher on post-tests of depth of vocabulary knowledge. Likewise, Silverman & Hines (Reference ‡Silverman and Hines2009) found that both English language learners and native speakers presented with video clips illustrating vocabulary items showed greater improvements in general vocabulary knowledge than those not receiving multimedia instruction, and that use of the multimedia system narrowed the gap between L1 and L2 users. Finally, Kim & Gilman (Reference ‡Kim and Gilman2008) found that the use of graphics, but not the use of a combination of media, supported vocabulary acquisition.
A different approach was adopted by two studies (Nakata Reference ‡Nakata2008; Lu Reference ‡Lu2008): different modes of sequencing and spacing vocabulary presentation were explored. Nakata used a computer to gradually increase spacing between repetitions of vocabulary items and compared outcomes to fixed list and flash card presentation; the only difference found was an advantage for computer-controlled spacing and flash cards over lists. Lu (Reference ‡Lu2008) investigated the use of text-messaging for the presentation of bite-sized vocabulary ‘lessons’ spaced throughout the day, during learners’ commuting time. Students who received these lessons learnt more vocabulary than those receiving pen-and-paper instruction, but the advantage was not maintained at delayed post-test.
Similarly, Chen & Li (Reference ‡Chen and Li2010) investigated the effects of mobile learning on vocabulary acquisition, focusing on a personalized context-aware ubiquitous (i.e. mobile) learning system adapting the content of vocabulary lessons delivered on a Personal Digital Assistant (PDA) to the learner's proficiency, location and the time of day. Learners working with the system performed better than a control group working with a non-adaptive system on a vocabulary knowledge post-test.
With yet another approach, Proctor, Dalton & Grisham (2007) investigated a reading tutor, the Universal Literacy Environment (ULE), which provides students with scaffolded reading instruction and strategy support through a bilingual avatar. This pre-experiment (i.e. no control group), comparing the effectiveness of the tutor for Spanish learners of English and native speakers of English, found no significant gains from pre-test to post-test for either group, a result possibly explained by the short duration of the intervention, and perhaps by the small sample size (n = 14 L1, 16 L2).
Li (Reference ‡Li2010) investigated the impact of using mono- and bilingual electronic dictionaries equipped with text-to-speech synthesis pronunciation models on the acquisition of receptive vocabulary knowledge by high-school Chinese students in Canada, through extensive reading activities presented in ‘e-Lective’ (Cummins 1998; Dyned Reference Dyned2011), a digital environment for supporting academic language learning. The study compared the use of electronic dictionaries embedded in e-Lective reading activities with the use of paper dictionaries and handheld electronic dictionaries incorporating these activities. It found that students acquired more vocabulary with electronic dictionaries than with paper or handheld dictionaries. The finding is possibly explained by the fact that students in the control condition, and the beginners in particular, looked up fewer words than the students in the experimental condition and a possible confound is that some controls also apparently used electronic dictionaries.
The effects of participating in online discussion forums, with and without teacher support, were investigated by Zhang et al. (Reference ‡Zhang, Gao, Ring and Zhang2007). No significant differences in vocabulary acquisition were found at post-test between the experimental groups and a control group not participating in the extra online activities, despite a reasonably long (five-week) intervention. However, there does not appear to have been any pre-testing, casting some doubt on the validity of the analysis. Sasaki & Takeuchi (Reference ‡Sasaki and Takeuchi2010) investigated the effects on vocabulary acquisition of e-mail interactions with a native speaker. The native speaker expressly seeded target words into the e-mails sent to the learners to see whether learners would imitate the vocabulary used and whether imitation would lead to acquisition. Results showed that learners imitated 60 of the 193 target words seeded, 36 of which were acquired. However, according to the learners, 16 of these 60 words were learnt outside the email exchange. Further, the learners acquired without imitation 39 of the words provided by the native speaker, 22 of these having also appeared in other contexts (e.g. English lessons).
This review of vocabulary studies appears to indicate some possible language-learning benefits in the use of CALL. However, we would argue that the results are inconclusive, with regard to claims that CALL facilitates vocabulary acquisition, because of problems in study design and data analysis methodology: the choice of statistical tests is often open to question, and although some researchers report statistically significant results, they rarely report effect sizes. The picture is confused by the varied applications and approaches in the interventions and the lack of homogeneity of outcome variables. Outcomes are not always relatable to a body of previous research evidence or theorizing; there is little discussion of why a particular technological feature should facilitate a particular aspect of vocabulary acquisition. We return to these methodological issues later.
6.2.2 Evidence from grammar studies
Three studies focused on grammar, and two were judged to make a medium or higher contribution to this review question (Tables 5–11). The outcome variables in these studies were: a reduction in the error rate of finite verbs in main clauses in written compositions (Coniam & Wong Reference ‡Coniam and Wong2004); and a general unspecified notion of ‘English grammar’ (Zhang et al. Reference ‡Zhang, Gao, Ring and Zhang2007), although the latter looked at a range of outcomes.
These studies explored whether grammar would improve following participation in chat (Coniam & Wong Reference ‡Coniam and Wong2004) or discussion forums (Zhang et al. Reference ‡Zhang, Gao, Ring and Zhang2007). Previous (non-technology) SLA research into grammar acquisition has tended to focus on the relative benefits of explicit/deductive versus implicit/inductive teaching and learning. Neither study related its findings to a grammar acquisition theory. In any case, no significant differences were found; thus neither study provided evidence that CALL facilitates grammar acquisition.
6.2.3 Evidence from pronunciation studies
Only one of the three pronunciation studies meeting the inclusion criteria also met the quality criteria. This study, Neri et al. (Reference ‡Neri, Mich, Gerosa and Giuliani2008), reported an evaluation of the automatic speech recognition (ASR)-based Computer-Assisted Pronunciation Training (CAPT) system Parling. Each module of the system focuses on a story. The system reads the story aloud while displaying it on screen. Some words are hyperlinked to their meanings and model pronunciations. Learners can record themselves pronouncing the selected word. Feedback (correct/incorrect) is provided through ASR. The experimental group (n = 13) received four 30-minute sessions of training with Parling, whereas the control group (arguably a group receiving a different treatment) (n = 15) received four 60-minute classroom teacher instruction sessions. The latter were given a paper copy of the story which the teacher read aloud, explaining the meaning of some words, and providing a pronunciation model for repetition. It is unclear whether feedback on mispronunciation was given to this group. The words in both conditions were selected to cover the most frequent phonemes of British English and classified according to difficulty level (easy/known vs. difficult/unknown). Neri et al. found that both groups made equal progress with pronunciation, which they claim is of some significance, given that the experimental group spent less time on the task. They also found that students’ pronunciation of difficult words improved more than their pronunciation of easy words.
It would seem that the use of technology for improving pronunciation is worth exploring further, perhaps through an approximate replication study. Neri et al. (Reference ‡Neri, Mich, Gerosa and Giuliani2008) was on a very small scale, only tested isolated word pronunciation, and did not test long term retention. One hopes that the number of studies in this field will increase as ASR technology improves.
Having reviewed the research literature on linguistic knowledge, we now turn to research studies on the four language skills.
6.2.4 Evidence from reading studies
Of the 17 studies focusing on reading, only seven made a medium or higher contribution to this review. Here, we focus on the three not examined elsewhere in this report.
First, note that very different aspects of reading were investigated. Troia (Reference ‡Troia2004), using ‘FastForWord’ software, described one of his outcomes as ‘basic reading skills’ and used a standardized reading test of English (normed on L1 speakers of English), including a sound blending task, word identification and word attack tests (Woodcock & Johnson Reference Woodcock and Johnson1990), whilst Tsou, Wang & Tzeng (2006), using a multimedia storytelling website, measured story comprehension through students’ writing, thereby introducing an additional variable. Chen, Chen & Sun (2010) used a measure of reading comprehension based on the tags that learners chose to characterize texts in their tag-based collaborative reading learning system, TACO. All these studies involved very different populations, ranging from Taiwanese primary classrooms to migrant workers’ children in the US.
The findings of these studies are inconclusive. Troia found no main effect of treatment on basic reading skills; in fact, the control group outperformed the experimental group on sound blending and rhyming. In the Tsou et al. study, gains on the story comprehension measure were similar for the two groups after ten weeks of instruction. In contrast, Chen et al., specifically investigating the impact of social-tagging on reading comprehension, found that students who worked with social-tagging and discussion forums performed better on a post-test of reading comprehension than those who only worked with discussion forums.
In summary, there is no clear advantage for CALL technologies in L2 reading.
6.2.5 Evidence from writing studies
Of the 17 studies focusing on writing, ten investigated writing alone. The most popular technology in these (all post-2005) was the wiki. CMC technologies, specifically, e-mail, chat and discussion forums, were also present. These studies employed qualitative rather than quantitative methods, perhaps reflecting a view that writing research should focus on processes rather than products.
Only one of these ten studies of writing was judged to make a medium or higher contribution to the review question. Mak & Coniam (Reference ‡Mak and Coniam2008), who measured the effects of a wiki-based activity on Hong Kong secondary students’ writing via a quantitative analysis of the amount, quality and type of the writing produced over a six-week intervention, found improvements in the amount of writing produced, in the length of students’ written texts, and in the discourse features of these texts.
6.2.6 Evidence from speaking studies
Four of the nine studies focusing on speaking were judged to make a substantial contribution to the review.
Troia (Reference ‡Troia2004) was reviewed in the reading section. For speaking, Troia's comparison of use of the ‘FastForWord’ software integrating synthetic speech (waveform manipulation) with normal classroom instruction found no effect for treatment on oral language skills.
Satar & Ozdener (Reference ‡Satar and Ozdener2008) considered the benefits of CMC, text chat, and voice chat on speaking proficiency for 90 secondary school participants in Turkey, randomly assigned to three equal groups: voice chat, text chat, and no intervention. The 40- to 45-minute treatment was in addition to normal classroom instruction. Significant differences were found at post-test between the voice chat and controls and between the text chat and controls on a range of speaking proficiency measures, with the experimental groups clearly benefiting from the intervention, but with no difference between the two experimental groups. Although this is a positive outcome, the question remains whether it was the technologies or the extra time spent on tasks that produced these results.
The final two studies (Liu Reference ‡Liu2009; Liu & Chu Reference ‡Liu and Chu2010) investigated a ubiquitous game-based system for developing English speaking and listening skills (see section 6.2.7). Children who used the system outperformed those in the control group on post-tests of speaking as well as listening.
In sum, the four studies offer no consistent evidence for the idea that technology facilitates speaking skills more than non-technology pedagogy, but there is a suggestion that combining the two can lead to speaking gains.
6.2.7 Evidence from listening studies
Five of the eight listening studies met the inclusion criteria. Of these five, only two focused narrowly on listening (and speaking). The first, Verdugo & Belmonte (Reference ‡Verdugo and Belmonte2007), investigated the use of interactive digital stories (Kindersite 2009) on listening proficiency, operationalized by tests devised by the authors. The subjects were 260 six-year-old Spanish children, drawn from six schools in Madrid, with classes designated as experimental or control. Interactive digital stories were used as part of the experimental group's lessons. After controlling for differences at pre-test, the researchers found that the experimental groups outperformed the controls. The second study (Liu & Chu Reference ‡Liu and Chu2010) investigated a ubiquitous game-based system incorporating a virtual representation of the children's school campus in Taiwan, for developing English listening and speaking skills. Each virtual campus location contained links to audio and video recordings of the types of conversation that the learner might have there. The children first explored the virtual campus from their classroom, practising speaking and listening skills. They then took the system out into the real world campus to complete a treasure hunt game. During the treasure hunt, they could access further audio and video recordings and related activities to practise situation-related speaking and listening. At each location, they were also set simulated conversation activities. Completed activities were rewarded. The ubiquitous game-based system was compared with printed materials and a CD. The experimental group outperformed the control group on post-tests of listening.
The remaining three studies, Greenfield (Reference ‡Greenfield2003), Troia (Reference ‡Troia2004) and Lin & Wu (Reference ‡Lin and Wu2010), investigated the impact on a range of language skills and sub-skills of a collaborative e-mail exchange focusing on the production of a writing anthology, using the ‘FastForWord’ program previously discussed and using netbooks respectively. While Greenfield's (2003) e-mail exchange students increased their confidence in listening, Troia (Reference ‡Troia2004) found no differences between the ‘FastForWord’ group and the group receiving regular classes only, and Lin & Wu (Reference ‡Lin and Wu2010) observed no differences in listening scores between students that used netbooks and those that did not.
For these five listening studies, the evidence for the effectiveness of CALL appears to be conflicting. However, this may simply be because three of the studies did not focus explicitly on listening. The two that did, found an advantage for technology.
6.3 Review question 3: Insights into CALL
We have so far reviewed in-depth studies investigating the use of CALL to facilitate or enhance linguistic knowledge acquisition and skill development. We will now look for insights into the possible non-linguistic benefits of using technology in language learning, categorized as follows: learners’ attitudes to CALL; learning behaviours; confidence and anxiety.
6.3.1 Learners’ attitudes to CALL
Lu (Reference ‡Lu2008) investigated attitudes towards ‘mini-lessons’ sent to Taiwanese grade 10 students’ mobile phones in terms of convenience, interest, facilitation of memorization, ease and frequency of access. More than 50% of the students appreciated the convenience and time management involved; and individual students found creative ways of adapting the pedagogy: for example, the student who constructed sentences from the target words and sent them in for feedback. However, less positive attitudes were also expressed: only 25% of participants reported enjoying the novelty and only 17% reported appreciating receiving the messages.
More consistently positive attitudes were expressed in Nakata's (2008) comparison of different vocabulary learning methods with 15–17-year-old Japanese learners. Of students using computer-based training, 70% expressed overall satisfaction, compared to only 40% and 30% in groups using card and paper systems respectively. The computer-based group also found the training less boring, although the author speculates that novelty value may have played its part. O'Hara & Pritchard (Reference ‡O'Hara and Pritchard2008) also reported positive reactions from younger learners in California (10–11-year-olds) to a hypermedia project using PowerPoint: participants reported better recall, more available information and more control over their learning. Reactions to the context-aware ubiquitous learning system investigated by Liu & Chu (Reference ‡Liu and Chu2010) were also positive. Students enjoyed using the system, found it helpful and appreciated its presentation format and active nature.
Taken together, these studies provide evidence of positive attitudes towards CALL. However, one must exercise caution in interpreting these results. The majority of the studies were less than a term in duration. Indeed, Lu's (2008) study was conducted over only two weeks and Nakata's (2008) study was conducted over a single lesson. Hence the results are likely to have been affected by novelty. Only Liu & Chu (Reference ‡Liu and Chu2010) was longer, conducted over eight weeks.
6.3.2 Learning behaviours
Five studies focusing on learning behaviours met the criteria of medium or high contribution to the review.
With respect to critical thinking, Zhang et al. (Reference ‡Zhang, Gao, Ring and Zhang2007) found that students who received teacher support during online discussions discussed topics in greater depth, as measured by number of indicators of cognitive reasoning, than those who did not receive such support.
Chandrasegaran & Kong (Reference ‡Chandrasegaran and Kong2006) found that students participating in discussion forums demonstrated argumentation skills which they had already mastered in oral interaction, but had had difficulty using in academic expository writing. Stance, an expression of the student's position in relation to the topic of discussion or a response to another student's position, was observed in 137 (87%) of the 159 postings analysed, and stance-support, reasons or grounds for a particular position, was observed in 75 (55%) of postings expressing stance. This was a descriptive study. However, the suggestion from the authors is that the technology did provide better opportunities for stance-taking than if, say, the students had been involved in face-to-face group activities, because of the non-threatening aspect of the technology.
Two studies by Lan, Sung & Chang (2007; 2009) with third-grade Taiwanese students focused on collaborative learning behaviours. In the first, whole class reading was compared to group work reading (neither using technology). Group leaders differed in their approach: some helped lower-ability students, others ignored them and read individually; others even teased them. In the second study, the researchers compared mobile peer-assisted learning (MPAL) system-supported group work with group work unsupported by technology. They found significantly fewer learning-unrelated behaviours and less conflict in the MPAL class than in the comparison class.
Other positive findings are reported in Parks et al. (Reference ‡Parks, Huot, Hamers and Lemonnier2005), where students initially used process writing to comply with the teacher, but over time started using it in other classes, because they understood its utility.
Like Parks et al. (Reference ‡Parks, Huot, Hamers and Lemonnier2005), Mak & Coniam (Reference ‡Mak and Coniam2008) investigated the effects of technology use on students’ writing processes. Students worked for six weeks in groups of four producing brochures about their school via wikis, after which the brochures were printed and distributed to parents. Qualitative analysis revealed that students began by generally discussing the topic and posting their comments to the wiki, and then simply adding to existing text. As the project progressed, they began to edit each others’ writing and alternately developed and expanded on the existing wiki.
6.3.3 Confidence and anxiety
Of the non-linguistic outcomes reported in the articles selected for review, only one investigated the link between use of CALL and foreign language learning (FLL) anxiety, and two investigated confidence. This is surprising for two reasons. Firstly, FLL anxiety has been extensively researched (since the seminal work by Horwitz, Horwitz & Cope 1986) from a skills perspective and from a situation perspective, both of which might have been of interest to CALL researchers. Secondly, there is anecdotal evidence that peer pressure can have negative effects on learning in traditional classroom settings and we would have expected to see more research comparing whole class learning with computer-based individual learning.
Satar & Ozdener (Reference ‡Satar and Ozdener2008) investigated language learning anxiety. Recall that this compared voice chat and text chat discussions centred around information gap, problem solving, jigsaw and decision-making activities. Anxiety levels decreased for both voice chat and text chat, but not for the control group, although the decrease was only significant for the text chat condition; and anxiety when communicating with L2 native speakers was higher in the voice chat condition than in the text chat condition. The authors conclude that it might be beneficial to start with text chat, especially for low-proficiency students.
A future review whose primary focus is the non-linguistic effects of CALL might provide a more detailed picture of which technologies used with which pedagogical activities work best to minimize language learning anxiety.
With regard to confidence, Tsou et al. (Reference ‡Tsou, Wang and Tzeng2006) found that students expressed greater confidence as a result of multimedia storytelling activities and Liu & Chu (Reference ‡Liu and Chu2010), who compared use of a context-aware ubiquitous learning system with comparable pen-and-paper activities, found that students’ confidence to apply the knowledge that they had acquired was greater in the former condition. Experimental students attributed their increase in confidence to engaging in real learning experiences not found in books and speaking activities. However, there was no group difference in self-efficacy with respect to the activities in the intervention.
We will now attempt to synthesize some of these review findings, making suggestions for improvements in future conceptualizations of CALL research and in its methodology. Finally we identify areas where further research is needed.
7. Synthesis and discussion
We have already summarized above our findings in relation to the keyword mapping process. Here we summarize the 47 studies which were reviewed in depth:
• There was a relatively even balance between studies conducted in primary (40%) and secondary (60%) schools.
• A much larger percentage of studies focused on writing (36%), reading (36%) and vocabulary (32%) than on speaking (19%) or listening (17%), which in turn substantially outnumbered studies of grammar (6%) or pronunciation (6%).
• Studies of grammar and writing more frequently involved secondary than primary students. Conversely, studies focusing on pronunciation, reading and listening more frequently involved primary than secondary students.
• The aspects of CALL most frequently investigated were multimedia (21%) and CMC (23%).
• There was more research on word processors, the internet, CMC and Web 2.0 with secondary than with primary students, and more research on multimedia with primary students than with secondary students.
• Regarding methodology, most studies employed mixed methods (49%), with an even balance between quantitative (28%) and qualitative methods (23%).
7.1 Contribution of the in-depth review to answering the review questions
7.1.1 What technologies are being researched, and why?
With respect to the question of ‘why’ the technologies were investigated, our review revealed that, where indicated, most of the studies were driven by policy rather than by SLA or educational theory. Policies generally focus on the need for a digitally literate population, rather than the proven or potential benefits of technology in language learning. Future research, we would argue, should focus on how technology can provide better conditions for language learning.
This exhaustive search of the literature on technology in primary and secondary teaching of English as an L2 has not yielded clear or sufficient evidence of its effectiveness. This is of some concern given the very large amounts of funding that are being made available worldwide for the purpose of encouraging its use for whatever reasons, political, economic or pedagogical. Note that we find only that the available research evidence is inconclusive; there may indeed be good practice involving CALL that simply has not been researched.
From the reviewed work on vocabulary, it would be difficult to recommend that technology in general or specific software should be used to improve vocabulary development. The little research into the effect on grammar is also inconclusive. In writing, as already indicated, the focus has been less on improving the writing skill (fluency, accuracy, complexity) and more on investigating the writing process. Nevertheless, there is some indication that digital technology may help to develop certain academic writing skills better than pen and paper, but the evidence is very slight.
Interestingly, it was in speaking and pronunciation studies that some tentative evidence for the effectiveness of CALL was found, although the number of studies was very small. There is potential, therefore, for future studies to replicate aspects of this research.
In any case, it is possible that a new conceptualization of CALL effects on language learning is needed: not simply moving the goalposts for the ‘traditional classroom’, but rethinking what is being compared. A striking finding, in more than one study, was that the strongest effect was not between a CALL-only condition and a traditional classroom condition, but from instruction combining the two. Another emerging piece of evidence worth pursuing further or replicating is that learners are not always able to benefit from CALL because of a lack of IT skills. This points to future research avoiding a simple binary design and including combinations of CALL with classroom or teacher support and ICT skill building.
While some feel that the normalization of technology in education implies that it is no longer necessary to justify the cost-effectiveness of CALL (Garrett Reference Garrett2009), it is clear from this review that it may still be necessary to ask whether CALL – on its own or in combination with traditional approaches – is more effective than traditional approaches alone. To answer this question a much more rigorous methodology is needed. First, one needs to isolate the key feature of each technology that makes it different from non-CALL pedagogy. An obvious example of this (and a research field which surprisingly offered us very few insights) is that more advanced technologies must surely be able to offer learners individualized feedback on spoken utterances, something that is more difficult in a large class of learners with one teacher. Having identified this key feature in a technology, one might then relate a study to theories of language learning – in this case perhaps the use of recasts and other forms of error correction, linked to the individual learner's development (Philp Reference Philp2003), again something that is more difficult in a large class. Finally, one would need to make a decision about the kind of comparison being made. If the comparison is simply of non-technology with technology, one would need to consider time on task and other variables. If the condition of interest is non-technology plus technology, one would need to measure magnitude of gains against time taken away from other activities. All these issues need to be thought through and justified much more than they have been to enable rigorous theorizing around CALL and SLA.
Unless these different approaches to CALL research are investigated, we will continue to be confronted with studies that mean well but offer teachers, learners and educational administrators very little guidance. Moreover, if researchers continue to carry out broad comparison studies it will be difficult for them to build a body of evidence generalizable to new technologies in language learning. This is important, given the pace of technological evolution.
If this review has produced few answers to our first review question, it has been more productive of insights about attitudes and learner behaviours.
A number of studies have shown that technology use can qualitatively change students’ learning experiences. There is evidence that learners have positive attitudes towards CALL, and that a number of higher-level strategies and classroom behaviours may be improved by the use of a variety of technologies. The results in this area once again highlight the need to reassess notions of effectiveness and approaches to technology evaluation. A different learning tool may produce different learning behaviours which are beyond linguistic outcomes but are no less educationally valid. This points to the desirability of starting from what is known about how primary and secondary learners acquire an L2 and investigating whether there are affordances of the technology which may accommodate this acquisition better than typical classroom configurations.
7.2 Quality of the research reviewed
In assessing the contribution of 47 studies to the overall review, as described above, two independent reviewers judged each study according to a number of criteria. The results are not encouraging (Tables 5–11). For our research question regarding the facilitative effect of CALL on language learning outcomes, only six studies (Troia Reference ‡Troia2004; Zhang et al. Reference ‡Zhang, Gao, Ring and Zhang2007; Nakata Reference ‡Nakata2008; Satar & Ozdener Reference ‡Satar and Ozdener2008; Chen & Li Reference ‡Chen and Li2010; Chen et al. Reference ‡Chen, Chen and Sun2010) were judged to have made a high contribution to the first review question. Unfortunately, one of these found no effect. Of the remaining studies reviewed in depth, 16 were judged to have made a medium and 25 a low contribution. This is not to say that the studies were not relevant to the review questions (relevance ratings in Tables 5–11). Indeed, many were asking precisely the kinds of questions that we were interested in answering. However, the design and methodology employed did not lead the reviewers to have much confidence in the findings. Let us look at this issue in more detail.
In the vast majority of studies, generalizability does not appear to be of importance. Although populations are generally referred to or at least implied, it is not clear what marks them out as special or different from other populations. Sampling frames are almost never provided; consequently, it is unclear how the sample relates to the population. For example, are all secondary schools in one country so similar that a sample taken from one school represents the behaviour of secondary students across that country? Or should the researchers have considered different geographic locations, socio-economic status, different entrance requirements, different funding provision (inter alia for technology), and so on? We believe that, when considering changes to pedagogies and the costs involved, some attempt should be made at least to match the sample to the population, even if that population has then to be described in quite circumscribed terms (e.g. grade 4 children in urban Taiwan as opposed to primary school children the world over).
The second problem we observed in many studies was a design validity one. Often the basis of assignment to experimental and control conditions was not stated. Many instruments were inadequately described and insufficiently piloted. It was often claimed that progress had been measured even though no pre-testing had apparently been carried out; and some of the statistical procedures adopted did not inspire confidence.
In terms of analysis we often found a reliability problem, with only very rare examples of intra- or inter-rater reliability procedures being reported. This poses questions of analyser impartiality and analyser error.
Perhaps our strongest criticism is reserved for the many studies that simply did not make the link between the learners they were researching, the technology that was being investigated, and the theories related to the outcome variables they were testing. This relates to our ‘what technology and why’ review question. It is important to understand what motivates a CALL study, especially if it is not an evaluation of new software that many schools are adopting. It needs to be asked why this CALL research is being done with this group of students at this time, and how it relates to the potential for theory building in SLA.
8. Conclusions
This systematic review sought to provide a general overview of research in CALL since 1990 in primary and secondary education language education, where governments are investing large amounts of money in technology. The reason for this investment is most often framed as the need for a technologically competent population. Whatever the reasons for the investment, we believe, as applied linguists, that the potential language learning benefits of technology merit investigation.
We also sought to identify, in some depth, CALL research in primary and secondary EFL/ESL in the first decade of the twenty-first century, in terms of both its impact on SLA and its potential for creating a different learning environment – our insights question. We conclude, particularly in the case of impact, that the research is too diffuse and too wide-ranging. Moreover, it does not build on previous work in a way that might enable it to contribute better to theories of language learning. In order to improve this situation, we suggest that the reasons for carrying out the research with particular populations are clearly expressed and argued. We also ask that the attributes of technology are clearly identified and rigorously tested for their contribution to language learning, and that the research in general aims for much higher levels of quality.
We hope that researchers will aspire to this, as we believe that both the research community and practitioners are very much in need of having this evidence available to them.
Acknowledgements
We would like to thank the systematic review teamFootnote 5, Oxford University Press for funding this project, the four anonymous reviewers for Language Teaching, and Graeme Porte for his help and patience throughout.
Appendix A: Keyword map
Note/Key:
Ernesto Macaro is Professor of Applied Linguistics in the Department of Education, University of Oxford. He supervises doctoral students and teaches on the M.Sc. in Applied Linguistics and Second Language Acquisition and on the PGCE (teacher education programme) for Modern Foreign Languages. His research interests are in language learner strategies and in teacher code-switching as part of classroom interaction.
Zöe Handley is an Oxford University Press funded research fellow in the Department of Education at the University of Oxford working on the project ‘New Technologies in English as a Foreign Language’.
Catherine Walter is a University Lecturer in Applied Linguistics in the Department of Education, University of Oxford. She supervises doctoral students and teaches on the M.Sc. in Applied Linguistics and Second Language Acquisition. Her research interests are in reading comprehension, pedagogical grammar and teacher development.
