2.1 Theoretical or practical perspectives on materials evaluation and design?

Effective materials evaluation or design relies on both theoretical insights from applied linguistics research and ‘the accumulated wisdom of best practices in the teaching profession’ (Dörnyei Reference Dörnyei2009: 267). These two approaches are complementary and interdependent, with research often providing the stimulus for innovation and change (Howatt Reference Howatt1984: 226) and the classroom acting as a solid testing ground for theories. And whilst empirical research maintains strong internal validity through careful control of experimental variables, classroom research has stronger external validity by more closely resembling the contexts it wishes to generalize to (Nunan Reference Nunan1992). Unfortunately, as has been often noted in the literature (for example, Clarke Reference Clarke1994), theory and practice do not always sit comfortably together – researchers and teachers often attend different conferences, publish in different journals and regard each other with a certain level of distrust. It would therefore be useful here to begin by assessing the extent to which theory and practice are brought together in the four books under review. One way to do this is by examining the number and type of citations included in each volume, and this information is summarized in Tables 1 and 2.

Table 1

Summary of citation characteristics

Table 2

Summary of most frequently cited journals or books

ⁱ Practical English Teaching, as the name suggests, focused on practical teaching ideas. It is no longer published.

ⁱⁱ IATEFL Issues (now IATEFL Voices) is a newsletter published by the International Association of Teachers of English as a Foreign Language (IATEFL).

The first column in Table 1 shows the average number of citations per page and this gives some indication of the frequency of references to other researchers’ work from the field. Although these figures do not support any qualitative claims – writing over-populated with irrelevant references is, after all, an unhelpful distraction – it suggests that Harwood (Reference Harwood2010) is the most grounded in the literature and McGrath (Reference McGrath2002) the least. The next three columns give an indication of the proportion of book to journal references, which shows a similar pattern to that mentioned above. However, it is clear that Tomlinson (Reference Tomlinson2003) relies more on citations from books than journals and Harwood (Reference Harwood2010) is heavily weighted towards journal references when compared to the other three publications. Part of the reason for these differences could be that there is, simply, a wider range of journal articles related to materials design available in the current research literature. This is supported by the final two columns of Table 1: the fifth column, which indicates the range of journals referenced, shows that Harwood (Reference Harwood2010) considers a much wider variety of journal titles than the other three books, covering areas such as applied linguistics, general education, educational technology, ESP and EAP. The final column shows the range of journals cited divided by the total number of journal references, which gives a further indication of the breadth of coverage of different types of journal (where the maximum value is one). This suggests that while Mishan & Chambers (Reference Mishan and Chambers2010) has fewer journal citations than Harwood (Reference Harwood2010) and Tomlinson (Reference Tomlinson2003), it does include a wide variety of journal types.

Table 2 summarizes the most frequently cited journals or books in each of the four publications, with percentages (in relation to total numbers) given in parentheses.

Column one, showing the journals most often referenced, suggests a more practical bias in McGrath (Reference McGrath2002) and Tomlinson (Reference Tomlinson2003), and a more theoretical bias in Harwood (Reference Harwood2010) and Mishan & Chambers (Reference Mishan and Chambers2010). McGrath (Reference McGrath2002) stands out as highly reliant on just three journal titles, whereas the other publications give space to a wider variety of perspectives from the literature. The second column, showing the top books referenced, indicates surprising similarities across the four publications, with just three titles commonly cropping up: Materials development in language teaching (Tomlinson Reference Tomlinson1998), Getting started: Materials writers on materials writing (Hidalgo, Hall & Jacobs Reference Hidalgo, Hall and Jacobs1995) and Developing materials for language teaching (Tomlinson Reference Tomlinson2003). This is perhaps symptomatic of the paucity of relevant publications in the field – itself a consequence of the assumption of triviality often ascribed to materials evaluation and design (Richards Reference Richards and Harwood2010: ix).

2.2 Which voices with a stake in the materials design process are represented?

McGrath (Reference McGrath2002) is primarily aimed at practising teachers who already have some classroom experience behind them. Its design, with interspersed reflective tasks, sample materials and answer keys, makes it ideal for use on M.A. modules dealing with materials evaluation and design, or by teachers working individually or collectively in their own educational contexts. Since the book emphasizes the selection or adaption of published textbooks, it is probably more suited to teachers in the earlier stages of their career, when more guidance is necessary. Once they are able to critically assess the quality of professionally produced coursebooks, they are in a better position to begin writing their own materials tailored to their specific teaching contexts. The focus on pre-, in- and post-use evaluations of published materials should also be of interest to educational administrators keen to improve the effectiveness of their courses, without committing the time and money necessary for the production of in-house materials. Unfortunately, it is often the case that very little effort is invested in the systematic choice or evaluation of input across English language programmes, and students receive a haphazard selection of materials ‘by lottery’ as they advance through the system. Of course, no programme will ever fully meet the communicative needs of all of its learners all of the time, but without a clear picture of what those needs are and the extent to which successive courses contribute to meeting them, administrators are working in the dark. McGrath has little to say himself on the empirical research underpinning the materials design process, referring instead to summaries in other work, such as McDonough & Shaw Reference McDonough and Shaw1993 (itself very dated, with an average publication year in the bibliography of 1988), so readers more interested the theoretical aspects of the area should look elsewhere.

Tomlinson (Reference Tomlinson2003) aims to inform working teachers or teachers in training of insights into materials development from developers and users throughout the world and therefore also has a strong practical bias. The variety of native and non-native voices from disparate teaching contexts (Singapore, Britain, Japan, Spain, the USA, the Philippines, Indonesia, Namibia, Turkey, Hong Kong, Argentina, Lebanon, Brazil, Hungary, Vietnam, New Zealand and Romania) certainly enriches the descriptions and is important, given the fact that the vast majority of English language teaching takes place outside ‘inner circle’ speech communities (Kachru Reference Kachru, Quirk and Widdowson1985). Bao Dat, for example, reminds us of the problems that slick internationally-marketed textbooks can cause at the local level: ‘One teacher at my institute in Vietnam reveals, “I hardly know what to do with this lesson that invites my students to talk about how to use parking meters or vending machines which they've never seen, which simply don't exist in our country”’ (Tomlinson Reference Tomlinson2003: 389). Of course, issues like these can be overcome with creative adaption: Vietnamese students could be encouraged to use clues from the L2 text to infer how to operate these machines, or to take a more critical stance by discussing how a society that uses parking meters and vending machines must differ from their own. Tomlinson, however, in this volume and elsewhere, sees the production of locally relevant materials as the way forward:

A number of authors in the book address this issue (see Chapters 8, 9, 18, 22, 30 and 31).

Another notable feature of Tomlinson (Reference Tomlinson2003) is its focus on materials development from the publisher's perspective (Chapters 2, 3, 7, 8 and 10), where ‘other forces than sound current pedagogy are at work’ (Mares, in Tomlinson Reference Tomlinson2003: 132): namely, acceptability, marketability and profit. This is illustrated by David Hill's interesting analysis of visual input in coursebooks (Chapter 10), where he estimates that around 55 per cent of illustrations in two well-known publications are purely decorative – an editorial decision motivated more, no doubt, by a desire for immediate market appeal during hurried selections between books (known as ‘flick-tests’) than long-term learning goals (Bell & Gower Reference Bell, Gower and Tomlinson1998: 125). According to Tomlinson (Reference Tomlinson2008: 7), this still holds true for more recent textbooks, where colourful photographs often adorn the top-right hand corners of right-hand pages to attract potential buyers making superficial comparisons between choices. The economic concerns of publishers can therefore often be seen as working against pedagogic interests or research-driven innovation, and it would be useful for users to carefully consider whether the advantages offered by global coursebooks, such as convenience and professional appearance, are outweighed by their disadvantages.

In terms of addressing the interests of applied linguistics researchers, the book is rather patchy. The editor notes that most materials writing is currently done largely intuitively and he supports a move towards a more principled approach, based on established findings from SLA research. This is an admirable position to begin from, and a step which, in my view, is essential to advance the field (see Tomlinson & Masuhara Reference Tomlinson and Masuhara2011 for efforts in this direction), but, unfortunately, the volume does not often live up to these lofty ideals. Tomlinson lists a number of ‘principles of SLA’ relevant to materials development (Tomlinson Reference Tomlinson2003: 20–22) but these are essentially a repetition of ideas from an earlier book (Tomlinson Reference Tomlinson1998), with no more than a token nod towards the research literature. Where SLA theory is referred to in the chapters, there is often an over-reliance on Krashen's out-dated models, which remain stubbornly popular in ELT literature despite their lack of empirical grounding or falsifiability (see, for example, Gregg Reference Gregg1984). Since the core target audience for the book is practising teachers, it could be argued that there is little appetite for empirical investigations, but surely it is the responsibility of writers on materials design to show how applied linguistics research is both relevant and useful to classroom practice – some authors in the volume, such as Nation (Chapter 23), do this more successfully than others.

Harwood (Reference Harwood2010), consisting of contributions from academics in the Anglophone world, naturally tends to foreground theoretical models pertinent to materials design. These cover a wide range of areas describing learners, language learning and language use in social contexts, including needs analysis (e.g. West Reference West1994), task design (e.g. Bygate, Skehan & Swain Reference Bygate, Skehan and Swain2001; Ellis Reference Ellis2003), vocabulary learning (e.g. Schmitt Reference Schmitt2000), cognitive approaches (e.g. Skehan Reference Skehan1998), metacognitive instruction (e.g. Vandergrift & Tafaghodtari Reference Vandergrift and Tafaghodtari2010), critical applied linguistics (e.g. Pennycook Reference Pennycook2001), genre analysis (e.g. Swales Reference Swales1990), interactional linguistics (e.g. Selting & Couper-Kuhlen Reference Selting and Couper-Kuhlen2001), specialized corpora (e.g. Connor & Upton Reference Connor and Upton2004) and community of practice approaches (e.g. Barton & Tusting Reference Barton and Tusting2005). Despite this emphasis on theoretical insights, the authors appearing in the book are careful to note their limitations: research results are often inconclusive and may, in any case, prove to be unhelpful when transposed into classroom contexts. There is also an awareness that applied linguistics researchers need to work harder not only to demonstrate the relevance of their work to practising teachers but also to test their theories through course design followed by ‘sustained course delivery’ (White Reference White2006: 250). The volume has a strong focus on English for specific or academic purposes, and is therefore likely to be of particular interest to material writers or practitioners involved in higher education, but many of the principles raised have broader relevance too. Each chapter in the book ends with discussion questions and tasks, making it useful for teacher education programmes or study groups, although there is an assumption of considerable background knowledge and experience which may ask too much of novice practitioners or material writers.

Mishan & Chambers (Reference Mishan and Chambers2010) promises ‘a snapshot of the contemporary influences on language learning materials development from diverse perspectives around the globe’ (blurb on the back cover) and that is largely what the reader gets. Contributors explore materials development from the perspectives of coursebook writers, teacher trainers, language teachers and university lecturers, and they cover a wide range of target groups including young learners (TEYL contexts), language teacher education (LTE) post-graduate students, adult immigrants in the UK (English for Speakers of Other Languages contexts) and English as a Foreign Language (EFL) university students in Japan, Tunisia and Venezuela. In this sense, the chapters do not attempt to come together to form a coherent whole; instead, they provide an interesting cross-section of a complex and diverse contemporary society. The book places a strong emphasis on the multiple ways technology is affecting materials development. First, language corpora are deepening our understanding of variation across spoken and written modes in different speech communities, and improved descriptions, particularly of spoken discourse, are beginning to find their way into grammar reference books (e.g. Carter & McCarthy Reference Carter and McCarthy2006), coursebooks (e.g. McCarthy, McCarten & Sandiford Reference McCarthy, McCarten and Sandiford2005a, Reference McCarthy, McCarten and Sandiford2005b, Reference McCarthy, McCarten and Sandiford2006a, Reference McCarthy, McCarten and Sandiford2006b) and locally-produced teaching or training materials. Frequency lists are influencing which words or multi-word units are included in the curriculum and at what stage they are introduced. Second, the Internet is allowing materials writers all over the world to access vast quantities of authentic language to create materials from, and enabling them to share their creations with peers. Third, editing and word processing software is facilitating the production of professional-looking ‘home-grown’ materials. Finally, Web 2.0 technology is allowing collaborative authoring of web sites by students and tandem learning projects between distant institutions. Technology-rich educational environments are clearly opening up exciting new possibilities in materials and task design but, recalling the cautionary note on research insights in Harwood (Reference Harwood2010), many of the authors stress the limitations of their work and the need for both ‘pedagogic cost-benefit analysis’ (Timmis Reference Timmis, Mishan and Chambers2010: 73) and extensive trialling in real classroom contexts before any strong claims can be made.

2.3 The search for general principles in language learning materials development

With language learning occurring across such a diverse range of contexts and cultures, the search for general principles to guide materials design and evaluation is an important starting-point. This is a formidable task, however, since it requires drawing together expertise from a wide range of disciplines, reconciling disparate views in order to formulate generalizable principles, and testing these out in different learning environments over extended periods to determine their effects on students’ motivation and acquisition (Hulstijn & Schmidt Reference Hulstijn, Schmidt, Hulstijn and Schmidt1994: 5). This level of effort and interdisciplinary collaboration has been extremely rare in the past, as Tomlinson (Reference Tomlinson2011: 25) remarks:

So how well do the four volumes reviewed here deal with this area? McGrath (Reference McGrath2002) devotes a chapter to the topic of systematizing materials design, but the principles he mentions are rooted more in classroom experience than theoretical models. He summarizes the views of a number of key writers in the field, such as Richards, Nunan and Hutchinson & Waters, but does not make any serious attempt to compare or contrast them or to critically evaluate their claims in terms of their empirical grounding. While useful for providing teachers in the early stages of their careers with practical advice (as the author intends), there is little there to advance the field of materials design.

Tomlinson (Reference Tomlinson2003) also includes extended discussions on principles, particularly in Chapters 1 and 6. In Chapter 1, he provides a long list of principles based on ‘findings’ from learning and SLA research, along with the get-out clause that we all have to decide for ourselves what evidence is convincing. While some of the points he makes seem empirically defensible, such as the importance of affect or noticing in learning, others are more controversial. He makes the strong claims, for example, that ‘experiential learning’ and ‘multidimensional processing of intake’ are essential for successful learning, without clearly defining these terms or describing the evidence which supports these conclusions. How do we measure the ‘depth’ at which learners are experiencing materials or the number of ‘dimensions’ they are processing input on at any particular moment? And how do we establish a causal link between these constructs and the language acquisition that takes place as a result? As Schmidt (Reference Schmidt, Hulstijn and Schmidt1994: 12) says, ‘subjective experience is irreducibly a first person phenomenon, whereas progress in science always depends upon taking a third person perspective. . .such facts as there may be about internal mental events can never be verified by objective methods and are not the data of science’. Unfortunately, none of the principles listed are scrutinized in any detail or with any scientific rigour. In Chapter 6, Tomlinson discusses two principled frameworks for materials development: one ‘text-driven’ and the other ‘task-driven’. The text-driven approach abandons attempts to structure the syllabus along lexico-grammatical lines, giving priority instead to cognitively and affectively engaging input. Attention to form only takes place later, after the texts have been responded to more holistically. This has obvious appeal for teachers and learners tired of the dreary texts, contrived to illustrate the grammatical point-of-the-day, that often populate published coursebooks. It is also underpinned by credible research from cognitive psychology into the importance of affect in learning (e.g. Schumann Reference Schumann1997), the role of noticing in acquisition (e.g. Schmidt Reference Schmidt1990) and constructivist theories (e.g. Williams & Burden Reference Williams and Burden1997). However, no hard evidence is presented for the superiority of the text-driven syllabus over more traditional PPP (present-practice-produce) approaches and we will have to wait for more classroom-based research to help resolve this issue. The promised discussion of a task-driven approach does not actually materialize in the chapter and, along with frequent typographical errors in the references provided, suggests that the book went through a hasty editing process.

Harwood (Reference Harwood2010), as mentioned in section 2.2, tends to stay close to theoretical principles throughout the volume, and while space here does not allow a comprehensive examination, I will briefly consider some of the key issues raised. In Chapter 2, Ellis explains how SLA research has informed task design and grammar teaching. While traditional PPP approaches are supported by skill-learning theories (e.g. Johnson 1996), which see declarative knowledge as gradually becoming automatized through practice activities, task-based language teaching is supported by emergentist theories (e.g. N. Ellis Reference Ellis1998), which see implicit knowledge as emerging gradually as learners, ‘eager to exploit the functionality of language’, are exposed to language data in a ‘communicatively-rich human social environment’ (ibid.: 644). Rod Ellis is careful not to draw any firm conclusions for materials design from these differing perspectives, but suggests that they provide a wealth of ideas to be tested out in classrooms. Reinders & White, in Chapter 3, describe two influential frameworks for principled materials design, produced by Chapelle (Reference Chapelle2001) and Doughty & Long (Reference Doughty and Long2003), and used in Task-Based Language Teaching (TBLT) approaches. These are largely overlapping in their goals, both aiming to create optimal environments for learning by reference to research from SLA or educational/cognitive psychology. Doughty & Long's framework of ten ‘methodological principles’ (MPs) is the more comprehensive of the two and includes: i) using tasks rather than texts as the unit of analysis; ii) promoting learning by doing; iii) elaborating, rather than simplifying, input; iv) providing rich input; v) encouraging ‘chunk’ learning; vi) focusing on form; vii) providing negative feedback; viii) respecting learners’ individual developmental processes; ix) promoting collaborative learning; and x) tailoring instruction to cater for individual differences (goals, interests, motivation, cognitive style and learning strategies). Although Doughty & Long (ibid.) accept that some of their MPs might not stand the test of time, this list seems like an excellent starting-point for language learning materials design.

The strongest message to emerge from Mishan & Chambers (Reference Mishan and Chambers2010) in terms of principled materials design is the need for input based on attested language use rather than writers’ intuitions, which are notoriously unreliable (e.g. Biber & Conrad Reference Biber and Conrad2001). Corpus studies have demonstrated clearly how language varies with mode (spoken or written) or register (conversation, fiction, newspapers, academic prose, etc.) and these insights have begun to find their way into reference books (e.g. Biber et al. Reference Biber, Johansson, Leech, Conrad and Finegan1999; Carter & McCarthy Reference Carter and McCarthy2006) and language-learning materials. The focus in the volume is primarily on evidence from spoken corpora since it is typically spoken discourse that has been poorly represented in coursebooks (e.g. Gilmore Reference Gilmore2004; Cullen & Kuo Reference Cullen and Kuo2007), and contributors show how a corpus-informed approach can help materials designers with decisions about what conversational strategies, vocabulary (single words or multi-word units) or grammar patterns to include in the syllabus. Again, recommendations are hedged with cautionary notes – Timmis (Mishan & Chambers Reference Mishan and Chambers2010: 63) warns against ‘the reflex fallacy’ (the mistaken notion that just because a particular language feature exists it must be taught) and McCarten & McCarthy (ibid.: 20–23) illustrate some of the difficult decisions materials writers need to make when mediating between authentic discourse and learners. In her chapter on tasks and task authenticity in the digital era, Mishan (ibid.: 165–167) challenges some of the conventional classroom practices designed to enhance comprehensibility (text simplification, pre-teaching vocabulary and comprehension questions), reminding us that, in our search for general principles relevant in contemporary classrooms, we also need to critically examine established approaches.