1. Introduction
Several recent developments have stimulated and justified the increased interest in the competence of teachers to estimate the learning impact of instructional materials. Firstly, new technologies such as the Internet enable teachers to explore the possibilities of these new technologies to present their learners with texts and learning task materials that are much more life-like, topical and attractive than the constraints of traditional printed materials would allow. Secondly, current educational reforms and pedagogical trends in Europe encourage more independent and autonomous learning. Most current textbooks do not lend themselves easily to this use, so teachers need to adapt or develop their own (self-access) materials for their learners. Furthermore, there is a growing demand on teachers to differentiate. Published materials are by their nature generally less suitable for this purpose. In such circumstances, as Marsden (Reference Marsden2001) has shown in his studies of textbooks, teachers will often have to adapt or design supplementary course materials. Finally we see a trend towards competence-based learning. In Europe this development has received an impetus by the introduction of the Common European Framework of Reference (CEFR; Council of Europe 2001; Little Reference Little2007). However, as Westhoff (Reference Westhoff2007) argues, the CEFR is not easily compatible with traditional grammar-oriented pedagogy, as reflected in the most commonly used textbooks. Those practitioners who want to shape their work according to principles underlying the CEFR will have to look for alternatives to the usual form-focussed textbook tasks.
These four developments demand from teachers an extended competence in evaluating learning materials. This is especially true of tasks that are life-like and content-oriented and, as a consequence, integrate many aspects of language acquisition. However, as de Matos (Reference Matos2000) points out, the critical analysis of such materials has often been overlooked or even neglected in teacher education programmes. In order to fill this gap, a scaffolding instrument for a priori assessment of the learning impact of language learning tasks seems to be useful. In this lecture I will describe the development of such an instrument, its theoretical basis, some of its characteristics and the subsequent stages we have gone through in our endeavour to develop a feasible, reliable and valid form for it, resulting in the creation of different tools for different purposes.
2. A ‘yardstick’ for complex language learning tasks
The incentive for the development of a tool for the assessment of language learning tasks was the fact that Dutch FL teachers looking for learning tasks that were realistic and embedded in rich, life-like contexts in which they were functional discovered WebQuests (Dodge Reference Dodge1995). Dodge defines the concept as follows: ‘A WebQuest is an inquiry-oriented activity in which some or all of the information that learners interact with comes from resources on the internet’ (http://webquest.org/index-resources.php). Dodge gives a set of well thought out, generic criteria to assess the quality of WebQuests. However, WebQuests are developed for all subjects in the current school curricula. Unfortunately, these generic criteria do not provide indications about those characteristics of tasks which specifically foster second language acquisition (SLA). Thus, the use of some WebQuests may contribute substantially to SLA, whereas others do not yield much progress. Our first aim was to help these teachers choose and/or adapt or design WebQuests that could be expected to foster SLA by developing an instrument for qualitative analysis. We created a new name for those WebQuests, which appeared to focus on SLA: LanguageQuests (LQ). However, during the development of this instrument we noticed that we actually defined criteria that could be applied to many types of integrated, life-like and functional tasks.
2.1 The theoretical framework of the first ‘yardstick’
In our view, we considered a first requirement to be that the instrument should be theory-based. We started with current insights from SLA theory in general and, more specifically, with the characteristics of tasks in a task-based approach (TBA). In general, no-one in cognitive psychology seriously doubts these days that acquisition of knowledge is the product of mental activity (see Anderson Reference Anderson2004). Specifically, knowledge is assumed to be stored in long term memory (LTM) in the form of traces resulting from mental actions in working memory (WM) (for an overview, see Baddeley Reference Baddeley1997, Reference Baddeley2001, Reference Baddeley2007; Carroll Reference Carroll2004). According to Baddeley (Reference Baddeley2001: 852), the working memory is the place where information, coming from the sensory stores and from LTM, is combined and assembled in a process of constructing meaning. This means that learning is basically activity-based and that a decisive part of this activity takes place in WM. Accordingly, tasks can be analyzed with respect to the type, variety and intensity of the mental activity they elicit in WM.
2.1.1 Five domains of mental actions
We reviewed current SLA theory and distinguished five activity domains in which mental actions could be expected to contribute to SLA. In spite of fierce debates on various issues, a certain consensus seems to exist regarding five basic assumptions on domains of activity that can be supposed to contribute to SLA (for an overview see Lightbown & Spada Reference Lightbown and Spada1999; Brown Reference Brown2000; Richards & Rodgers Reference Richards and Rodgers2001; Mitchell & Myles Reference Mitchell and Myles2004).
1. Exposure to input. Without extended exposure to a rich input, there is little SLA (Krashen Reference Krashen1985). Although very few of Krashen's ideas could be confirmed empirically, and most of them have now been largely challenged (McLaughlin Reference McLaughlin1987; Ellis Reference Ellis1990; Norris & Ortega Reference Norris and Ortega2000; DeKeyser Reference DeKeyser, Doughty and Long2003), there seems to be a broad consensus in the recent scientific literature that being exposed to a rich FL input is a crucial, although not exclusive, prerequisite for SLA. In order to be functional, this input should not be too far above the actual competence level of the learner. Expressed in Vygotskian terms, this level could be described as the learners' ‘zone of proximal development’. In the literature, this level is commonly referred to as ‘i + 1’, where ‘i’ stands for ‘interlanguage’.
2. Content-oriented processing. Mere exposure is not enough. There seems to be little doubt that being exposed to input is only effective if the input is processed, or, in more practical terms, if the learner has tried to understand its meaning. Something meaningful has to be constructed in WM before it can be passed on to and stored in LTM (see e.g. Johnson Reference Johnson1982; Wesche Reference Wesche, Krueger and Ryan1993; Skehan Reference Skehan1998).
3. Form-oriented processing. There is far less agreement on the role of grammar, or so-called ‘form–focused instruction’ (for an overview, see Burgess & Etherington Reference Burgess and Etherington2002; DeKeyser Reference DeKeyser, Doughty and Long2003, Reference DeKeyser2005; Doughty Reference Doughty, Doughty and Long2003). Although research outcomes have been presented recently which advocate the role of formal instruction (Housen & Pierrard Reference Housen and Pierrard2005), support seems to be growing for the so called ‘weak interface hypothesis’ (Ellis Reference Ellis1990). This hypothesis tries to explain the paradox that extended content-oriented input processing, combined with form-focused instruction, leads to better results than input processing alone, but that taught grammar rules are seldom used in producing output. The weak interface hypothesis claims that part of the learners' output is rule-directed, but that we do not know the rules they actually use. Learners probably construct their own rules by formulating hypotheses about aspects of the form of the FL by processing input. This process of hypothesis formation is supposed to be stimulated by directing the learner's attention to form aspects of the input. Such instruction is characterized as ‘Focus on Form’, to be distinguished from explicit grammar instruction, which is labelled ‘Focus on FormS’ (Long Reference Long, Bot, Ginsberg and Kramsch1991; Doughty & Williams Reference Doughty and Williams1998). Altogether, there seems to be growing agreement that some kind of form-awareness-raising activity can be supposed to foster SLA.
4. (Pushed) output. Several researchers have argued for the facilitating and stimulating role of (pushed) output production (‘pushed’ in this context means that demands are being made for correct and appropriate use of the foreign language). According to the output hypothesis (Swain Reference Swain, Cook and Seidlhofer1995; Swain & Lapkin Reference Swain and Lapkin1995; De Bot Reference De Bot1996) it is assumed to enhance fluency, and it makes language learners conscious of their deficits, and so increases their motivation to learn. The output hypothesis claims further that pushed output contributes to form-orientation and gives the teacher or the communication partner the opportunity to provide corrective feedback (for an overview of its effect, see Spada Reference Spada1997; Toth Reference Toth2006). Experiments seem to confirm this claim (Nobuyoshi & Ellis Reference Nobuyoshi and Ellis1993; Swain & Lapkin Reference Swain and Lapkin1995). Two varieties of output can be distinguished. One part of our language utterances consists of unanalyzed combinations (chunks) that are perceived as wholes (Lyons Reference Lyons1968). Their use is labelled ‘formulaic speech’ (Myles, Hooper & Mitchell Reference Myles, Hooper and Mitchell1998). Pushed output increases the learners' ability to use these ‘chunks’ in different situations and combinations (Schmitt & Carter Reference Schmitt, Carter and Schmitt2004; Schmitt et al. Reference Schmitt, Dörnyei, Adolphs, Durow and Schmitt2004). The other variety is somewhat misleadingly labelled ‘creative speech’ (Ellis Reference Ellis1986: 167–170). Misleading, because it has little to do with poems or creative writing; the term is used for rule-guided production.
5. Acting strategically. Generally speaking, only limited time is available for foreign language education. This means that there will always remain gaps in our knowledge. For that reason, it is useful to develop a repertoire of strategies to compensate for these deficiencies. We can compensate for deficiencies in receptive knowledge by applying reading and listening strategies such as inferring unknown elements, using prior knowledge, etc. (Westhoff Reference Westhoff1991a, Reference Westhoffb). To make up for deficiencies in productive competence, we can use communication skills such as negotiating meaning, avoiding, description, fillers (Bialystok Reference Bialystok1990; Poulisse Reference Poulisse1990; Littlemore Reference Littlemore2001).
These five assumptions can be considered as the basic ingredients of effective SLA activities. Together they form what Westhoff (Reference Westhoff2004) calls the ‘SLA penta-pie’, containing all the basic ingredients of a complete and nutritious ‘language meal’.
2.1.2 Criteria from a task-based approach
There is an extended literature about tasks and the TBA. There are many different definitions of the concept of task, as well as differences in the claims regarding a task's effectiveness (for an overview, see Bygate, Skehan & Swain Reference Bygate, Skehan and Swain2001; Ellis Reference Ellis2003). They vary from very broad definitions like Breen's description that in fact encompasses as good as all classroom activities directed at language learning ‘form the simple and brief exercise type to more complex and lengthy activities such as group problem-solving or simulations and decision-making’ (Breen Reference Breen, Candlin and Murphy1987: 23) to Skehan's (Reference Skehan1998) extensive definition which states, among other things, that tasks should be meaningful, related to real-world activities, and learners should not be asked to regurgitate other people's meanings. Willis (Reference Willis1996) emphasizes the need for goal-orientedness of a task, requiring that the target language has to be used for communicative purposes for an outcome to be achieved. According to Swain & Lapkin (Reference Swain, Lapkin and Bygate2001) tasks should involve learners in interactive and collaborative activities in order to elicit negotiation of meaning. However, there is common core of characteristics for effective tasks; they are as follows:
• Tasks should be oriented to content. To understand and to be understood should be the main concern.
• Tasks should be life-like and functional in the sense that they produce an outcome in the form of a useful product.
• Learners should be allowed or even encouraged to make use of their own linguistic resources.
• The activity should be holistic.
• It should preferably be carried out as teamwork, mostly with divided roles.
• Although the product has to meet pre-set specifications, a considerable flexibility should be allowed in the way learners go about executing the process.
2.2 The construction of a ‘yardstick’
In addition to the generic criteria for WebQuests as developed by their originator Bernie Dodge, we constructed two supplementary sets of criteria, one covering the five action domains presented earlier, inferred from SLA theory, and one where we aimed to summarize the task characteristics tapping into the extent to which the analyzed LanguageQuest could be considered to foster SLA according to the claims of the TBA.
The first version of the instrument was designed in the Netherlands in cooperation with a group of FL teachers whose main objective was the construction of new powerful WebQuests for SLA. The ‘yardstick’ was a by-product of the construction process. It had the form of a checklist consisting of statements. For outsiders the initial product appeared to be not very user-friendly and its use required more knowledge of SLA theory and terminology than practitioners usually possess. In the second stage we used a Delphi-like procedure. First we revised the instrument quite substantially on the basis of earlier experiences. This new tool was tried out by a group of teachers in a workshop setting. With the help of their (not very favorable) comments and suggestions for improvement, the wording of the criteria was substantially adapted again and the tool became an elaborate rubric. This third version was tried out anew by a further group of teachers for fine tuning. At this stage it became clear that, in order to be transparent, unambiguous and understandable for those who might not have read extensively in the field of SLA theory, the formulations of the descriptors had to be rather extensive. As a consequence the instrument became lengthy and too unwieldy for practical application. We tried to solve this problem by digitizing the ‘yardstick’ so that we could use a hypertext structure. As is demonstrated in Table 1, only the global descriptor of each criterion is visible at the surface. More detailed specifications and explanations are given in the underlying text, which can be reached on demand by clicking the surface text. In every extended description, significant keywords are highlighted (bold text in Table 1).
Two examples of criteria for assessing LanguageQuests.
The final version was translated from Dutch into English and German. The English version was tried out in an international workshop of the European Centre for Modern Languages (ECML) in Graz (Austria) with satisfactory results (http://www.ecml.at/mtp2/Lquest/html/LQuest_E_Results.htm). The German version was used in a regular seminar for undergraduate students of L2 German at the University of Vienna. These students used the ‘yardstick’ both as an evaluation tool for existing WebQuests and as a structuring tool in the process of constructing new complex tasks. It was used by the students without support. The students were positive in their evaluation of both functions.
The final versions (in Dutch, English and German) were complemented with a grading system (one, two or three ‘stars’). Subsequently, the ‘yardstick’ was put on the Internet in an interactive format (http://webquest.kennisnet.nl/talenquest/talenquest_beoordelen). Once the questions are completed, the end score per category and the total score, including the grading in terms of ‘stars’, are calculated automatically. Comments and suggestions for improvement of the reviewed task can be added by the reviewer.
3. An assessment tool as learning material for teachers
Although the LQ ‘yardstick’ appears to help practitioners in estimating an LQ's appropriateness for FL learning, it tells little about the nature of elicited mental activities and their relative effectiveness in terms of learning impact. Evidently, not all activities in the domain of ‘form-oriented processing’ found in current textbooks can be assumed to be equally effective for SLA. Although theory-based, our ‘yardstick’ did not provide an in-depth analysis, necessary to tap into those differences in effectiveness. An analysis that aims at determining the characteristics of mental actions elicited by a task would be so time-consuming that it seemed unfeasible to construct a tool that would cover an extensive and complex task like an LQ, let alone an extensive body of materials such as a complete textbook. On the other hand, experiences with early versions of the tool led to the supposition that working with an instrument for in-depth analysis of relatively small tasks enhanced one's competence to make ‘educated guesses’ about the potential effectiveness of more complex ones. Users seemed to develop a keen eye for the learning impact to be expected from tasks, so we decided to explore the possibilities of a tool for in-depth analysis of small tasks which would serve as an instrument of learning for teachers, aiming at the enhancement of their competence in estimating the impact of language learning tasks. For this purpose, we had to develop a new tool because existing ones did not provide assistance in delineating the characteristics of mental actions in WM and their role in constructing and storing meaning. Such tools do not contribute to enhancing a teacher's insight into which characteristics of a task promote learning. And this insight was, in our view, precisely the basis of their increased competence. We needed a theoretical framework that would give us more clues regarding those characteristics of mental actions that account for learning results.
In tackling this problem we used the Multi-Feature Hypothesis, as presented by Westhoff (Reference Westhoff2004) and further developed and refined by Moonen, De Graaff, & Westhoff (Reference Moonen, De Graaff and Westhoff2006). In this hypothesis, insights are combined from schema theory (Rumelhart & Ortony Reference Rumelhart, Ortony, Anderson, Spiro and Montague1977) and connectionism (McClelland, Rumelhart & Hinton Reference McClelland, Rumelhart, Hinton, McClelland and Rumelhart1986; Gasser Reference Gasser1990; Bereiter Reference Bereiter1991, Reference Bereiter and Smith2002; Plunkett Reference Plunkett and Plunkett1998). Among cognitive psychologists, it is widely accepted that the product of a cognitive learning process should not be perceived as a template, but as a more or less open mental structure of neural units (Greeno & Simon Reference Greeno and Simon1993). Some call it a ‘structure of features’ (Klausmeier & Allen Reference Klausmeier and Allen1978), others a ‘schema’ (Rumelhart & Ortony Reference Rumelhart, Ortony, Anderson, Spiro and Montague1977; Rumelhart Reference Rumelhart, Spiro, Bruce and Brewer1980), and others still ‘associative networks’ (Bereiter Reference Bereiter1991). Anderson (Reference Anderson2004: 22) writes: ‘We can be sure that human cognition is achieved through large patterns of neural activity’. Such patterns or networks are not necessarily distinct entities. According to Gasser (Reference Gasser1990), the network structure of a concept is distributed over many units, each of which can also participate in the representation of many other concepts. Properties or characteristics of concepts consist of combinations of such units. For such combinations the term ‘features’ is often used. Features can be linguistic as well as non-linguistic. The identity of a concept consists of a distinctive set of features.
We find parallel views among cognitive psychologists concerning the way knowledge of such patterns of combinations is formed. Gasser (Reference Gasser1990) and Anderson (Reference Anderson2004), for example, suggest that they emerge by having been (repeatedly) processed in combination with each other in WM. According to this connectionist theory, our brain keeps track of the regularities in the occurrence of combinations and of the frequency of occurrence of these combinations in WM. The frequency determines the ‘weight’ of the established connections between the features. This ‘weight’ accounts for the ease of retrieval from LTM. On the basis of these insights, the Multi-Feature Hypothesis (MFH) predicts that retention and ease of activation are improved by mental activities involving many features from many different categories, manipulated in such a way that they occur in current combinations, in great frequency, and simultaneously. This led us to the design of a typology of tasks, distinguishing task types in such a way that it would provide information about the estimated amount, variety and frequency of features (linguistic and non-linguistic) apparently involved in the elicited mental action. In ‘ranking according to a given criterion’ (such as in ordering based on chronology), for example, only one feature has to be considered (point in time), whereas in ‘generalizing’ (such as deducing a rule from a series of cases), many features of different kinds have to be considered (in order to establish which ones have enough in common to be considered as a group under a single heading). Combinations of these actions may be possible in carrying out a specific task. Based on these considerations, seven mental actions were defined (see Table 2).
Mental actions in SLA with descriptions, explanations and illustrations (from Driessen et al. Reference Driessen, Westhoff, Haenen and Brekelmans2008).
We then developed a scaffolding instrument containing a set of theoretically-motivated questions and guidelines by which the users were guided to determine systematically the activity domain covered by a task and the type of activity elicited by the assignment (for a full description, see Driessen et al. Reference Driessen, Westhoff, Haenen and Brekelmans2008). Our assumption was that having to choose the task type would elicit the mental action of examining the assignment to be analyzed regarding variety of features to be processed. This in turn would yield awareness of, and deeper insight into, the role of feature variety in a mental learning action. We evaluated the instrument in the form of a systematically monitored trial using both quantitative and qualitative methods. The results show that, even after brief training, teachers reported that the use of the tool contributed to their competencies. Most teachers were satisfied with the new knowledge base they said they received. The data show that the teachers were not only able to work with the tool, but also reported that they gained knowledge and insight into which characteristics of tasks account for learning effects (Driessen Reference Driessen2003). However, it should be borne in mind that the outcomes of the evaluation also show the need for a quite extensive clarification of the concepts. This must include theoretical background information in concrete terms and practical illustrations. Furthermore, successful use requires some introductory training and guidance.
4. The search for reliability and validity
4.1 Variety and frequency of features
The tool we had developed appeared to be useful as a learning task for practitioners. However, although it provided insight about variety and frequency of features that are probably or potentially involved in the elicited mental action, these insights remained rather general and non-specific. Moreover, the fact that teachers who had completed a training in the use of this tool reported enhanced competence in estimating the learning impact of a task does not prove that they were right in their judgements. Even the fact that they agree in their judgements is no evidence for the actual effectiveness of the assessed tasks. Accordingly, we decided to investigate the possibility of designing an instrument that enables the user to predict the quality of a task in a more reliable and valid way. Again we took the MFH as the theoretical basis but used it in a more detailed way. First, according to connectionist theory a learning result consists of a network of features. Features in a neural network activate each other and activation of a network can start from any connected feature. From that it can be concluded that networks consisting of many different features from different (linguistic and non-linguistic) categories have more chance of becoming activated in diverse situations by manifold evocators. Hence, a variety of features seems to be one important characteristic of a learning activity. Secondly, retention and ease of activation depends on the ‘weight’ of a connection. ‘Heavy’ connections yield prominent features. The ‘weight’ of connections is determined by the frequency of their presence in WM. Combinations will be easier to retrieve from LTM the more frequently they are processed in WM; therefore, the frequency of being processed in WM is another important factor. Hence, we decided to get a closer look at the variety and frequency of features processed in WM. We tried to get a better grasp of the variety of features involved in a mental action by describing a number of linguistic and conceptual categories, and distinguished the following linguistic feature categories (for a full description, see Moonen Reference Moonen2008):
• Orthographical features, carrying information about different aspects of the written word form.
• Phonological features, containing information about the phonological structure.
• Syntactic features, dealing with ‘what part of speech it is and what grammatical patterns it can fit into’ (Nation Reference Nation2001:55).
• Morphological features, giving information about various alternative forms of words.
• Semantic features, containing information about the linking of the correct labels of the L1 and the L2, in either direction.
• Collocational features, containing information about what words frequently co-occur with the target item, at the sentence level, including unanalyzed chunks.
We also described a number of conceptual feature categories that can participate in the representation of a target item and that belong to general world knowledge or our personal knowledge of specific events.
• Pragmatic features, referring to what a concept can be used for or what can be achieved with it.
• Sensory features, relating to a wide range of sensory aspects of a target item, such as visual appearance, color, taste, texture and sound.
• Affective features, referring to personal emotional or evaluative associations, e.g., positive and negative emotions, feelings and personal experiences associated with a target item.
• Situational features, containing information about the general, non-linguistic context or environment in which a concept commonly or frequently occurs.
In order to be able to estimate whether such features from these categories are processed, we formulated a working definition of each feature category. The working definitions were tried out a few times until we had formulations that appeared to be satisfactorily distinctive. Processing morphological features, for example, was assumed to take place when inflections or derivations of a word had to be processed. For the a priori estimation whether processing of feature categories would probably be elicited by the task, we designed a task sheet in the form of a matrix in which for each task element the feature categories which will possibly be involved in the elicited mental action can be indicated (Moonen Reference Moonen2008). In this way the tool is also usable for (relatively small) composed tasks.
4.2 Design and procedure
In order to test the reliability and the validity of the instrument we designed two versions of a vocabulary learning task: a ‘rich version’, in which the learners were assumed to process a broad variety of features, and a ‘poor’ one, which was assumed to elicit only a limited number of features. In the former, the learners were asked individually to categorize a given set of German words into groups that can form the outline of a micro-story and subsequently, in pairs, to choose the best outline for such a story, work out an appropriate formulation for it, write it down in German and memorize it by rehearsing the story in dyads. In the ‘poor’ version the students were given the German words with their Dutch translations and asked to memorize them individually and to rehearse them afterwards in pairs. Both tasks were judged a priori by two raters with the help of the task sheets. Both percent agreement and interrater reliability (Cohen's kappa) were calculated. Using the labels attributed to the relative strength of agreement proposed by Landis & Koch (Reference Landis and Koch1977: 165), the kappa statistics of the orthographical, phonological, syntactic, morphological, collocational and situational feature categories were considered ‘almost perfect’. The kappa of the semantic and pragmatic features were considered ‘substantial’. The kappa of the sensory features were labelled ‘fair’ and the kappa of the affective features ‘slight’ (Moonen Reference Moonen2008). We believe that these results sufficiently confirm the instrument's reliability for practical use.
However, these results do not prove the superior learning impact of the task that was estimated to be ‘rich’. For this purpose we measured retention of the words presented in the analyzed tasks. For that purpose both tasks were carried out in intact classes of a Dutch secondary school (for the ‘rich’ task N = 46, for the ‘poor’ task N = 44). One week before the experiment took place, a pre-test was administered to measure prior knowledge of the vocabulary to be presented. Learning gains were measured by a translation test and a written description test in which the students were asked to describe situations in a cartoon. The translation test examined whether the learners were able to reproduce the target words in a situation which provided minimal contextual support and which was expected to make minimal demands on WM. The written description test was used to examine whether the learners were able to use the target words in a relatively rich context, which was expected to make considerable demands on WM capacity. Two days after the experiment, a direct post-test was administered, one week thereafter a delayed post-test. We administered an ANCOVA with the pre-test as covariate.
4.3 Outcomes
Results show that the ‘rich’-task group significantly outperformed the ‘poor’-task group (p < .05) on both test types and on both direct and delayed tests (Moonen Reference Moonen2008). On the written description test the differences appeared to be greater than on the translation test. This can be interpreted as further indication that the tasks that were rated ‘rich’ indeed yielded learning results that can be more easily activated in complex, life-like situations.
5. Some conclusions
Our research shows that it is possible and profitable to delineate a systematic way of assessing the impact of language learning tasks on a sound theoretical footing. Indeed, practitioners who are not experts in applied linguistics appear to be able to apply a tool based on principles derived from SLA theory and cognitive psychology. Furthermore, these results confirm our expectation that our instrument for assessing learning tasks a priori predicts the potential learning impact of such tasks in a reliable and valid way. However, such a reliable and valid in-depth analysis is cumbersome and time-consuming. In order to prove reliability and validity of the instrument as described in section 4, we had to restrict ourselves to relatively simple tasks (the acquisition of 16 words). It remains questionable whether such an instrument is very workable in everyday teaching practice. The ‘yardstick’, as described in section 2, may offer the best solution for practical use. Although in only scratches the surface of what happens in WM, it has a sound theoretical basis and allows for the assessment of differentiation in learning tasks. As was shown in our second experiment, this profit can be enhanced by incidentally performing a more in-depth analysis of smaller tasks with the help of the instruments described above, because it can be expected that this will contribute to the heightening of awareness in these matters. As a consequence it can be expected that the incidental performing of in-depth analyses would make the use of the ‘yardstick’ more trustworthy.
Assessing the effectiveness of language learning tasks appeared to require a certain, albeit elementary, knowledge about SLA and cognitive psychology. In our second experiment, in spite of the extensive and explicit wording of the criteria, its use required an introductory training session with a detailed clarification of the concepts used. Even in the restricted and very concrete and operationalized form of the instructions and criteria of our last experiment, one of the raters, a trainee teacher of L2 English needed a short supplementary training session before he could carry out the analyses. Given the fact that the significance of teachers' competence in estimating the learning impact of instructional materials is increasing, this finding underscores the importance of SLA theory and cognitive psychology instruction in language teacher education.
Gerard Westhoff is Professor of L2 pedagogy at Utrecht University (IVLOS Institute of Education) in the Netherlands. As a teacher educator he currently works mainly in research and in-service training. His research focuses on instructional design, more specifically on task design for L2 acquisition. He is director of the National Bureau on Modern Languages, created and financed by the Dutch Ministry of Education. The Bureau's mission is to improve the quality of modern language learning and teaching in the Netherlands.
