Lexical retrieval is an essential process in both oral and written language production and an important aspect of fluency. It involves the selection of lexical concepts, resulting in lemmas representing a word's syntax that are subsequently morphologically, phonologically, and phonetically encoded to be either articulated (Levelt, 1989; Levelt, Roelofs, & Meyer, 1998) or written down. In contrast to the closely related process of lexical access (“word recognition”), where forms have to be recognized in order to arrive at the correct lexical concept, in lexical retrieval concepts are transformed into linguistic forms. Although the order in which the components of the process are active is reversed in lexical access and lexical retrieval, most components are interchangeable (Kroll & Stewart, 1994; Levelt, Roelofs, & Meyer, 1999).

In their influential model of the writing process, Hayes and Flower (1980) distinguished lexical retrieval (“translating”) from other processes involved in text production, such as generating ideas, planning, organizing, and reviewing. Kellogg (1996) also stressed the importance of lexical retrieval in his discussion of the processes involved in written text production. In an experimental study, Kellogg (1994) showed that lexical retrieval is highly effortful, even for native speakers. Although planning and reviewing placed the greatest demands on cognitive resources, as measured by reaction time (RT) interference on a secondary task (an auditory probe), it turned out that translating content into sentences, which involves lexical retrieval and syntactic processing, was also highly effortful and slowed reaction times about 350 ms above baseline times. As cognitive resources are limited, it is possible that devoting too much attention to lexical retrieval could have adverse effects on the writing process (for a discussion of attention limits, see Kellogg, 1999; Lea & Levy, 1999). More attention being devoted to lexical retrieval may result in less attention available to planning, reviewing, and other cognitive operations necessary for writing (Hayes, 1996; Kellogg, 1996).

Within reading research, there is a well-established school of thought that cognitive resources devoted to one subprocess preclude attention being given to other important subprocesses, and that this may also be reflected in the quality of the process as a whole. Evidence has also been found for the influence of high resource demands on text comprehension (Favreau & Segalowitz, 1983; Just & Carpenter, 1992; Koda, 1996; Perfetti, 1985).

Within writing research, some studies have shown a relationship between resource demands and writing by experimentally increasing the cognitive load in writing tasks (for a review, see Alamargot & Chanquoy, 2001). Penningroth and Rosenberg (1995) showed that narrative writing with a higher information-processing load negatively affected ratings of coherence, but not of holistic quality. Lea and Levy (1999) demonstrated that a secondary task (i.e., detecting and signaling the visual alternation of letters and digits) negatively affected both writing fluency and writing quality. Scardamalia, Bereiter, and Goelman (1982), on the other hand, only found a relationship between resource demands and text length but not between resource demands and text quality. They showed that for Grades 4 and 6 children, neither slow rate of output nor interference from mechanical demands (e.g., spelling and punctuation) led to a decline in text quality. Nevertheless, in another experiment (cf. Scardamalia et al., 1982) they found that when children were forced to retain information in short-term memory, there was sometimes information loss between forecasts (i.e., the words they intended to say) and actual writing. This finding implies that a slow writing rate may be a disadvantage in that children may forget some of the words they are going to say. Finally, in a study focusing on lexical retrieval, Brown, McDonald, Brown, and Carr (1988) showed that when lexical material had to be retrieved from memory this process usurped attention from execution processes, resulting in reduced legibility and a greater number of errors.

Some correlational studies have also served to emphasize the importance of lexical retrieval for writing. McCutchen, Covill, Hoyne, and Mildes (1994) demonstrated that more proficient first language (L1) writers (across Grades 3, 4, 7, and 8) showed better performance on a reading and speaking span task than their less proficient counterparts. The more proficient writers had larger spans (i.e., they remembered more last words or were able to generate grammatical sentences for longer lists) and produced longer sentences. Ransdell and Levy (1996) found a similar relationship between a writing span measure and writing. In another experiment in their 1994 study, McCutchen et al. also found that the more proficient writers had shorter latencies and higher accuracy rates on lexical decision tasks. They followed Flower and Hayes (1980) in arguing that writers have to handle a number of constraints simultaneously. The possible strain that this constraint juggling places on attentional resources could mean that the more attention that is devoted to translating and lexical retrieval, the less that can be devoted to other processes such as planning, generating ideas, organizing, and reviewing (for a review, see also McCutchen, 1996). In our view, this would seem to imply that once writers can retrieve words fluently, they can devote attention to other processes, enabling them to write better texts.

Flower and Hayes (1980) suggested the possibility of providing training to increase the efficiency of sentence production processes in the L1, for example, through sentence-combining exercises, the ultimate aim of which would be to enable writers to concentrate more on other important writing processes. To date, no studies have been carried out that specifically investigate the effect of training sentence production processes on writing quality. However, a small number of studies have shown that reducing resource demands may facilitate writing. Although research in this direction has been scarce, this approach is promising because it may have important practical implications for education. Van Gelderen (1997) showed that error detection and revision activities at the macrolevel of the text could be considerably improved by facilitating the revision process of L1 writers (Grades 5–6) by such means as providing standard phrases for evaluation and reformulation and providing opportunities for improving someone else's text. Recently, Chenoweth and Hayes (2001) have shown in a small cross-sectional think-aloud protocol study that second language (L2) courses resulted in direct retrieval of the words needed in their texts and there was a (small) increase in the length of the sentence parts (“bursts”) that were proposed and evaluated in constructing sentences. In their view, linguistic experience led to increased lexical retrieval skills and increased facility with complex grammar, reducing resource demands and enabling writers to handle more complex and longer language structures. Finally, Glynn, Britton, Muth, and Dugan (1982) showed that in L1 persuasive writing, both preliminary drafts and final drafts improved when students (in a graduate course) were instructed not to pay attention to logical sequence or to structure demands (i.e., correct syntax and spelling) in their preliminary drafts. In particular, the students were able to use more arguments that argued for a certain position. Interestingly, a second experiment, involving undergraduates of average and low verbal ability (as measured on the Scholastic Aptitude Test), showed that when demands on preliminary draft production were absent, argument production only increased for writers with average verbal ability. Writers with low verbal ability did not profit from the relaxed demands, suggesting that positive effects on quality depend not only on cognitive demands but also on verbal ability.

Even though the importance of resource demands in writing has been stressed repeatedly, most of the evidence is either correlational or shows the negative effects of increased resource demands on writing. As mentioned, few studies have tried to improve writing by reducing resource demands. Moreover, there has been no research trying to improve L2 writing by enhancing lexical retrieval skills. When an L2 is involved, lexical retrieval is likely to be more attention demanding than in the L1 and therefore is also likely to have more negative effects on L2 writing quality. In addition, the study by Glynn et al. (1982) suggested that positive effects on argument production appear conditional on verbal ability of the writers. As lexical retrieval is an important aspect of L2 verbal ability, enhancing lexical retrieval in the L2 seems to be a promising approach. The current study addresses the lack of L2 research in this area by investigating whether experimentally enhancing lexical retrieval in the L2 has a positive effect on L2 writing quality.

In an earlier study, we demonstrated that L2 lexical retrieval could be enhanced through an experimental intervention (Snellings, Van Gelderen, & De Glopper, 2002). We showed that a 4-week, highly focused training aimed at speeding up lexical retrieval resulted in both higher levels of accuracy and faster RTs for Dutch learners of English (L2). The effect sizes in this study were medium to large. The study indicated that training learners to focus on speed and providing them with immediate feedback on both speed and correctness effectively altered and improved an essential cognitive subprocess involved in L2 production, thus potentially reducing the cognitive demands of lexical retrieval in L2 writing.

The current study, which uses the same participants, examines the effects of enhanced lexical retrieval on several aspects of written L2 narratives. In this study, we first investigate whether the lexical retrieval training results in more frequent use of the trained words in the writing tasks. Second, we investigate whether the training results in learners being better able to express the details of the story and the desired content. Third, we explore whether enhanced lexical retrieval has an effect on global communicative quality of the texts. Global text quality is the result of many different skills associated with written language production, such as audience awareness, text organizational skills, and sensitivity to style. Clearly, fluency of lexical retrieval is but one skill that may influence global text quality.

Providing answers to these research questions involved dealing with a number of important considerations that guided the development of the instruments and scoring procedures in the study. An important consideration was the need to have an appropriate measure for speed of lexical retrieval. In previous studies, picture naming tasks have often been used as a measure of lexical retrieval. Picture naming tasks are good measures for lexical retrieval because, just as in written language production, the process they tap into proceeds from meaning to word form. However, the set of words in which we were interested, which includes nouns, pronouns, articles, verbs, adverbs, adjectives and combinations of these categories, is difficult to depict unambiguously with pictures. Because a specific focus on this diverse set of words was needed, a word translation task was deemed to be more appropriate. Translation tasks and picture naming tasks tap into similar processes in that both kinds of task involve concept mediation (Potter, So, von Eckardt, & Feldman, 1984). For related word translation tasks, see De Groot, Dannenburg, and Van Hell (1994) and Potter et al. (1984).

In view of our aim to increase the retrieval fluency of familiar words, rather than to teach new words, another important consideration was the need to compile a set of familiar words to be used in the training. Word-frequency ratings may not be the best indication of familiarity, especially in view of the fact that there are no learner corpuses for the productive vocabulary of Dutch English as a foreign language (EFL) learners. Thus, in order to obtain a selection of familiar (productive) words, we conducted a pilot study in which 9th grade students had to write narrative texts using the cartoons to be used in the main study (see the Method section). In view of the relevance of collocations, for reducing cognitive effort (Weinert, 1995), we also selected some word combinations used in the narrative texts from the pilot study. Following Weinert (1995), we define word combinations, collocations, or formulaic language as multiword strings (e.g., “just in time”) that can be retrieved as a whole chunk and are similar to individual lexical items, rather than being derived by linguistic rules.

A third important consideration was the need to appropriately assess the three aspects of the narrative texts relevant to the research questions, namely, number of trained words used, expression of essential content elements, and global text quality. Assessing the number of trained words used in the narratives is a relatively straightforward procedure. For the rating of global text quality, there are agreed-upon procedures (Schoonen et al., 2003). Only for the assessment of content elements did a special approach need to be developed. To assess whether the details of the story and the desired content were expressed, we first defined “content elements,” using the definition of “narrative schemata” described by Stein and Trabasso (1982). According to these authors, schemata “specify the components of a story that should be included in a text, the temporal sequence of the narrative events, and the type of causal relations that should connect the events.” These schemata enable us to establish whether certain parts of a story have been omitted. As pointed out by Stein and Trabasso, the underlying representation of a story in memory can be inferred from a text. As a result, even though some parts of the story may be deleted from the text structure (e.g., “internal responses”), they should be inferable from the text. Stein and Trabasso divided stories into a Setting and an Episode. The latter consists of five categories that contain specific information and that serve a different function in the schema. These categories are psychological rather than linguistic entities. The five categories are Initiating event, Internal response, Attempt, Consequence, and Reaction. An Initiating event can be an action that causes the protagonist to respond emotionally and to formulate a goal. Internal response is the emotional reaction of the protagonist that causes him or her to initiate action. An Attempt is the overt action to attain a goal, a Consequence is the event indicating goal attainment, and a Reaction is the emotional response of the protagonist to the outcome of his or her actions (for the schema structures, see the section on scoring). By determining whether content belonged to one of these five categories, we were able to establish whether essential content was expressed.

METHOD

Participants

A total of 103 Dutch students participated in this study, all at the beginning of their third year of secondary education (Grade 9, age 14–15). Students were from four different classes, two classes from lower general secondary education (n=47) and two classes from preuniversity education (n=56). Students had received 2 full years of formal English tuition (two–three lessons per week) at secondary level, and a 2-year introductory course at the elementary level, focusing mainly on basic oral communication skills (maximum one lesson per week). Students filled in a questionnaire, which was completed by 101 students, concerning their gender and language background and containing questions such as whether they spoke Dutch with their parents and whether they had acquired Dutch as their L1. Of these students 44.6% were male and 55.4% were female, and the results showed that 90 students were native speakers of Dutch. The remaining 11 students were L2 speakers of Dutch whose native language was not English (although one student who had acquired Dutch as his L1 spoke English with his father and Dutch with his mother, and another student spoke Dutch with her parents but had acquired both English and Dutch as the L1). There was an even division of L2 speakers of Dutch across the two conditions.

Design

In a counterbalanced design, students from four classes were randomly assigned to one of two speed-training treatments. Each class was randomly split in half and the two resulting treatment groups were trained concurrently by two experimenters over a 4-week period. Each treatment used a different set of stimuli, either Word Set A or Word Set B. Students in the Trained A words condition were trained on Word Set A. Students in the Trained B words condition were trained on Word Set B. Students in each group were not trained on the other group's word set. After the training, all students were tested on speed and accuracy for both word sets (Snellings et al., 2002) and also wrote two narrative texts based on cartoons. Narrative A was likely to elicit the words in Word Set A, whereas Narrative B was likely to elicit the words in Word Set B.

Materials

Selection of training words

In order to select familiar (productive) words for the training, we conducted a pilot study in which ninth grade students had to write narratives using the same cartoons as in the current study. Words that were used at least once by the pilot study students were candidates for selection. Only candidate words that were likely to be commonly included in EFL textbooks in The Netherlands were selected. In this selection process, the first author was able to use his extensive experience in teaching English to Dutch students of this age group. Another criterion for selection was that the selected words had to have straightforward translations in Dutch. It was also necessary to ensure that for each picture there were sufficient words selected to describe the events. The selection included words belonging to different grammatical classes (e.g., single nouns, nouns with articles or determiners, verbs, adjectives, and adverbs) and we also selected some word combinations. The two sets of words are contained in Appendixes A and B.

Tasks and procedures

Prior to the training, a vocabulary test, to be used as a covariate, was administered. The students were subsequently trained in class for 4 weeks. To eliminate experimenter effects, each of the two trainers trained students in both conditions. All students received the same type of computer training and the same number of computer assisted exercises (but with different words). The training was given using specially designed software¹

This software was a joint development of the current study and a reading intervention study by Annegien Simis and Jan Hulstijn. Although the exercises in both studies differed, the framework was identical and was programmed by Boudewijn de Roovere. For a more detailed description, click the NELSON tab on the following site: http://members.ams.chello.nl/b.roovere/.

that could be used outside the laboratory in a classroom setting, allowing a whole group of learners to be trained at the same time.

As our objective was to increase retrieval speed, students were actively encouraged to respond more quickly. The program provided them with a baseline against which they could chart their progress (i.e., previous RT on the same word). As the words were largely familiar, we assumed that working under time pressure would be a motivating factor as well, so that two instances of the translation exercise used the flashcard principle. The students encountered each word 10 times and received immediate feedback on correctness and current speed.

We consistently trained the same meaning of the words in order to ensure that mappings of stimuli to responses were consistent (Schneider & Shiffrin, 1977; Segalowitz & Gatbonton, 1995). For example, if the aim is to train the word “line” in a story about fishing, the word should always be used in the fishing sense and not in both the fishing line sense and the telephone line sense.

Students were trained in scheduled English classes. We used spaced presentations (in contrast to massed presentations), because psychological learning theories suggest that this is more effective (see discussion in Ellis, 1995). The standard duration of each training session was 50 min (except for one 45-min training session). There was a 1-week interval between each training session. After completion of the training sessions, we collected data on lexical decision skills and lexical retrieval skills in order to establish whether lexical retrieval had been enhanced. Within the next week, students completed the posttest writing tasks. All students first completed the writing task that corresponded to their training condition. Each word was trained using four different types of exercises, moving from receptive skills to productive skills. The exercises are described below.

Block exercise

The aim of this exercise was both to strengthen the receptive word knowledge and to train syntagmatic knowledge (i.e., knowledge of collocational restrictions). Students had to complete a short sentence as quickly as possible by choosing from two constituents (i.e., two “blocks”), the choice of one of which resulted in an incorrect word order.

• Example
• After some time
• woke up she
• Z M

Students were required to press either the Z or M key in order to indicate the constituent they thought should follow After some time.

Detection exercise

This exercise had a strong focus on the meaning of the words involved, in order to strengthen the connections between concepts and lexical forms. Students had to decide whether the meaning of the words and expressions was appropriate to the specific contexts provided. They had to focus on the meaning of the words in short sentences and detect semantic anomalies as fast as possible by indicating which word caused the anomaly.

• Example
• Three hunters (Z) had to be milked soon (M).

Students had to indicate whether hunters (Z) or soon (M) makes the sentence odd. Here, the word hunters makes the sentence odd because a word like cows is more likely.

Correction exercise

In the correction exercise, students were asked to assess words for appropriate use and, if necessary, to choose a better substitute from two alternatives. Like the detection exercise, this exercise also aimed at enhancing the connections between concepts and lexical forms. Students again had to focus on the meaning of words in short sentences. They also had to detect semantic anomalies as fast as possible, but this time they had to indicate which of two words would make the sentence logical (or accept the original word marked X).

• Example
• Sue felt very ill. After some ages (X) she felt better though.
• time centuries
• Z M

Students had to indicate whether ages (X) is semantically acceptable, or whether it should be replaced with time (Z) or centuries (M). In the example, time (Z) is the more acceptable alternative.

Translation exercise

In the word translation exercise, students provide translations for words presented in a meaningful context. Of the four exercise types, the translation exercise most closely matches lexical retrieval in written language production. The use of a translation task makes it possible to focus on more words at the same time (in contrast to picture tasks).

Students translated Dutch words presented in simple English carrier sentences. Both responses that provided the targeted training words and responses that offered acceptable (spelling) alternatives for the same concepts were scored as correct. In addition to the ordinary format, we developed a flashcard version in which the Dutch words and the English sentences disappeared from the screen after 15 s, thus encouraging the students to answer more quickly.

• Example
• voordat the bridge collapsed he had reached the other side
• Correct translation: before

The order of the items in all the exercises was randomized for each presentation. In the block exercise, the order of the two constituents was also randomized. The exercises were tested in a small-scale pilot study in order to test the functioning of both software and hardware and also to gauge the exercises' effectiveness. We used information gained in this pilot study to make the instructions clearer and to place more emphasis on speed. Problems with the program and the design of the exercises were also solved. In the final experiment, students began with the block exercises, followed by the correction, detection, and translation exercises.

Dependent measures

After we trained lexical retrieval and established an increase in retrieval speed, students completed two cartoon-based writing tasks (Writing Tasks A and B). The restricted range of vocabulary elicited by the cartoon-based tasks provided a direct way of testing the effects of enhanced lexical retrieval on writing. Writing Task A was about two boys who go fishing, and Writing Task B was about a man who goes walking in the mountains and encounters a tiger (see Figures 1 and 2). In both tasks, the pictures were presented in chronological order. The dependent variables, which are “Number of trained words used,” “Expression of content elements,” and “Global text quality,” are measured for both Writing Task A and Writing Task B.

The cartoon used for Writing Task A. Note. From Composition through pictures, by J. B. Heaton, 1981, Harlow, UK: Longman. Copyright 1966 by Longman. Reprinted with permission. Illustrations by James Moss.

The cartoon used for Writing Task B. Note. From Composition through pictures, by J. B. Heaton, 1981, Harlow, UK: Longman. Copyright 1966 by Longman. Reprinted with permission. Illustrations by James Moss.

Number of trained words used

We counted words appearing in the students' texts that were trained either in isolation or as part of a word combination. In addition, we looked at the occurrence of trained word combinations. Because our aim was to look into the effects of more efficient lexical retrieval and reduced resource demands, we only considered the use of words and word combinations that were trained literally.

Expression of content elements

On the basis of the cartoons, the first author devised the schema structures that represented the schema of each narrative. These narrative schemata assisted in identifying content elements, which were subsequently placed on a structured checklist (see Appendixes C and D). Before scoring, students' texts were divided into propositions. Scorers indicated whether each proposition contained an essential content element, added other information, or added irrelevant information. To facilitate scorer agreement, we constructed a set of guidelines and trained the scorers according to these. Decisions made while scoring the texts used in the training session were made explicit and written down, to be used as an extra guideline during scoring. An example of a guideline is that whenever a content element was not linguistically realized but was implied by other words in the text, it was also scored (e.g., “the owner warns the man” implies “there is an owner”). The structured checklist ensured that only content elements that occurred in the appropriate part of the story were scored.

Global text quality

The writing tasks instructed students to use the appropriate register for addressing their peers, to do their best to interest the reader, and to write in such a way as to produce a story that would be coherent in the absence of the cartoons.

Two independent raters scored the texts for global quality. The raters were instructed to give a general rating for the quality of the text in relation to five benchmark texts, using any numerical value they thought appropriate. The raters were able to consult the benchmark texts while scoring. The five benchmark texts had been selected on the basis of a separate analysis according to a scaling procedure (Blok, 1986), involving six raters. The benchmarks represented a very weak text (10th percentile), a weak text (25th percentile), an average text (50th percentile=score of 100), a good text (75th percentile) and a very good text (90th percentile). Raters were instructed to concentrate on content and organization and to consider language errors only to the extent that they interfered with the understanding of the story.

Covariate

To control for initial differences in vocabulary knowledge, we used the scores on the vocabulary test as a covariate. The vocabulary test (an adapted version of a test used in another NELSON project study; see Schoonen et al., 2002) consisted of multiple-choice items in English. Each item contained a neutral carrier sentence with the target word in bold print, and students had to choose between four alternative Dutch translations of the target word. The words were selected from the 5000 most frequent words in the COBUILD corpus (COBUILD, 2000). On the basis of a pretest, we removed items that were too easy (p>.90) to obtain a more discriminative test. In addition, we removed items with negative item–total correlations. The final test contained 64 items. Scores on the vocabulary test correlated significantly (p<.05, one tailed) with all three dependent variables, making the test suitable for use as a covariate. In the case of Writing Task A, r=.24 with Number of trained words used, r=.19 with Expression of content elements, and r=.23 with Global text quality. In the case of writing assignment B, r=.21 with Number of trained words used, r=.40 with Expression of content elements, and r=.38 with Global text quality.

Data collection

Both the vocabulary test and the writing assignments were paper and pencil tests. The class teacher administered the vocabulary test before the start of the training. Following the training period, a trained research assistant presented the instructions for the writing tasks.

Analyses

Means, standard deviations, and estimates of rater reliability were calculated (i.e., percentage of rater agreement for Use of content elements and Cronbach's α for the Global text quality score). The hypotheses were tested by two one-way multiple analyses of covariance (MANCOVAs) on the measures we derived from the writing tasks, with Condition (A or B) as an independent measure; Number of trained words used, Expression of content elements, and Global text quality as dependent measures; and Vocabulary knowledge as a covariate.

In the case of the MANCOVAs on the writing tasks, results of evaluation of the assumptions of independent observations, equal population covariance matrices for the dependent variables, a linear relationship between the dependent variables and the covariate, homogeneity of the regression slopes, and reliability of the covariate were satisfactory (Stevens, 1996). The only exception was the assumption of multivariate normality of dependent variables in each group for the MANCOVA on the Writing Task B measure. In the case of Expression of content elements in the Trained B words condition, the Kolmogorov–Smirnov test was significant, indicating that multivariate normality did not hold here. However, because this is the only group and dependent variable for which this assumption was not valid, the effect on power is small (Stevens, 1996, p. 247). In addition, MANCOVA is rather robust regarding a violation of multivariate normality.

Scoring

In the case of Number of trained words used, we used the summed scores of the number of isolated words, the number of words used in isolation that were trained as part of a word combination, and the number of the word combinations. We did not include derivations or inflected forms. If words were used more than once, each token was included in the total score.

In the case of Expression of content elements, after establishing rater reliability, a single trained scorer scored all the texts. Rater reliability was established by comparing the percentage of agreement between the scorer and the first author on a sample of 20 texts from each writing task. The percentage of agreement was 89.9% for Writing Task A and 87.4% for Writing Task B.

Finally, in the case of Global text quality, two raters scored each text independently. Panel reliability (Cronbach's α) was .84 for Writing Task A and .88 for Writing Task B.

Missing values

Only students who had been present at all the lexical retrieval training sessions were included in the writing task analyses for the trained tasks (Writing Task A, N=41; Writing Task B, N=42). Presence at training sessions was not required for inclusion of data from the tasks involving untrained words (i.e., Writing Task B for students trained on word set A, and vice versa, so that Writing Task A, N=52, Writing Task B, N=50). As a result, the analysis of Writing Task A was based on scores from 93 students and the analysis of Writing Task B on 92 students.

RESULTS

In this section we first present descriptive measures for the tests used in this study. Then, we discuss the effects of the lexical retrieval training on our dependent measures for both Writing Tasks A and B.

Descriptive and psychometric statistics

Dependent measures

As explained in the Missing Values section, the number of students in each condition differs, depending on which text is analyzed (see Table 1). When examining these data, it has to be taken into account that comparison of scores for Writing Tasks A and B is not warranted, because the writing tasks are not equivalent. Only comparisons between groups on the same writing measure are justified.

In Table 1 we give an overview of the unadjusted scores for the Writing Task A and B measures. These unadjusted scores give a first impression of the differences on the dependent variables that exist between the two experimental groups, but they cannot be used to evaluate experimental effects. These effects are analyzed in the following sections.

Covariate

The mean score for the vocabulary test was 45.19 (N=103, maximum possible score=64, SD=8.74) and the reliability was .87. Thus, on average the vocabulary test was of intermediate difficulty and its internal consistency was good.

Experimental effects

We expected superior scores for students in the trained words condition on the writing task measures. Our multivariate test (Wilks' lambda) for the Writing Task A measures was significant with F (3, 88)=18.105, p<.001, η²=.382. Our multivariate test (Wilks' lambda) for the Writing Task B measures was also significant with F (3, 87)=14.387, p<.001, η²=.332. All reported means are derived from a MANCOVA and therefore adjusted for vocabulary knowledge.

Number of trained words used

In the case of number of trained words used in narrative A, our follow-up univariate analyses show a significant main effect for condition with F (1, 90)=32.032, p<.001, partial η²=.262. As expected, students in the Trained A words condition (M=40.83, SE=1.56) used more A words than students who were not trained on these words (M=29.04, SE=1.38). We expected similar results for narrative B, with number of B words used higher for students in the Trained B words condition. Indeed, once again, our follow-up univariate analyses show a significant main effect for condition with F (1, 89)=27.574, p<.001, partial η²=.237. Students in the Trained B words condition (M=31.32, SE=1.31) used more B words than students who were not trained on these words (M=21.96, SE=1.20). Effect sizes can be transformed into Cohen's f (Cohen, 1988). A small effect is an f value of .10 and this can be transformed into η²=0.0099 (the transformation uses η² instead of partial η² which SPSS reports); a medium effect is an f value of .25, which equals η²=0.0588; and a large effect is an f value of .40, equaling η²=0.1379. In the case of both the A and B words, the effect size was large, with respective values of 25% (η²=0.2469) and 23% (η²=0.2262) of the variance in scores due to the training.

Expression of content elements

In the case of Expressed content elements we also expected higher scores for the trained students. For Expressed content elements A, our follow-up univariate analyses show a significant main effect for condition with F (1, 90)=5.567, p<.05, partial η²=.058. Students in the Trained A words condition obtained a higher score on Expressed content elements A (M=25.30, SE=.85) than students who were not trained (M=22.61, SE=.76). A total of almost 6% (η²=0.0562) of the variance in scores, close to a medium effect in Cohen's terms, can be accounted for by the training. Although students in the Trained B words condition obtained a slightly higher score on Expressed content elements B (M=17.21, SE=.56) than students who were not trained (M=16.05, SE=.52), this difference is not significant with F (1, 89)=2.307, p=.13, partial η²=.025.

Global text quality

Results of our univariate follow-up analyses show no significant effect of the training on Global text quality for narrative A with F (1, 90)=2.253, p=.14, partial η²=.024. Nevertheless, adjusted scores are in the expected direction, with students in the Trained A words condition scoring higher on the Global text quality score A (M=100.39, SE=1.79) than students who were not trained (M=96.80, SE=1.58). For narrative B, results from the univariate follow-up analyses also show no significant effect on Global text quality with F (1, 89)=0.042, p=.84, partial η²=.000. The mean Global text quality score for narrative B of students in the Trained B words condition (M=95.64, SE=1.63) was nearly identical to the mean score of students who were not trained (M=95.19, SE=1.49). Because the interaction between the covariate vocabulary and condition (homogeneous regression slopes assumption) was borderline significant (p=0.05) for the dependent variable of Global text quality, we also carried out an analysis without a covariate. However, the results of this analysis were also nonsignificant.

APPENDIX A

APPENDIX B