1. Introduction
An important textual distinction in the field of second language (L2) reading is made between authentic and simplified texts (Crossley, Louwerse, McCarthy & McNamara Reference Crossley, Louwerse, McCarthy and McNamara2007). While the linguistic differences between these reading texts might seem subtle, the theoretical impact that they have made is quite sizeable. At present, there is no concurrence within the field as to the true value of these texts and whether language learners are better served by one type over the other (Day & Bamford Reference Day and Bamford1998). Additionally, while authentic text is theoretically more appealing (Swaffar Reference Swaffar1985; Bacon & Finnemann Reference Bacon and Finnemann1990; Tomlinson, Dat, Masuhara & Rubdy Reference Tomlinson, Dat, Masuhara and Rubdy2001), the majority of L2 reading texts at the beginner and intermediate levels use simplified input and many specialists highlight the practical value of simplified text (Shook Reference Shook1997; Young Reference Young1999). Theoretical positions supporting either text type are weakened because few studies have investigated the linguistic properties of L2 reading texts, but researchers often depend on simplified or authentic texts' linguistic features, syntax, and discourse structures to justify their use in language learning.
This research gap was partially filled when Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007) conducted an exploratory analysis of the linguistic features found in simplified and authentic texts using the computational tool, Coh-Metrix (Graesser, McNamara, Louwerse & Cai Reference Graesser, McNamara, Louwerse and Cai2004). Their study investigated differences in the linguistic structures of beginner level, simplified and authentic reading texts to provide a better understanding of what linguistic features actually comprised these reading texts. The results demonstrated that while many of the theoretical assumptions about the differences between simplified and authentic texts were accurate, many were not.
As observed by Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), a weakness of their preliminary investigation was the limited size and lack of specificity in the corpus they used. Therefore, the purpose of the present study is to approximately replicate the earlier study using a larger and more specific corpus that takes into consideration not only learner level, but also the language skill being taught. Additionally, the present study will examine how differences between simplified and authentic texts support or do not support assumptions that L2 reading theorists have held about the value of these texts. Lastly, the present study will demonstrate the practical potential of the findings for L2 learning and material development. Because the Crossley et al. paper is a relatively recent study, there has been little critical research in the field since it was published. Thus, this paper will depend on many of the same theoretical premises that they introduced and much of their background information is briefly re-presented below. Readers interested in more than a cursory overview of the problem should refer back to the original Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007) study.
1.1 Simplified text
In the field of L2 reading, the simplification of text is common. So common that publishing houses and editors provide material developers with linguistic guidelines to assist in the construction of text (Simensen Reference Simensen1987; Young Reference Young1999). The motivation behind simplification is to make text more comprehensible for L2 learners and to help prepare those learners for more advanced, authentic text (Young Reference Young1999). It is thought to be beneficial because it excludes unnecessary and distracting, idiosyncratic styles without suffering a loss of valuable communication features and concepts (Allen & Widdowson Reference Allen, Widdowson, Brumfit and Johnson1979) and because it contains increased redundancy and amplified explanation (Kuo Reference Kuo1993). However, simplified text is also widely criticized for removing authentic language in favor of more simplified and frequent forms, thus denying learners the opportunity to learn natural forms of language (Long & Ross Reference Long, Ross and Tickoo1993).
1.2 Authentic text
Authentic text is text written to fulfill a social purpose for native speakers within a language community (Lee Reference Lee1995). Authentic text is thought valuable because it introduces students to natural and contextualized language (Larsen-Freeman Reference Larsen-Freeman2002) and because of its use of authentic linguistic features, especially cohesive devices, which are vital for the development of reading comprehension skills and information processing skills (Cowan Reference Cowan1976; Mackay Reference Mackay, Barkman and Jordan1979; Graesser, McNamara & Louwerse Reference Graesser, McNamara, Louwerse, Sweet and Snow2003). Authentic text is also favored because the natural lexical redundancy it is thought to contain is advantageous for reconstructing text and understanding unfamiliar lexicon (Goodman Reference Goodman1986; Johnson Reference Johnson1981). However, authentic text is also criticized for its linguistic difficulty, specifically its lexical and syntactic complexity and its conceptual and cultural density (McLaughlin, Rossman & McLeod Reference McLaughlin, Rossman and McLeod1983; McLaughlin Reference McLaughlin1987; Shook Reference Shook1997; Young Reference Young1999).
1.3 Linguistic analysis of text types
Overall, authentic text is assumed to provide more natural language and more naturally occurring cohesion than simplified text, while simplified text is criticized as creating discourse that is unnatural and serves to reduce helpful redundancy, thus increasing text readability (Crandall Reference Crandall and Byrd1995). Simplified text, however, is thought to benefit L2 learners because it is lexically, syntactically, and rhetorically less difficult than authentic text. Until recently, though, none of these theories had been empirically tested for their validity based on the actual linguistic features of the text. Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007) addressed this research gap when they used the computational tool Coh-Metrix to analyze a general corpus of 105 texts that were taken from seven beginner L2 learning textbooks. The results of this study, while only preliminary, suggested that simplified texts provide L2 learners with more co-referential cohesion and relies on more frequent and familiar words than authentic texts. The results also indicated that simplified texts demonstrated less diversity in their part-of-speech tags, displayed less causality, depended less on complex logical operators, and were more syntactically complex than authentic texts. Lastly, the results suggested that no significant differences existed between simplified and authentic texts in reference to the polysemy and hypernymy values of the words in the texts (see Table 1 for an overview). The results of this initial study provided an important contribution to fields in L2 teaching and learning, not only because they explained how simplified and authentic texts differed in their linguistic construction and how these differences benefited or hindered L2 acquisition, but also because they demonstrated how computational tools could be of assistance to L2 reading researchers, material developers and publishers of L2 teaching materials. However, the authors recognized the limitations of the study and called for further research into the differences between simplified and authentic texts, particularly with regard to larger and more specialized corpora and studies that considered different types of learners, learner genres, and learner levels. This paper attempts to address these earlier limitations by approximately replicating the methods of the Crossley et al. study with an augmented corpus.
Findings of Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007): predicted versus result.
*Finding was significant at p < .05 **Finding was significant at p < .001
1.4 Coh-Metrix
As noted by Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), Coh-Metrix (Graesser et al. Reference Graesser, McNamara, Louwerse and Cai2004) connects recent developments in computational linguistics and discourse processing to allow for the investigation of text difficulty and comprehension. Coh-Metrix features advanced syntactic parsers, part-of-speech taggers, shallow semantic interpreters, distributional models, and psycholinguistic databases. These components are integrated into an automated tool that can generate over 400 indices of language difficulty, readability, and cohesion. Many of the variables reported by Coh-Metrix closely match the linguistic features used to support both authentic and simplified text.
2. Predictions
Based on the close correlation between Coh-Metrix indices, research carried out within the field of material development, and the Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007) study, seven sets of metrics from Coh-Metrix were selected for this analysis. As with the Crossley et al. study, the chosen metrics were selected on the basis of the linguistic methods used in the simplification process including the deletion of low-frequency vocabulary words, the revision of complex syntax, the shortening of text length, plot simplification, the simplification of difficult phrases (Cripwell & Foley Reference Cripwell and Foley1984; Young Reference Young1999), the control of information, the control of language, and the control of discourse (Simensen Reference Simensen1987). These linguistic features are thought to most influence the text simplification process and are prominent in both the scholarly literature and publisher guidelines. The Coh-Metrix metrics that correspond to these linguistic features include causal cohesion, connectives and logical operators, co-reference measures, density of major parts of speech measures, polysemy and hypernymy measures, syntactic complexity, and word information and frequency measures. Based on these metrics, predictions were made prior to the actual analysis of the corpus in this study. These predictions were based on expected differences between authentic and simplified text.
2.1 Causal cohesion
Causal cohesion is relevant to texts that depend on causal relations between events and actions. Additionally, it demonstrates causal relationships between simple clauses at the sentential level (Pearson Reference Pearson1974–75). It is measured in Coh-Metrix by calculating the ratio of causal verbs (make, cause) to causal particles (consequently, as a result). For this study, causality was predicted to be higher in authentic texts based on the accounts of Simensen (Reference Simensen1987), who reported that publishing houses suggest that material writers simplify plot structure. This simplification should result in a lower ratio of causal verbs and particles and mirror the findings of Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007).
2.2 Connectives
Connectives play an important role in the creation of cohesive links between ideas. They are measured by Coh-Metrix in two dimensions: positive versus negative connectives and additive, temporal, and causal connectives. Logical operators are also measured in Coh-Metrix and include variants of ‘or’, ‘and’, ‘not’, and ‘if-then’ combinations, all of which have been shown to relate directly to the density and abstractness of a text and correlate to higher demands on working memory (Costerman & Fayol Reference Costerman and Fayol1997). In the present study, it was predicted that some differences would be found in the incidence of overall connectives (all the connectives measured by Coh-Metrix conflated into one index) and logical operators based both on the idea that publishers' guidelines call for lexical control and the careful use of connectives in L2 reading text (Simensen Reference Simensen1987) and from the findings of Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), which indicated that authentic texts were likely to have higher incidences of some connectives and logical operators.
2.3 Co-reference
Lexical co-reference has been shown to aid in text comprehension and reading speed (Kintsch & van Dijk Reference Kintsch and van Dijk1978). In Coh-Metrix, it is measured in three forms: noun overlap between sentences, argument overlap between sentences, and stem overlap between sentences. Coh-Metrix also measures semantic similarity between parts of text using latent semantic analysis (LSA) at the text, paragraph, and sentence level. Unlike lexical markers of co-referentiality (noun, argument, and stem overlap), LSA provides for the tracking of words that are semantically similar, but may not be related morphologically (Deerwester et al. Reference Deerwester, Dumais, Furnas, Landauer and Harshman1990; Landauer & Dumais Reference Landauer and Dumais1997; Landauer, Foltz & Laham Reference Landauer, Foltz and Laham1998). In this study, it was predicted that co-referentiality in the form of both LSA scores and co-reference scores would be greater in simplified texts as they are created with considerations for increased clarification and elaboration (Young Reference Young1999) and publisher guidelines urge writers of simplified texts to take great care with pronominal reference (Simensen Reference Simensen1987). This prediction was also supported through the findings of the Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), who found that simplified texts were more likely to demonstrate greater co-referentiality than authentic texts.
2.4 Part-of-speech density
Material developers and writers often want to know how frequently particular parts of speech (POS) occur in the text in order to control for infrequent parts of speech (e.g., predeterminers, superlative adverbs, wh-determiners). Coh-Metrix tallies incidence scores for specific classes of POS as defined by the Brill (Reference Brill1995) POS tagger. In the present study it was predicted that authentic texts would demonstrate greater part-of-speech density in less frequent lexical categories as a result of publishers' guidelines urging lexical constraints with regard to part-of-speech types used in simplified texts (Simensen Reference Simensen1987). This prediction was also supported through the findings of Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), who found that authentic texts was more likely to include less common parts of speech than simplified texts.
2.5 Syntactic complexity
Syntactic complexity is measured by Coh-Metrix in four major ways. The first index calculates the mean number of modifiers per noun phrase. The second and third indices measure the mean number of high-level constituents, defined as sentences and embedded sentence constituents, per word and per noun phrase. The fourth index tracks the ratio of words classified as pronouns to the incidence of noun phrases in the text. The present study predicted that simplified texts, if modified for sentence length, might exhibit more syntactic complexity because they would depend on shorter sentences with reduced language features and specified grammatical constructions (Cripwell & Foley Reference Cripwell and Foley1984; Long & Ross Reference Long, Ross and Tickoo1993). Additionally, if simplified texts at the intermediate level depended on basic parts of speech at the expense of less frequent parts of speech (Cripwell & Foley Reference Cripwell and Foley1984; Long & Ross Reference Long, Ross and Tickoo1993), there might be an increase in the number of modifiers per noun phrase and number of words before the main verb of the main clause, thus increasing the syntactic complexity of the texts.
2.6 Polysemy and hypernymy values
Coh-Metrix tracks the ambiguity and abstractness of a text by calculating the polysemy values, which refers to the number of senses a word has, and hypernymy values, which refers to the number of levels a word has in a conceptual, taxonomic hierarchy. The number of senses and the number of levels attributed to a word are measured using WordNet (Miller et al. Reference Miller, Beckwith, Fellbaum, Gross and Miller1990; Fellbaum Reference Fellbaum1998). Polysemy and hypernymy values are important as the process of simplification consists of trimming words and phrases, deleting low-frequency vocabulary words (Young Reference Young1999), and controlling for ambiguity and vocabulary (Simensen Reference Simensen1987). One possible shortcoming of simplified texts is that lexical simplification could lead to a reliance on more common words in the texts. Because more common words in English are more likely to have multiple meanings than less common words (Zipf Reference Zipf1949; Davies & Widdowson Reference Davies, Widdowson, Allen and Corder1974), it has been argued that simplified texts would exhibit higher degrees of polysemy and hypernymy. This was not, however, supported by Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), who found no differences.
2.7 Word information
Coh-Metrix calculates word information on five indices: familiarity, concreteness, imagability, meaningfulness, and age of acquisition. These measures come from the MRC (Medical Research Council) Psycholinguistic Database (Coltheart Reference Coltheart1981) and are based on the works of Paivio (Reference Paivio1965), Toglia & Battig (Reference Toglia and Battig1978) and Gilhooly & Logie (Reference Gilhooly and Logie1980), who used human subjects to rate large collections of words for psychological properties. Coh-Metrix measures word frequency using CELEX (Baayen, Piepenbrock & Gulikers Reference Baayen, Piepenbrock and Gulikers1995), which contains word frequencies taken from the 1991 version of the COBUILD corpus, a 17.9 million word corpus. Word frequency is important because frequent words are normally read more rapidly and understood better than infrequent words (Just & Carpenter Reference Just and Carpenter1980; Haberlandt & Graesser Reference Haberlandt and Graesser1985). In this study, it was predicted that simplified texts would display a greater degree of word familiarity, showing lower CELEX word frequency scores. This is because the simplification process consists of trimming words and phrases and deleting low-frequency vocabulary words (Cripwell & Foley Reference Cripwell and Foley1984; Young Reference Young1999). In light of the findings reported by Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), no significant differences were expected in the categories of concreteness, meaningfulness, age of acquisition, and imagability.
3. Method: text selection
A total of 224 texts were taken from 11 intermediate L2 reading textbooks. Because this analysis is intended as a more discriminating one than that used by Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), a more specialized corpus was produced that reflected a greater restricted audience and purpose. The corpus used by Crossley et al. was a general corpus and not well defined in terms of age level of audience and language skill being taught, and only contained a small, unbalanced sampling of authentic texts. Their corpus was constrained by the realities of beginner L2 reading textbooks, which rarely include authentic reading texts. This forced their analysis to unevenly examine many more simplified texts than authentic texts. Additionally, the lack of access to authentic beginner texts did not allow them to consider age level, with their corpus containing texts marketed for both children and adults, nor the language skill with their corpus containing texts taken from grammar books, reading books, and writing books. These restrictions limited the extendibility of their findings and forced the study to be exploratory in nature. In contrast, the present analysis was designed to benefit from a much larger corpus that was more narrowly angled and used more rigorous selection criteria including ability level, age level, and language skill. To correct for the restrictions found in Crossley et al., a larger sampling of texts was selected. Additionally, all texts for this analysis were taken from intermediate reading textbooks that were marketed for adult learners in either an ESL or EFL (English as a Foreign Language) environment. While the texts were all from reading textbooks, the genres varied. Common genres found in both the simplified and authentic reading textbooks included history texts, inspirational texts, social science texts, informational texts, biographies, and fiction. As in the earlier study, all reading texts in the selected readers that contained approximately 100 words or more were included in the analysis.
Of the 11 selected textbooks, four contained authentic texts (Fellag Reference Fellag2000; Pimsleur Reference Pimsleur1995; Ryall Reference Ryall2000; Sokolik Reference Sokolik1999), while the others included simplified texts (Collins Reference Collins2004, Reference Collins2005; Malarcher Reference Malarcher2004a, Reference Malarcherb; Pickett Reference Pickett1991; Smith & Mare Reference Smith and Mare2004; Zukowski/Faust Reference Zukowski/Faust2002). There was no overlap between the texts. The total size of the two corpora was 128,294 words. The simplified corpus contained 57, 961 words (M = 471, SD = 222) and comprised 123 texts. The authentic corpus contained 70,333 words (M = 696, SD = 555) and contained 101 texts. Descriptive data comparing the corpus used in this study to the corpus used in Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007) are presented in Table 2. Text size was not considered a factor, particularly because Coh-Metrix either normalizes its findings based on text length or provides normalized ratio scores.
Descriptive corpus comparison.
4. Results
4.1 Statistical analysis
This study follows the statistical methodology of Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007) in that it uses t-tests to distinguish between differences in text types.Footnote 1 As discussed by Crossley et al., multiple statistical analyses increase the probability of Type 1 errors. However, like Crossley et al., the analyses in this study were theoretically motivated and predictions were made a-priori (see ‘Predictions’ section for more information). Thus, confidence levels for the statistical analyses were not lowered.
4.2 Causal cohesion
As predicted and found by Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), there was a trend for authentic texts to contain more causal particles and verbs (see Table 3). However, this difference was not statistically significant.
Comparison of authentic and simplified texts for causal particles and verbs: means (M), standard deviations (SD), t-value, and p-value.
4.3 Connectives
The results for connectives were mixed (see Table 4). As predicted, the overall incidence of connectives did not differ significantly between authentic and simplified texts. However, in contrast to Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), no significant differences were found in more specific categories of connectives such as positive causal connectives, negative additive connectives, and negative temporal connectives.
Comparison of authentic and simplified texts for connectives: means (M), standard deviations (SD), t-values, and p-values.
4.4 Density of logical operators
Contrary to the findings of Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007) and the prediction that no difference would be found in the use of logical connectors, authentic texts showed a significantly greater density of all logical connectors (see Table 5). When comparing specific instances of logical operators, authentic texts were shown to have a greater density of ors as compared to simplified texts, but this only approached significance. Simplified texts, in contrast, were shown to have a significantly greater density of noun phrases. No significant differences were noted in the other categories of logical operators.
Comparison of authentic and simplified texts for logical operators: means (M), standard deviations (SD), t-values, and p-values.
4.5 Lexical co-reference
As predicted and in line with the findings of Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), when comparing simplified and authentic texts, significant differences were found in overall co-referentiality (see Table 6), with simplified texts exhibiting greater co-referentiality than authentic texts. In two of the three primary categories of co-reference cohesion, simplified texts displayed significantly greater co-referentiality: stem overlap and noun overlap. Simplified texts also showed a trend toward greater argument overlap than authentic texts, but this finding only approached significance.
Comparison of authentic and simplified texts for lexical co-referentiality: means (M), standard deviations (SD), t-values, and p-values.
4.6 Latent semantic analysis (LSA)
As predicted and found by Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), significant differences were noted between the LSA scores of authentic and simplified texts (see Table 7). Overall, simplified and authentic texts showed significant differences in their LSA scores with simplified texts demonstrating more semantic overlap than authentic texts. Additionally, LSA sentence to sentence comparing adjacent sentences was significantly greater in simplified texts, as was LSA sentence to sentence comparing all sentences, and LSA sentence to paragraph scores than in authentic texts. These results indicate that simplified texts exhibited more semantic similarity within sentences and within paragraphs than did authentic texts. However, no significant difference was found in LSA sentence to text comparisons.
Comparison of authentic and simplified texts for latent semantic analysis (LSA): means (M), standard deviations (SD), t-values, and p-values.
4.7 Incidence of major parts of speech
As predicted, significant differences were found between simplified and authentic texts in relation to their part-of-speech use, but to a much lesser extent than in Crossley et al.'s (Reference Crossley, Louwerse, McCarthy and McNamara2007) exploratory analysis (see Table 8). In this analysis, authentic texts revealed a significantly greater number of infrequent linguistic features such as gerunds, past participles, and wh-pronouns. Authentic texts also displayed a tendency toward having a higher incidence of adjectives and subordinating conjunctions and prepositions, but these findings only approached significance. Simplified texts, on the other hand, showed significantly greater incidence of nouns, and past tense verbs. Unlike the exploratory analysis, no significant differences were noted between simplified and authentic texts in other categories.
Comparison of authentic and simplified texts for parts of speech: means (M), standard deviations (SD), t-values, and p-values.
4.8 Polysemy and hypernymy
Contrary to the findings of Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), where no differences were found in polysemy and hypernymy scores, significant differences were noted in their values when comparing the simplified and authentic texts in the present analysis (see Table 9). Verb hypernymy was significantly greater in simplified texts,Footnote 2 while polysemy values were significantly greater in simplified texts. No significant difference, however, was noted in noun hypernymy.
Comparison of authentic and simplified texts for polysemy and hypernymy: means (M), standard deviations (SD), t-values, and p-values.
4.9 Syntactic complexity
Contrary to prediction that syntactic complexity would be greater in simplified texts and to the findings of Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), the results of this study indicated that authentic texts are significantly more syntactically complex than simplified texts (see Table 10). Additionally, authentic texts showed more higher-level constituents per sentence and higher syntactic logic scores than simplified texts. Authentic texts also demonstrated a propensity to have more modifiers per noun phrase, but this finding only approached significance. For higher-level constituents per word and number of words before the main verb of a clause, no significant differences were noted.
Comparison of authentic and simplified texts for syntactic complexity: means (M), standard deviations (SD), t-values, and p-values.
4.10 Word information
As predicted and in line with Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), this investigation showed that authentic texts and simplified texts differ significantly in the word information categories of familiarity and frequency (see Table 11). With regard to familiarity, simplified texts were significantly more likely to contain words that were more familiar than those in authentic texts, while word frequency comparisons demonstrated that simplified texts contained significantly more frequent words than authentic texts. Additionally, this study demonstrated that simplified texts showed significantly lower age of acquisition scores and higher Paivio Meaningfulness scores. In the word information categories of concreteness, Colorado Meaningfulness, and imagability, no significant differences were noted.Footnote 3
Comparison of authentic and simplified texts for word information: means (M), standard deviations (SD), t-values, and p-values.
5. Discussion
Using the computational tool Coh-Metrix, this study has demonstrated that many properties of both simplified and authentic texts that were examined in Crossley et al.'s. (Reference Crossley, Louwerse, McCarthy and McNamara2007) exploratory analysis held true in this larger and more specific analysis. While differences between the two analyses were evident, these differences are explainable and are likely a consequence of the linguistic difficulty of the text being modified to suit the learning needs of the advancing student. Additionally, it is possible that because the corpus constructed for this analysis was much larger than the corpus used by Crossley et al. it allowed for greater coverage of lexical items such as connectives, logical operators, part-of-speech tags, polysemy and hypernymy values, and word information measures. The findings reported here thus allow for an expansion of the criteria that can generally be used to classify the language features found in authentic and simplified texts (see Table 12 for an overview) and should be seen as a progression of the Crossley et al. study. This study, as an improvement on the Crossley et al. study, is important as it provides material developers, reading researchers, and classroom teachers with more explicit information about the value of reading texts.
Restrictive study findings: predicted versus result.
*Finding was significant at p < .05 **Finding was significant at p < .001
This analysis has demonstrated that intermediate texts, unlike the beginner texts used by Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), exhibit no significant difference in the use of causal particles and verbs. While this finding contradicts the finding of the exploratory study, it is quite likely that the amount of causality displayed in simplified texts advances as learners advance. The present study also demonstrates that the use of connectives is similar to the use of connectives in Crossley et al., with the exception of positive causal, negative additive, and negative temporal connectives, which were higher in the exploratory analysis. This finding should likely be taken to mean that simplified texts at the intermediate level have begun to develop and link ideas with more complex connectives than had simplified texts at the beginner level. However, unlike the findings reported by Crossley et al., this study demonstrates that authentic texts are more likely to employ logical connectors than is simplified texts. Logical operators are important for discussing hypothetical situations and logically linking phrases and their comparative absence from simplified texts might limit the discourse structure of the texts.
In support of Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), this study indicates that authentic texts exhibit less overall lexical and semantic co-referential cohesion than do simplified texts and provides further evidence that simplified texts, with their dependency on noun phrases, avoidance of pronominal reference, and simple syntactic structure, provide greater co-referentiality. This study also finds that authentic texts at the intermediate level reveal greater diversity of word types than simplified texts, especially in more complex part-of-speech types. However, when compared to the beginner level analysis of Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), the differences between authentic and simplified texts are not as great. With regard to syntax, this study demonstrates that simplified texts at the intermediate level do not show a tendency to have more complex syntactic constructions than authentic texts. These findings are likely the result of simplified texts at the intermediate level moving toward more natural syntactic constructions and moving away from short sentences that rely on simple sentence constructions and elaboration.
When considering word information and frequency, the findings of this analysis were similar to those of Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007) in that authentic texts used significantly less familiar words than simplified texts and used significantly less frequent words than simplified texts. Additionally, unlike the exploratory analysis of Crossley et al., simplified texts showed a significantly lower age of acquisition score and significantly higher Paivio Meaningfulness scores. These results should be viewed as indications of the strength of simplified texts to provide more developmental and substantial language to L2 learners. In polysemy and hypernymy categories, some key differences were noted that were not apparent in the earlier study. Most importantly, with regard to polysemy and hypernymy values, significant differences between simplified and authentic texts were discovered. In this larger and more specific analysis, verb hypernymy was significantly greater in simplified texts leading to the conclusion that simplified texts uses more abstract verbs than authentic texts. Also, polysemy values were significantly greater in simplified texts, leading to the conclusion that simplified texts provide more ambiguous lexical items than authentic texts.
These findings help to further substantiate how authentic and simplified texts differ in their use of linguistic features and provide an important foundation for what these differences are. However, in the absence of transfer potential, the results are little more than descriptive. Nevertheless, this study, when considered in tandem with Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), opens important avenues for discussion about the benefits of reading texts types and how these texts can be better designed to suit the needs of readers based on language processing theories.
For instance, knowing that logical operators are important lexical features that constitute cohesive bonds between sections of texts (Halliday Reference Halliday1985), it would be beneficial to avoid the minimalization of such bonds when writing simplified texts. This is because reducing the amount of logical operators in a texts could lead to texts that do not elaborate, extend, and enhance the ideas of the texts as much as might be found in authentic texts. Thus, this may create texts that are more difficult for L2 learners to process (Johnson Reference Johnson1981). A tool such as Coh-Metrix would be invaluable in measuring this lexical feature. Another major criticism of simplified texts has been their dependence on lexical simplification, which could mean more lexically ambiguous texts resulting from more complex and precise words being replaced with simpler and more frequent words (Davies & Widdowson Reference Davies, Widdowson, Allen and Corder1974). This criticism appears to be accurate, but is addressable. Specifically, by tracking the polysemy and hypernymy scores of simplified texts using Wordnet, material developers would be able to select lexical items that are frequent but not necessarily ambiguous or abstract. This could produce texts that are easier to process as well as produce texts that would promote greater language acquisition. Texts controlled for frequency in this way would allow for quicker lexical processing and thus could be an advantage for L2 readers with regard to automaticity and word recognition (Woodinsky & Nation Reference Woodinsky and Nation1988; Carrell & Grabe Reference Carrell, Grabe and Schmitt2002). Lastly, simplified texts appear to not provide as robust causal relations as authentic texts. Using a computational tool such as Coh-Metrix would allow material developers to measure causal relationships in simplified text and produce texts that better assist the reader in relating events to actions by integrating stronger causal relationships between clauses (Pearson Reference Pearson1974–75).
Authentic texts, however, are far from optimal, but computational tools such as Coh-Metrix can also be a valuable resource in aiding material developers with selecting authentic texts that allow maximum potential for language acquisition and textual processing. For instance, the co-referentiality found in simplified L2 texts is considered important in providing helpful redundancy (Swaffar Reference Swaffar1981; Crandall Reference Crandall and Byrd1995; Nuttall Reference Nuttall2005; Day & Bamford Reference Day and Bamford1998). Co-referentiality has also been shown to assist readers in understanding the message and intention of a text and is an important means by which readers can connect to a text (Haber & Haber Reference Haber and Haber1981; Smith Reference Smith1988). However, authentic texts lack the same degree of co-referentiality as simplified texts. Measuring the lexical and semantic co-referentiality of authentic texts prior to adoption would allow for the selection of authentic texts that best provide the reader with helpful redundancy. Additionally, authentic texts could be compared and selected on the basis of multiple measures of syntactic complexity beyond the length of the sentence. These more robust indices of syntactic complexity would help in better matching the text to that of the reader's abilities, allowing for greater language comprehension and intake.
6. Conclusion
The aim of this study was to provide an approximate replication study of the Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007) analysis of L2 reading texts. To improve upon this original analysis, the present study looked at a larger corpus of simplified and authentic texts that were selected under more rigid parameters including audience and purpose. Generally, the results of this study replicate those of Crossley et al. and provide strong empirical evidence as to the type and extent of the differences between the two most prominent forms of L2 reading material: authentic and simplified texts.
There are also possible limitations to the study. The assumption that texts taken from different learner levels can be compared mutually raises theoretical questions about how well beginner and intermediate texts represent similar genres. While this study would have benefited from a selection of beginner texts that were different from those used by Crossley et al. (Reference Crossley, Louwerse, McCarthy and McNamara2007), this was not a practical alternative as authentic texts at the beginner level are rare and often marketed for children. Moreover, considering that the Crossley et al. study was exploratory, this study only attempted to replicate the findings. Any differences between the two studies were explained partially based on linguistic advancements that mirrored the progression of the texts' learning level. Additionally, the corpora used in this study, while larger than those used in Crossley et al., do not likely reach a level of representativeness. If feasible, future replication studies should consider even larger corpora.
One interesting by-product of this analysis, when viewed in light of Crossley et al.'s (Reference Crossley, Louwerse, McCarthy and McNamara2007) exploratory study, is that some measurements differed between the study of beginner texts and the present study of intermediate texts. To discern whether these differences are the result of the size of the two corpora (with the larger corpus in this analysis allowing for more coverage of lexical items) or of advancing linguistic features between beginner and intermediate texts, a comparison needs to be made between simplified beginner texts and simplified intermediate texts. The same needs to be done for authentic texts used at the beginner and intermediate levels. Such analyses would allow for a comparison of linguistic features and provide answers regarding whether the differences noted between these two studies are a result of controlled linguistic changes within the texts themselves, their advancement from the beginner level to the intermediate level, or the result of differences in coverage. Such analyses would also prove helpful in investigating how well Coh-Metrix can measure texts for linguistic difficulty between levels and perhaps provide support for the notion that Coh-Metrix provides an alternative to traditional readability formulas. Moreover, an analysis comparing authentic or simplified texts between beginner and intermediate levels could provide telling information about how efficient the linguistic strategies used by material developers are in terms of selecting authentic texts or writing simplified texts.
This analysis, while providing evidence of the differences between simplified and authentic texts, also demonstrates that modifying texts according to pedagogical and theoretical principles can cause unintended consequences to the natural structure of the discourse. These modifications can affect how a text is processed, comprehended, and understood. In the case of L2 learners, this may have important ramifications. The recognition of how these modifications interact with language processing can allow for important changes to be made in L2 reading texts. Progress such as this would allow for more readable texts and greater access to comprehensible input. This would have important implications for the development of L2 reading material, L2 acquisition, and L2 reading theory.
Acknowledgments
The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant IES R3056020018-02 to the University of Memphis. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education. The authors would like to thank Drs. Charles Hall and Max Louwerse for their help with the early forms of this paper. The authors would also like to express their appreciation to Heinle, a part of Cengage Learning, for their help in finding and providing many of the texts analyzed in this study. The authors are deeply indebted to the anonymous reviewers whose comments and ideas were crucial in producing the final version of this article.
Scott Crossley is an Assistant Professor and director of the TESOL program at Mississippi State University. His interests include computational linguistics, corpus linguistics, discourse processing, and discourse analysis. He has published articles in genre analysis, multi-dimensional analysis, discourse processing, speech act classification, cognitive science, and text linguistics.
Danielle McNamara is a cognitive scientist and director of the Cognitive Program at the University of Memphis. Her work involves the theoretical study of cognitive processes as well as the application of cognitive principles to educational practice. She has published widely in cognitive science with a recent emphasis on text comprehension and the effects of text coherence.
