A foreign accent is a common, normal aspect of second language (L2) learning among those who acquire their L2 after childhood (Asher & Garcia, 1969; Flege, Munro, & Mackay, 1995; Major, 2001; Scovel, 1988). Although accented speech is a salient indicator of a nonnative linguistic background, the impact of an accent on communication is complex. In some instances, speech marked by nonnative segmentals and prosody appears to be understood by native listeners just as well as native-produced speech from a familiar dialect (Munro & Derwing, 1999). However, an accent can sometimes have adverse consequences for the L2 speaker (Flege, 1988b). First, listeners might experience difficulty in understanding speech that differs from the patterns of oral production to which they are accustomed. Thus, accentedness sometimes leads to a loss of intelligibility. Second, interlocutors might respond negatively to accented speech because of impatience, inexperience with L2 speakers, or prejudice (Derwing, Rossiter, & Munro, 2002; Dávila, Bohara, & Saenz, 1993; Kalin & Rayko, 1978; Lippi-Green, 1997; Munro, 2003; Raisler, 1976; Rubin, 1992).

In the study reported here, we focus primarily on the first concern—the intelligibility of L2 speech—rather than on issues of bias against accents or L2 speakers. Derwing and Munro (1997) and Munro and Derwing (1995, 1999) have emphasized that L2 speech must be considered in terms of a variety of different dimensions. They defined intelligibility as the extent to which a speaker's utterance is actually understood and emphasized the importance of distinguishing this notion from comprehensibility, which refers to the listener's estimation of difficulty in understanding an utterance, and from accentedness, the degree to which the pronunciation of an utterance sounds different from an expected production pattern. Although comprehensibility and accentedness are related to intelligibility, they are partially independent dimensions of L2 speech. An utterance that is rated by a listener as “heavily accented,” for instance, might still be understood perfectly by the same listener. Furthermore, two utterances that are fully intelligible might entail perceptibly distinct degrees of processing difficulty, such that they are rated differently for comprehensibility.

A number of studies have explored comprehensibility and accentedness through listener judgments on equal-interval rating scales. Usually, listeners evaluate how difficult an utterance is to understand or how strongly accented it is. Such tasks tend to yield reliable results (Brennan & Brennan, 1981; Burda, Scherz, Hageman, & Edwards, 2003; Derwing & Munro, 1997; Thompson, 1991), and the assumption that accentedness can be partitioned into equal intervals (e.g., on a 9-point scale) has been upheld (Southwood & Flege, 1999).

The intelligibility of both normal—native language (L1) or L2—and disordered speech has been assessed using a number of different methods (see Kent, Miolo, & Bloedel, 1994, for a review). In L2 work, one of the most common of these is the dictation task in which listeners hear utterances and write them out in standard orthography (Bent & Bradlow, 2003; Brodkey, 1972; Burda et al., 2003; Derwing & Munro, 1997). The number of words correctly transcribed is understood as an index of speaker intelligibility. Other approaches have entailed comprehension questions (Anderson-Hsieh & Koehler, 1988), cloze tests (Smith & Rafiqzad, 1979), picture selection in response to a stimulus (Smith & Bisazza, 1982), elicitation of summaries (Perlmutter, 1989), and determination of truth value (Munro & Derwing, 1995). Each of these approaches has strengths and drawbacks, although probably none on its own can give a complete understanding of the intelligibility of a particular speaker.

A full discussion of how intelligibility can be assessed is beyond the scope of this review. However, it is worth noting that speech elicitation techniques and the nature of the listening task have important effects on the kinds of conclusions that can be drawn about intelligibility. When using dictation tasks based on extemporaneous speech, for instance, it seems reasonable to suppose that scoring the number of words correctly identified by a listener provides a useful window on the listener's comprehension. As noted by Zielinski (2004), that score might not correlate perfectly with how well the listener has actually grasped the full message intended by the speaker. She reported cases in which all of the words were correctly identified, with the listener still puzzling over what the speaker was trying to communicate. In other cases, the entire meaning of a sentence could be lost because of one missing word. For these reasons, we must consider transcription data as providing only one perspective on the intelligibility of an utterance or of a speaker.

Interpreting listeners' reactions to speech produced by L2 learners requires an understanding of the bases of their responses. On the one hand, their reactions might indicate something about the speech itself because they are influenced by the phonological properties of the speaker's output. It has been established, for instance, that foreign accent ratings correlate with the frequency of segmental and prosodic divergences from typical native speaker (NS) patterns (Anderson-Hsieh, Johnson, & Koehler, 1992; Brennan & Brennan, 1981; Magen, 1998; Munro & Derwing, 2001). On the other hand, responses to an utterance might vary with the listener because of familiarity with accents or because of the listener's L1 background. For example, Gass and Varonis (1984) found that exposure to specific foreign accents, to particular speakers, or to accents in general influenced how well listeners understood L2 speech. With respect to L1 background, Bent and Bradlow (2003) reported that nonnative listeners might find L2 speech more intelligible than native speech, whereas the opposite might be true for native listeners. Also, some research has suggested that speakers from a particular L1 background might have an advantage in understanding accented utterances from speakers who share that background. Smith and Bisazza (1982), for instance, observed an advantage for Japanese speakers listening to Japanese-accented English. However, the results of that study taken together with more recent work by Major, Fitzmaurice, Bunta, and Balasubramanian (2002) suggest that such an advantage is probably small and not consistently observable. In Major et al.'s study of 400 listeners, Spanish speakers showed a small intelligibility advantage when hearing Spanish-accented speech in comparison with other varieties, whereas Chinese and Japanese speakers showed no parallel advantage for their L1 accents. These findings appear to agree with research on native English speech showing that between-listener differences are generally a less important factor in speech comprehension than are the acoustic-phonetic characteristics of the speech itself (Hazan & Markham, 2004).

Taking these findings into account, Munro (2005) observed that intelligibility, comprehensibility, and accentedness scores must be seen as arising from various sources, including stimulus properties (the SP component)—describable in linguistic terms—and listener factors (the LF component)—relating to the listeners' language experience and accent familiarity. A critical issue yet to be resolved concerns the relative contribution of each component to listeners' judgments of speech. If stimulus properties were overwhelmingly important and listener factors were only a minor contributor, we would expect a high degree of consensus among listeners regarding the intelligibility, comprehensibility, or accentedness of a particular utterance. In that case, listeners, regardless of background, would be influenced by the stimulus characteristics in the same way, and their scores would be largely predictable from a linguistic analysis of the speech. In contrast, if such scores were highly subjective and influenced strongly by individual experience, interrater agreement from a diverse group of listeners should be low or even nonexistent, because different listeners would respond to the same stimulus properties in different ways.

In previous research, the relative contributions of SP and LF in listeners' judgments have been given little attention (but see Flege & Fletcher, 1992). In particular, few studies have yielded systematic, detailed comparisons of the responses of listeners from diverse linguistic backgrounds in order to determine the role of language experience in measures of intelligibility, comprehensibility, and accentedness. One exception is Flege's (1988a) study in which Chinese listeners were found to scale the accents of Chinese speakers in much the same way that native English listeners did. In further work by Mackay, Flege, and Imai (in press), it was found that Arabic listeners' ratings of Italian-accented English correlated strongly with the ratings of native English listeners. These findings suggest that even listeners from very different L1 backgrounds might agree more than they disagree with respect to their perceptions of L2 speech. However, these studies focused only on accentedness.

Much more work must be carried out to determine whether listeners from diverse backgrounds share similar responses with regard to intelligibility and comprehensibility. Of particular concern here is the fact that in English as a second language (ESL) pronunciation research, teaching, and testing, intelligibility has usually been evaluated from the perspective of native English listeners responding to foreign-accented speech. However, this is unrealistic in diverse communities with significant numbers of L2 speakers because many of the people with whom ESL speakers interact are not NSs themselves (see Smith & Bisazza, 1982). Thus, there is no a priori reason to assume that native listeners' responses are representative of the reactions that might come from the target linguistic community; nor is there any reason to suggest that the evaluations of native listeners are inherently more valid or meaningful than those of any other group. Rather, the most valuable information about whether a particular speaker is intelligible is likely to come from the people with whom the speaker seeks to interact.

In this study, we report the results of transcription and rating tasks that replicate and extend the work of Derwing and Munro (1997). The chief goal in this new study is to establish the extent to which listeners from a variety of L1 backgrounds share a response to extemporaneous L2 speech. Although our primary concern is intelligibility, we also consider comprehensibility and accentedness. In the original investigation, a group of native English listeners transcribed and rated the English utterances of 48 intermediate learners of English from four different L1 backgrounds: Cantonese, Japanese, Polish, and Spanish. The resulting data were used to establish the relationships among intelligibility, comprehensibility, and accentedness from the perspective of native English listeners hearing nonnative speech. In the present study, we use the same utterances evaluated in Derwing and Munro (1997) but present them to three groups of nonnative listeners for transcription and rating. Two of the new groups—Cantonese and Japanese listeners—are included because they are NSs of L1s represented in the original speech samples. Members of the third group—Mandarin listeners—do not speak any of these L1s. The fourth and final group consisted of native English listeners who participated in the original experiment. This choice of participants allows us to compare responses from listeners familiar with the pronunciation patterns of the speakers with those who have no such familiarity. We focus on the following research questions:

1. To what extent do nonnative English listeners from different L1 backgrounds share a response to L2 speech when they assess it for intelligibility, comprehensibility, and accentedness?
2. Do nonnative English listeners find speech produced with their own accent to be more intelligible, more comprehensible, or less accented than speech produced with other foreign or unfamiliar accents?
3. Do the intelligibility, comprehensibility, and accentedness scores of native English listeners responding to L2 speech differ radically from those of nonnative listeners?

Question 1 will be answered by computing intergroup correlations for the nonnative listeners on the various rating dimensions. Answering question 2 will entail comparing the between-group scores for the various listener groups. Question 3 will be addressed through further correlational analyses comparing the native English listeners in the original study with the three new listener groups.

METHODOLOGY

Speakers

The speech samples evaluated in this study are the same as those used by Derwing and Munro (1997). They were elicited from 48 participants, 12 NSs each of Cantonese, Japanese, Polish, and Spanish, all of whom were enrolled in high-intermediate-level ESL classes in a community college program in Edmonton, Canada. They will be referred to here as the CAN-S, JAN-S, POL-S, and SPA-S groups, respectively. Twelve of the speakers had completed the TOEFL, with scores ranging from 400 to 565 (M = 479). They ranged in age from 19 to 64 years (M = 29.1). It was not possible to balance the speaker sample for gender (18 males, 30 females). In an informal evaluation of the speakers' productions, the authors observed that their foreign accents ranged from moderate to strong.

Speech Stimuli

In the original study, the speakers provided a short, extemporaneous narrative by describing a cartoon story. Prior to recording, they viewed the cartoons and were invited to ask for clarification of the story and for any vocabulary they might need to describe it. Their narratives were recorded on audiotape and converted to computer audio files (22 kHz, 16 bits) to facilitate randomized stimulus presentation to the listeners. Because the speech material was to be used in a dictation and rating task, the experimenters extracted from each narrative a single continuous sample that ranged in length from 4.5 to 10.5 s (M = 7.0 s) to be presented to the listeners. There were no particular criteria for selection, except that each sample had to constitute a syntactically complete phrase in order not to confuse the listeners. The final stimulus set consisted of 48 utterances.

Listeners

Forty listeners participated in the current study. Of these, 10 were native English listeners who had participated in the earlier study and 30 were new participants—10 NSs each of Cantonese, Japanese, and Mandarin, who will be referred to here as the CAN-L, JAN-L, and MAN-L groups, respectively. These listeners were advanced ESL speakers enrolled in regular degree programs at a major Canadian university in metropolitan Vancouver. Prior to participating, they passed a pure-tone hearing screen (250–4,000 Hz) to verify that they had no serious hearing problems.

In the original study, 26 native Canadian English speakers, all living in metropolitan Vancouver, served as listeners. For the purposes of the current study, it was not possible to locate parallel L2 listener groups of that size. To allow appropriate statistical comparisons between the original and new groups of listeners, we used only a portion of the original listener data by randomly selecting a subset of 10 listeners (7 female, 3 male). These listeners will be referred to as the ENG-L group.

A small honorarium was paid to the listeners as an incentive to participate. Basic information was collected on each listener through a language background questionnaire administered prior to the listening tasks. The participants estimated their daily use of English and reported their amount of exposure to English produced with Cantonese, Japanese, Polish, and Spanish accents. Exposure was estimated using a scale of 0 (no contact) to 5 (every day/extended conversations).

From Table 1, which presents a summary of the listeners' characteristics by group, it can be seen that the members of the JAN-L group were slightly older on average than the other groups and that they had arrived in Canada at slightly later ages. They also reported more average daily use of English. The CAN-L, MAN-L, and ENG-L groups all reported more familiarity with Cantonese-accented English than with any other accent. The JAN-L group showed more familiarity with Cantonese accents than with either Polish or Spanish accents. These findings are not surprising, given the large numbers of Cantonese speakers in Vancouver relative to the other three language groups. As expected, both the CAN-L and JAN-L groups reported higher levels of experience with their own L1 accents in English than with other accents. In fact, their estimates might well be very conservative because they might not have taken their own English productions into account. Yet, they are probably the source of most of their accented English input.

Characteristics of the CAN-L, MAN-L, JAN-L, and ENG-L listener Groups (n = 10 per group)

One-way ANOVAs with Tukey post hoc tests were used to compare the groups' experience with the accents used in the study. There were no between-group differences in experience with Spanish accents. However, the CAN-L reported significantly more experience with Cantonese-accented English than did the JAN-L or native ENG-L groups (p < .05). Also, the JAN-L group reported more experience with Japanese-accented English than did any other listener group. The only other significant difference revealed that the ENG-L group indicated that it had more experience with Polish accents than did the MAN-L group (p <.05). It was also noted that one of the ENG-L participants—unlike all the others—rated his experience with Cantonese accents as 5 (every day/extended conversations). These observations allow us to investigate research question 2 in more detail than originally anticipated. In particular, it will be possible to determine whether the ENG-L group shows a greater comprehension of Polish-accented speech than the MAN-L group and whether the ENG-L participant with extensive experience with Cantonese accents shows a greater understanding of Cantonese speech than the other ENG-L participants.

Procedure

The new participants performed the same rating and transcription tasks used in the original study. As earlier, they first transcribed the 48 utterances into standard orthography and then rated them for accentedness and comprehensibility. In almost all respects, we followed the same protocol as Derwing and Munro (1997); however, there was one difference. The original judgment tasks were carried out in group listening sessions in a sound-treated lab. The listeners heard the utterances through high-quality loudspeakers and provided their responses on a response sheet by writing out the utterances in the dictation task and by circling numerical responses in the two rating tasks. For the new study, the original equipment and room were no longer available. This time the data were collected during individual sessions in a sound-treated suite, where the listeners heard the randomly ordered presentations of the stimuli through headphones. Although they wrote out their transcriptions as earlier, they provided their comprehensibility and accent ratings by pressing buttons on a computer screen in a self-paced task.

Stimuli were presented one at a time. Upon hearing an utterance, each listener was instructed to write out exactly what was said. Because previous work indicates that ratings of accentedness might become slightly harsher with repeated hearing of an utterance (Flege & Fletcher, 1992; Munro & Derwing, 1994), we counterbalanced the order in which comprehensibility and accentedness were rated, as in the original study. Immediately after transcribing an utterance, half of the participants rated it for comprehensibility by pressing a computer button on a 9-point scale ranging from 1 (easy to understand) to 9 (extremely difficult or impossible to understand). Once the comprehensibility rating was entered, the computer played the next item. These participants then heard the full set of utterances a second time and rated each one for accentedness on a 9-point scale ranging from 1 (no accent) to 9 (extremely strong accent). The other half of the participants followed the same procedure, except that they rated accentedness immediately after transcribing and rated comprehensibility during the second pass through the utterances. All participants completed a short familiarization session prior to beginning the task, during which they practiced on stimuli not used in the actual study. On average, the task took about 30–35 min to complete.

Scoring of the transcriptions was carried out using the exact-match method of the original study. This procedure entailed counting the words correctly transcribed in each utterance and assigning a percent-correct score based on the proportion of words in the sentence that were correctly transcribed.

RESULTS

Reliability

Interrater reliability was assessed for the intelligibility, comprehensibility, and accentedness scores. Table 2 presents intraclass correlations (Shrout & Fleiss, 1979) for the three sets of measures for each group of listeners. These correlations range from .825 to .922, indicating good interrater reliability for all groups on all measures. In other words, the members of each group of listeners tended to agree with one another on the relative intelligibility, comprehensibility, and accentedness of the speakers.

Average intraclass correlations by listener group (n = 10 per group) for each assessment dimension

Intergroup Correlations

For each of the three measures, we next pooled the scores on each stimulus item over the 10 listeners in each group to compute a mean listener group score for each item. The pooled scores were used to examine the intergroup correlations on intelligibility, comprehensibility, and accentedness. The relationships are expressed as Pearson r values for all possible pairings of the four listener groups. Every one of the correlations was significant at p < .01. The results are summarized in Table 3.

Intergroup correlations (Pearson r) on the listeners' responses to the 48 foreign-accented speech samples

For intelligibility, the correlations (r) ranged from .652 to .893, with all except the correlation between the Mandarin and Japanese listeners' scores above .70. Thus, the percentage of variance (r²) in any one listener group's scores that could be accounted for by any other group's scores ranged from about 42% to 80%. In general, there were moderate to strong intergroup correlations on intelligibility.

With respect to the listeners' ratings of accentedness and comprehensibility, correlations tended to be slightly weaker, although they still generally indicated moderate to moderately high intergroup agreement; all but one of the correlations was above .600. The one exceptional score was the relationship between the Japanese and Cantonese listeners' ratings of comprehensibility, which was slightly weaker (r = .470).

Effects of Listeners' L1 on Intelligibility Scores

To assess the effect of the listeners' L1 on their comprehension of the L2 speech, we computed a two-way mixed-design ANOVA on the listeners' scores. The grouping factor was listener (four levels), with speaker (four levels) serving as a within-listeners factor. Unless otherwise indicated, p < .05 was used as the criterion for significance in the tests reported here.

A preliminary examination of the intelligibility scores indicated some small but significant deviations from normality in the scores assigned to the Japanese and Spanish speakers (Shapiro-Wilks = .925 and .935, respectively, p < .05). For this reason, an arcsine transformation was applied to the intelligibility scores, an approach commonly used for proportional data such as these (Howell, 1992). The analysis of transformed intelligibility scores yielded a nonsignificant effect of listener, F(3, 36) = 0.202, p = .894, whereas the effects of speaker and the Listener × Speaker interaction were both significant, F(3, 108) = 57.149, h_p² (partial eta squared) = .835, p = .000 and F(9, 108) = 4.256, h_p² = .262, p = .000.

The significant Listener × Speaker interaction suggests that the listener groups differed in their comprehension of the four different accents, although the effect size (h_p²) associated with this interaction (accounting for about 26% of the variance) was much smaller than that of the main effect of speaker. One reason for the interaction might have been that one or more listener groups understood a particular accent better than one or more of the other listener groups. To explore this possibility, one-way between-groups ANOVAs were computed on the scores for each group of speakers. Only one of these analyses—that of the Japanese speakers' scores—yielded a significant effect of listeners, F(3, 36) = 4.249, p = .011. Post hoc (Tukey) tests revealed that this was because the Japanese listeners found the Japanese speakers more intelligible (91%) than did the English listeners (83%). This effect can be seen in Figure 1, which shows intelligibility scores broken down by listener and speaker groups. This was the only significant difference among listener groups in the entire corpus of intelligibility data.

Mean intelligibility scores (percent words correct) for each group of speakers. Each cluster shows the scores of the four listener groups for one speaker group.

A second reason for the Listener × Speaker interaction in the original ANOVA might have been different patterns of intelligibility scores for the speaker groups within the listener groups. For instance, the Japanese listeners might have found the Japanese speakers more intelligible than the other groups of speakers, whereas the Cantonese listeners might have found the Cantonese speakers to be the most intelligible. Post hoc t tests (Bonferroni-adjusted for familywise error, p < .05) were computed for the comparisons of interest. The results showed that the Japanese listeners indeed found the Japanese speakers more intelligible than all three of the other speaker groups. However, the Cantonese listeners did not find the Cantonese speakers to be more intelligible than any of the other speaker groups, although they did find the Japanese speakers to be more intelligible than either the Polish or the Spanish speakers. The Mandarin listeners found both the Cantonese and the Japanese speakers to be more intelligible than the Spanish speakers. The native English group did not find any speaker group to be more intelligible than any other speaker group.

As noted earlier, the native English listeners reported more familiarity with Polish-accented English than did the other listener groups. However, the analysis carried out here revealed no advantage for the English group in terms of its comprehension of Polish-accented speech. Also, one English listener reported a high degree of familiarity with Cantonese-accented English. An examination of that participant's overall intelligibility score for the Cantonese speakers (pooled over all 12 speakers) revealed that his score (76%) was actually lower, rather than higher, than the mean score for the English listener group as a whole (83%).

It is important to note that although some different patterns indeed emerged among the listener groups, the effect size associated with the Listener × Speaker interaction was actually quite small. The limited contribution of the listener group differences to intelligibility is evident in Figure 1. What is most noteworthy is that all listener groups showed similar levels of comprehension for each accent. For the CAN-S, POL-S, and SPA-S groups, the listener groups' scores were all within six points of each other. In contrast, the mean intelligibility score attributed to individual speakers (pooled across listeners in each group) ranged from 46% to 100%. Thus, it is clear that most of the variability in the intelligibility scores is due to something other than the effect of listener group.

One further way to examine how the listener groups performed on the intelligibility task is to identify the 10 most and 10 least intelligible speakers for each group of listeners and determine the number of intergroup agreements on who those speakers are. Table 4 summarizes the rates of pairwise agreement using this approach. Overall, all four of the listener groups agreed on 6 of the 10 most intelligible speakers and on 5 of the 10 least intelligible ones. As can be seen in Table 4, the pairwise agreements (i.e., agreements between each of the possible pairings of listener groups) are somewhat higher, with all but one exceeding 5/10.

Pairwise agreement among listener groups on the 10 most and 10 least intelligible speakers

Effects of Listeners' L1 on Comprehensibility and Accentedness

Mixed-design ANOVAs were computed for the comprehensibility and accentedness ratings, again with listener and speaker as factors. For comprehensibility, the results of the ANOVA were parallel to those observed for intelligibility. As earlier, there was a nonsignificant effect of listener, F(3, 36) = 1.068, whereas the effects of speaker and the Listener × Speaker interaction were both significant, F(3, 108) = 10.725, h_p² = .486, p = .000 and F(9, 108) = 5.982, h_p² = .333, p = .000, respectively. Also, as earlier, follow-up between-group ANOVAs revealed that only the Japanese speakers' scores were affected by listener group differences. Tukey tests showed that this was because the Japanese listeners rated the Japanese speakers as more comprehensible than did either the Cantonese or the Mandarin listeners.

Bonferroni-adjusted post hoc tests revealed that the Cantonese listeners rated the Cantonese speakers as easier to understand than the Japanese, Polish, and Spanish speakers. Also, the Japanese listeners rated the Japanese speakers as easier to understand than the Cantonese speakers, but not the other groups. The Mandarin listeners rated both the Cantonese and the Japanese speakers as easier to understand than the Spanish speakers. The native English listeners rated all speaker groups as equally comprehensible. These results are illustrated in Figure 2.

Mean comprehensibility ratings (1 = easy to understand, 9 = extremely difficult or impossible to understand) for each group of speakers. Each cluster shows the scores of the four listener groups for one speaker group.

The ANOVA on the accentedness data yielded the same general pattern as the other ANOVAs, with a nonsignificant effect of listener, F(3, 36) = 1.071, p = .373, a significant effect of speaker, F(3, 108) = 11.417, h_p² = .502, p = .000, and a significant Listener × Speaker interaction, F(9, 108) = 4.303, h_p² = .264, p = .000. Follow-up ANOVAs revealed, once again, that only the Japanese speakers' scores evidenced listener group effects. Tukey tests indicated that this was because the Japanese listeners assigned the Japanese speakers better (i.e., less accented) foreign accent ratings than did the Cantonese listeners.

Bonferroni tests showed that the Cantonese listeners rated the Cantonese speakers' accents as better (i.e., less accented) than the Japanese speakers' accents, but not better than any of the other groups. The Japanese listeners did not rate the Japanese speakers as having better accents than any of the other groups. Both the Mandarin and native English listeners rated all speaker groups as equally accented (see Figure 3).

Mean accentedness ratings (1 = no accent, 9 = extremely strong accent) for each group of speakers. Each cluster shows the scores of the four listener groups for one speaker group.