Speech is perceived by both ear and eye. When the auditory signal is degraded or embedded in noise, seeing the speaker significantly enhances speech understanding (Sumby & Pollack, Reference Sumby and Pollack1954). The contribution of visual information to speech is not restricted to conditions of degraded auditory input, but it also occurs when the sound is clear. As originally demonstrated by McGurk & MacDonald (Reference McGurk and MacDonald1976), the perception of non-ambiguous speech tokens can be modified by the simultaneous presentation of visually incongruent articulatory gestures. Typically, when presented with an auditory syllable /ba/ dubbed onto a face articulating /ga/, participants report hearing /da/. The occurrence of this so-called McGurk effect has been taken as a particularly powerful demonstration of the use of visual information in speech perception.
The use of lip-read information for speech perception has not only been demonstrated in adults, but also in children and infants. Several developmental studies suggest that integration of visual and auditory speech is present early in life. For example, infants at 4·5 months of age prefer to look at a face in which the articulatory gestures are in congruence with the heard vowel rather than to a face with incongruent speech gestures (Kuhl & Meltzoff, Reference Kuhl and Meltzoff1982). Five-month-olds are also susceptible to the McGurk effect, as for example demonstrated by Rosenblum, Schmuckler & Johnson (Reference Rosenblum, Schmuckler and Johnson1997; see also Burnham & Dodd, Reference Burnham and Dodd2004; Desjardins & Werker, Reference Desjardins and Werker2004).
Despite demonstrations that lip-reading contributes to speech perception in early infancy, it has also been demonstrated that the impact of lip-reading on speech perception increases with age. In the original study by McGurk & MacDonald (Reference McGurk and MacDonald1976), adults (18–40-year-olds) were compared with children. Results showed that adults were more influenced by incongruent visual information than two younger groups (3–4- and 7–8-year-olds) that did not differ consistently. This developmental trend has been confirmed in later studies (Bruce et al., Reference Bruce, Campbell, Doherty-Sneddon, Import, Langton, McAuley and Wright2000; Burnham & Sekiyama, Reference Burnham and Sekiyama2004; Desjardins, Rogers & Werker, Reference Desjardins, Rogers and Werker1997; Sekiyama & Burnham, Reference Sekiyama and Burnham2004; Wightman, Kistler & Brungart, Reference Wightman, Kistler and Brungart2006). For example, Hockley & Polka (Reference Hockley and Polka1994) presented audiovisual conflicting stimuli to five different age-groups: five-, seven-, nine- and eleven-year-olds and adults. Results again showed that the impact of lip-reading on speech identification increased with age, even beyond the age of twelve, as only half of the eleven-year-old group responded in an adult-like manner. Likewise, Massaro (Reference Massaro1984) compared six-year-olds with adults on an audiovisual speech identification task and found that children had only half the visual influence shown by adults. In this study, children and adults also performed a secondary task in which they indicated whether or not the mouth had moved. Despite this attentional focus on the region of the mouth, the difference in visual influence between the two age groups remained, thus suggesting that it was unlikely that children's low susceptibility to visual information was due to paying less attention to the face.
Such a developmental trend in the impact of visual speech may either suggest that lip-reading is an ability that simply needs to mature, or alternatively that linguistic experience is necessary, possibly because visible articulation is initially not well-specified. Exposure to audiovisual speech may then be necessary to develop phonetic representations more completely. This linguistic argument is in line with the finding that Japanese-speaking children are less skilled than English-speaking children at lip-reading at six years old (Sekiyama & Burnham, Reference Sekiyama and Burnham2004). This interpretation is also in line with the finding that dyslexics with poor phonological representations are also poor lip-readers (de Gelder & Vroomen, Reference de Gelder and Vroomen1998). Alternatively, rather than experience with listening to speech, it may also be that experience producing speech contributes to lip-reading ability. A finding that is in agreement with the latter is that, among preschoolers (four- to five-year olds), those who substitute phonemes in speech production are also poorer lip-readers and tend to be less influenced by the visual component of an incongruent audiovisual syllable than children who do not substitute phonemes (e.g. Desjardins et al., Reference Desjardins, Rogers and Werker1997).
In this present study, we further explored the developmental trend in the use of lip-read information. So far, all previous studies have looked at the biasing effect that a lip-read stimulus has on a simultaneously presented auditory speech token. Recently, though, it has been shown that besides such a direct bias effect, lip-reading may also serve a very different role in speech perception, namely in the recalibration of ambiguous auditory speech (Bertelson, Vroomen & de Gelder, Reference Bertelson, Vroomen and de Gelder2003; van Linden & Vroomen, Reference van Linden and Vroomen2007; Vroomen, van Linden, de Gelder & Bertelson, Reference Vroomen, van Linden, de Gelder and Bertelson2007; Vroomen, van Linden, Keetels, de Gelder & Bertelson, Reference Vroomen, van Linden, Keetels, de Gelder and Bertelson2004). In this view, lip-read information serves as a teacher of the phonetic system, showing how to interpret an initially ambiguous speech sound. Recalibration by lip-read speech has been demonstrated by exposing participants to an auditory ambiguous speech token (henceforth A?) intermediate between /aba/ and /ada/ dubbed onto a visible face articulating either /aba/ or /ada/ (A?Vb and A?Vd, respectively; A=Auditory, V=Visual). Following a short exposure phase (usually in the order of eight trials) to either one of these audiovisual stimuli, participants were given immediately thereafter auditory-only test trials in which they categorized the auditory ambiguous token (A?) and its two closest neighbours on the phonetic continuum (A?–1 and A?+1) as either /aba/ or /ada/. Results showed that following exposure to A?Vb, participants were more likely to categorize the ambiguous test tokens as /aba/ if compared with exposure to A?Vd. The occurrence of this after-effect (i.e. the difference in the percentage of /aba/ responses on test trials following exposure to A?Vb versus A?Vd) represents the basic recalibration phenomenon. Presumably, listeners resolved during the exposure phase the conflict between the heard and the lip-read speech information by shifting the phoneme boundary such that the ambiguous sound was categorized in accordance with the lip-read information it was combined with. This learning effect could then in turn be observed as an after-effect in the test trials.
To ensure that this phenomenon was not due to some kind of priming or response bias (e.g. in the test phase participants might simply be biased to repeat what they had heard during the exposure phase), participants in the study by Bertelson et al. (Reference Bertelson, Vroomen and de Gelder2003) were also exposed to auditory non-ambiguous and audiovisual congruent adapters AbVb and AdVd. These adapters were not expected to evoke recalibration – because there was no conflict between the heard and seen information – but selective speech adaptation (Eimas & Corbit, Reference Eimas and Corbit1973). Unlike recalibration, selective speech adaptation does not depend upon exposure to an audiovisual conflict, but upon prolonged exposure to an auditory non-ambiguous speech sound as it probably reflects ‘fatigue’ of some of the relevant acoustic processes (Eimas & Corbit, Reference Eimas and Corbit1973; Samuel, Reference Samuel1986). In line with this prediction, following exposure to AbVb, participants were in subsequently delivered test trials less likely to categorize the auditory ambiguous test tokens as /aba/ if compared with exposure to AdVd, an effect that was in the opposite direction of recalibration. Further tests also showed that participants were unable to distinguish the exposure stimuli with ambiguous speech sounds (causing recalibration) from non-ambiguous speech sounds (causing selective adaptation). Participants could thus not reliably discriminate between A?Vb and AbVb, and between A?Vd and AdVd when explicitly asked to do so in a discrimination task. This therefore excluded the possibility that results were caused by any deliberate response strategy (Bertelson et al., Reference Bertelson, Vroomen and de Gelder2003).
Here, we wanted to test whether there is, as in the case of the McGurk effect, a developmental trend in the use of visual speech for the purpose of recalibration. Recalibration of auditory speech by lip-read information has never been tested in children, as it has been demonstrated only very recently in adults (for lexically induced recalibration, see also Eisner & McQueen, Reference Eisner and McQueen2005; Kraljic & Samuel, Reference Kraljic and Samuel2005, Reference Kraljic and Samuel2006, Reference Kraljic and Samuel2007; Norris, McQueen & Cutler, Reference Norris, McQueen and Cutler2003). In studies with adults, it has been observed that those who are more susceptible to the McGurk effect will also tend to show larger recalibration effects (van Linden & Vroomen, Reference van Linden and Vroomen2007). Assuming that the contribution of lip-reading to speech increases with age, one might expect a corresponding trend in lip-read-induced recalibration. Younger children might then show smaller recalibration effects than older ones, possibly because they lack a sufficiently detailed representation of the lip-read speech.
To test for cross-age differences, a group of five-year-olds and eight-year-olds participated, as in the original studies by McGurk & MacDonald (Reference McGurk and MacDonald1976). Following exposure to the auditory ambiguous and incongruent audiovisual speech tokens A?Vb and A?Vd, we expected to find after-effects indicative of recalibration (i.e. more /aba/ responses following exposure to A?Vb than following exposure to A?Vd) that might be smaller or even absent in the younger age group.
As a control condition, we included the auditory non-ambiguous and audiovisual congruent adapters AbVb and AdVd. We did not expect these adapters to produce substantial selective adaptation effects, but they were nevertheless important from a methodological point of view. The effect of these adapters was expected to be small because the optimal number of exposure trials for obtaining selective adaptation is usually much bigger than for recalibration. Whereas recalibration occurs fast and is at ceiling after only very few exposure trials, selective speech adaptation takes more time to develop and increases with number of exposures (Vroomen et al., Reference Vroomen, van Linden, de Gelder and Bertelson2007; Vroomen et al., Reference Vroomen, van Linden, Keetels, de Gelder and Bertelson2004). Given that the focus of the current study was on recalibration, we used only a limited amount of exposure trials (eight) that was optimal for obtaining recalibration, but not selective speech adaptation. Another reason why selective speech adaptation was expected to be small is that a developmental trend has been reported here as well (Sussman, Reference Sussman1993; Sussman & Carney, Reference Sussman and Carney1989). Sussman & Carney (Reference Sussman and Carney1989) explored selective speech adaptation in five–six-year-olds, seven–eight-year-olds, nine–ten-year-olds and adults. They observed that when adapted to /ba/, only adults showed a boundary shift on a /ba/–/da/ continuum indicative of selective speech adaptation (i.e. more /da/ responses following exposure to /ba/). For the seven–eight-year-olds and nine–ten-year-olds, no boundary shift was observed, while the youngest age-group actually showed a shift in the opposite direction (i.e. more responses consistent with the exposure stimuli), a finding for which no explanation was offered. The differences between children and adults were explained by the children's inability to process all acoustic details, as the youngest children were also less proficient in discriminating the speech tokens. (In a later study, though, it was found that the /ba/ token was not a particularly powerful adapter, and when adapted to /da/, both five-year-olds and adults showed a boundary shift (Sussman, Reference Sussman1993)).
Despite the fact that the adapters with non-ambiguous speech tokens were not expected to produce a substantial selective speech adaptation effect, we included them in the design of our study because they allowed us to address a methodological concern. One possible strategy that children may adopt when tested in an ‘exposure-test’ design is that, whenever unsure, they repeat the phoneme heard during the exposure phase. Such a deliberate response strategy would effectively boost the after-effect thought to reflect recalibration, which is a problem that might become particularly pertinent when comparing across age-groups, because different-aged children might adopt different strategies. This response strategy might also explain why children sometimes display no selective speech adaptation effects, and why five–six-year-olds may actually show a reversed trend (Sussman & Carney, Reference Sussman and Carney1989).
In the present study, we sought to alleviate this problem of response bias by comparing the after-effects of audiovisual adapters containing ambiguous sounds with those containing non-ambiguous sounds (i.e. we subtracted (A?Vb – A?Vd) – (AbVb – AdVd)). This comparison between after-effects of ambiguous versus non-ambiguous sounds annuls any contribution of deliberate response strategies to the extent that its contribution is equally big in both cases. This can be assumed to be the case, because even adults have great difficulty discriminating adapter stimuli containing ambiguous from non-ambiguous sounds when explicitly asked to do so (Bertelson et al., Reference Bertelson, Vroomen and de Gelder2003). Hence, one can safely predict that if recalibration is at stake, after-effects of audiovisual adapters containing ambiguous sounds will be bigger than those containing non-ambiguous sounds independent of participants' response biases.
METHOD
Participants
Twenty five-year-olds and twenty eight-year-olds from a public elementary school in a Dutch village, Hapert, were tested. All children were native Dutch speakers, and only children with normal age-related speaking and hearing proficiency, as judged by their teacher, participated in the experiment. Eight boys and twelve girls with a mean age of 5 ; 5, and ten boys and ten girls with a mean age of 8 ; 6 were tested. Data from two of the five-year-olds were discarded from analyses because one child completed only two experimental sessions, while the other frequently failed to respond during the test. This left eighteen children in the younger age group.
Stimuli
Details of the stimuli have been described elsewhere (Bertelson et al., Reference Bertelson, Vroomen and de Gelder2003). In short, a 9-point /aba/–/ada/ speech continuum was created by varying the frequency of the second (F2) formant in equal steps. The midpoint token of this continuum (A?), the most ambiguous one as determined by pilot tests, was used to create the audiovisual ambiguous exposure stimuli. It was dubbed onto the video of a face articulating /aba/ or /ada/, resulting in the two exposure stimuli used to induce recalibration: A?Vb and A?Vd. The two endpoint tokens, clear /aba/ and clear /ada/ were dubbed onto the congruent lip gestures to create the exposure stimuli AbVb and AdVd for the control condition. The three tokens closest to the phoneme boundary, A?, and its two neighbours on the continuum (A?–1 and A?+1), were used for the test trials.
Design and procedure
Design and procedures were adopted from our previous study (Bertelson et al., Reference Bertelson, Vroomen and de Gelder2003). The children were tested individually in a quiet room in their school. They were seated in front of a table at a distance of about 50 cm from a 15-inch laptop (Dell Inspiron 2650) on which the visual stimuli were presented. Sounds were delivered over two loudspeakers that were placed besides the screen. The experimenter sat at the side of the table in front of a keyboard. Testing began with a series of practice trials followed by four experimental sessions. Total testing took about half an hour with small pauses in between sessions.
There were four experimental sessions, each consisting of four exposure-test blocks. During an exposure-test block, one of the four bimodal exposure stimuli (AbVb, AdVd, A?Vb or A?Vd) was successively presented eight times (ISI=1 s.), followed by six auditory-only test tokens. The test tokens consisted of the three midpoint tokens of the continuum (A?–1, A? and A?+1), presented twice in counterbalanced order. All four bimodal exposure stimuli were presented once in an experimental session. Total testing thus consisted of 16 exposure-test blocks with 96 test trials (4 exposure stimuli×4 sessions×6 test trials). Presentation order of the exposure stimuli was counterbalanced over the experimental sessions. Participants were asked to silently watch the video during exposure. In the test phase, the video turned grey. After participants heard a test stimulus, they were required to report orally whether they had heard /aba/ or /ada/. The experimenter then pressed a corresponding key on the keyboard. Trials in which the child was inattentive or failed to respond were discarded from analyses. The next test trial started after a key press by the experimenter. A practice session containing each of the four different exposure-test blocks was included at the start of the experiment. Practice continued until the child completely understood the task and successfully completed at least two exposure-test blocks. When necessary, the child was encouraged to keep attention on the face during audiovisual exposure.
RESULTS
Figures 1a–2b present performance in the test phase for each exposure condition and age group. Figures 1a and 1b show the results for the five-year-olds, and Figures 2a and 2b for the eight-year-olds; Figures 1a and 2a show performance following exposure to the ambiguous adapters A?Vb and A?Vd, and Figures 1b and 2b for the non-ambiguous adapters AbVb and AdVd. Each graph shows the proportion of /aba/ responses for the ambiguous token (A?), and its two neighbours on the phonetic continuum (A?–1 or A?+1). The first thing to note is that for both the non-ambiguous and ambiguous exposure conditions, children responded more often with /aba/ following exposure to ‘aba’-adapters (AbVb and A?Vb) than following exposure to ‘ada’-adapters (AdVd and A?Vd). This is visible in the figures in that the solid lines are always above the dotted lines. As mentioned before, our design allowed us to subtract out such an overall tendency to repeat the phoneme heard during the exposure phase, as recalibration devoid of response bias should manifest itself in that the after-effect (the difference between the dotted and solid lines) of ambiguous sounds should be bigger than that of non-ambiguous sounds. As is clearly visible, for the five-year-olds there was no difference in the size of the after-effects of ambiguous and non-ambiguous sounds (the difference between dotted and solid lines in Figures 1a vs. 1b is the same or even appears slightly bigger in the case of non-ambiguous sounds). For the eight-year-olds, though, after-effects of ambiguous sounds were bigger than those of non-ambiguous sounds (Figure 2a vs. 2b).
Performance on test trials for the five-year-olds after exposure to adapters containing auditory ambiguous sounds (A?) combined with visual /aba/ (A?Vb) or visual /ada/ (A?Vd). The graph shows the proportion of /aba/ responses for the test tokens A?–1 (/aba/-like), A? (ambiguous) and A?+1 (/ada/-like).
Performance on test trials for the five-year-olds after exposure to adapters containing auditory non-ambiguous sounds combined with visual /aba/ (AbVb) or visual /ada/ (AdVd).
Performance on test trials for the eight-year-olds after exposure to adapters containing auditory ambiguous sounds (A?) combined with visual /aba/ (A?Vb) or visual /ada/ (A?Vd).
Performance on test trials for the eight-year-olds after exposure to adapters containing auditory non-ambiguous sounds combined with visual /aba/ (AbVb) or visual /ada/ (AdVd).
In the 2 (Age-group five- or eight-year-olds)×2 (Adapter Sound Ambiguous or Non-ambiguous)×2 (Identity of the visual Adapter /aba/ or /ada/)×3 (Test Token A?–1, A?, A?+1) overall ANOVA on the proportion of /aba/ responses, there was a main effect of the identity of the adapting stimulus (F(1, 36)=16·50, p<0·001), because there were more /aba/ responses following exposure to /aba/-adapters than /ada/-adapters, and a main effect of test token (F(2, 72)=49·37, p<0·001), because there were more /aba/ responses for the /b/-like test token A?–1 than for the /d/-like test token A?+1. There was, furthermore, an interaction between test token and age-group (F(2, 72)=11·75, p<0·001). Inspection of the figures shows that the eight-year-olds had steeper identification functions than the five-year-olds, indicating that the eight-year-olds responded in a more ‘categorical’ way. Most importantly, there was a significant interaction between age group, ambiguity of the adapter sound and identity of the adapter stimulus (F(1, 36)=6·48, p<0·015). In the figures this interaction is clearly visible as for the eight-year-olds, the difference between the solid and dotted lines (the after-effect) is bigger for ambiguous than non-ambiguous speech sounds, while there is no such difference for the five-year-olds. No other interactions were significant.
For ease of interpretation, we computed after-effects by subtracting the proportion of /aba/ responses following exposure to A?Vd from A?Vb and AdVd from AbVb (see Table 1), thereby pooling over the three test tokens. Separate paired t-tests confirmed that the after-effect of the eight-year-olds for ambiguous sounds was bigger than of the non-ambiguous sounds, 0·23 versus 0·08, respectively (t(1,19)=2·64, p=0·016), while for the five-year-olds there was no such difference, 0·13 versus 0·20, respectively (t(1,17)=1·12, p=0·28). The eight-year-olds thus showed the predicted recalibration effect in that their after-effect of ambiguous sounds was bigger than that of non-ambiguous sounds (a 0·15 difference in the predicted direction), while the five-year-olds showed no sign of recalibration (a non-significant 0·07 difference in the opposite direction).
Mean proportion of /aba/ responses and mean after-effects induced by audiovisual ambiguous and audiovisual non-ambiguous exposure sounds on auditory-only test trials
DISCUSSION
There is by now accumulating evidence that when listeners hear an ambiguous phoneme, they flexibly adjust the phonetic categories of their language in accordance with disambiguating formation that tells what the phoneme should be (Kraljic & Samuel, Reference Kraljic and Samuel2006, Reference Kraljic and Samuel2007; Norris et al., Reference Norris, McQueen and Cutler2003; van Linden, Stekelenburg, Tuomainen & Vroomen, Reference van Linden, Stekelenburg, Tuomainen and Vroomen2007; van Linden & Vroomen, Reference van Linden and Vroomen2007; Vroomen et al., Reference Vroomen, van Linden, de Gelder and Bertelson2007). Here we demonstrate, for the first time, that there is also a developmental trend in this phenomenon. Eight-year-olds, but not five-year-olds, learned to categorize an initially ambiguous speech sound in accord with previously seen lip-read information. Presumably, eight-year-olds adjusted their phoneme boundary to reduce the phonetic conflict in the audiovisual stimuli A?Vb or A?Vd. This shift may occur in the older group but not the younger one because lip-reading is not yet very effective at the age of five. One possibility why the five-year-olds are less proficient is that they lack a sufficiently detailed representation of the visual information. This interpretation is in line with findings showing that young children are less proficient in decoding lip-read speech (Hockley & Polka, Reference Hockley and Polka1994; Massaro, Reference Massaro1984; McGurk & MacDonald, Reference McGurk and MacDonald1976; Sekiyama & Burnham, Reference Sekiyama and Burnham2004). Experience producing speech (e.g. Desjardins et al., Reference Desjardins, Rogers and Werker1997) may be another critical step to develop a visual or articulatory based representation with sufficient detail so that it can be used to recalibrate the existing auditory representation. Young children may in this view thus show no lip-read-induced recalibration because they do not yet decode the lip-read message in sufficient detail.
To examine this further, one might explore whether recalibration in young children can be induced by lexical information. Lexical information can, like lip-read speech, inform a listener what an initially ambiguous sound should be. For example, an ambiguous sound halfway between /b/ and /d/ in the context of ?utter is likely to be a /b/ rather than a /d/ because butter but not dutter is a word in English. Several studies have shown that recalibration can indeed be evoked by lexical knowledge (Eisner & McQueen, Reference Eisner and McQueen2005; Kraljic & Samuel, Reference Kraljic and Samuel2005, Reference Kraljic and Samuel2006, Reference Kraljic and Samuel2007; Norris et al., Reference Norris, McQueen and Cutler2003; van Linden et al., Reference van Linden, Stekelenburg, Tuomainen and Vroomen2007), and that lip-read and lexically induced recalibration are similar to each other (van Linden & Vroomen, Reference van Linden and Vroomen2007). If the young children's lack of recalibration is caused by their poor lip-reading abilities, it might nevertheless be possible to observe lexically induced recalibration to the extent that the words are part of the children's lexicon.
Our findings also have methodological consequences for studies measuring after-effects that use a typical exposure-test design. Here, we were concerned that children might adopt a response strategy of simply repeating the phoneme perceived during the foregoing exposure phase. Such a deliberate response strategy would artificially boost the after-effect considered to reflect recalibration. The use of adapters with non-ambiguous and congruent speech sounds, though, allowed us to subtract out the contribution of this response strategy. By so doing, we clearly observed that for the eight-year-olds after-effects of ambiguous and non-ambiguous sounds differed in the expected direction, thus indicating that there was indeed recalibration, while there was no sign of this for the five-year-olds.
Our results may also shed new light on previous studies showing that adults have larger selective speech adaptation effects than children (Sussman & Carney, Reference Sussman and Carney1989). This has been attributed to the fact that adults are more categorical in perceiving the test stimuli, a phenomenon we also observed with the older children (see also Hazan & Barrett, Reference Hazan and Barrett2000; Morrongiello, Robson, Best & Clifton, Reference Morrongiello, Robson, Best and Clifton1984; Nittrouer & Studdert-Kennedy, Reference Nittrouer and Studdert-Kennedy1987). The present results, though, make it clear that in order to measure after-effects in an exposure-test design, a baseline is needed that takes into account the various response strategies that participants might use. One strategy that we observed here is that children may have a tendency to report the exposure stimuli presented during the foregoing exposure phase. For future research on selective speech adaptation or recalibration, it seems important to include a baseline that takes such strategic effects into account, especially when comparing different age-groups because children might differ on this strategy.
In conclusion, here we demonstrated that eight-year-olds, but not five-year-olds learned to categorize ambiguous speech sounds by the use of lip-read information. The five-year-olds may not yet be able to adjust their phonetic boundary because they are not adept decoders of lip-read information. One may thus expect proficiency in lip-reading not only to correlate with susceptibility to the McGurk effect (e.g. Massaro, Reference Massaro1987), it may also appear to be a critical skill for lip-read-induced recalibration (van Linden & Vroomen, Reference van Linden and Vroomen2007). Further testing is needed, though, to understand what critical experience is needed that helps to promote lip-reading, and how it relates to lip-read-induced recalibration of phonetic categories in detail.
