Hostname: page-component-74656f9884-6grmk Total loading time: 0 Render date: 2026-11-06T05:09:46.840Z Has data issue: false hasContentIssue false
  • English
  • Français

Multimodal input in second-language speech processing

Published online by Cambridge University Press:  17 November 2020

Debra M. Hardison
Debra M. Hardison*
Affiliation:
Michigan State University, East Lansing, USA
*
*Corresponding author. Email: hardiso2@msu.edu
Rights & Permissions [Opens in a new window]

Extract

This timeline provides an update on research since 2009 involving auditory-visual (AV) input in spoken language processing (see Hardison, 2010 for an earlier timeline on this topic). A brief background is presented here as a foundation for the more recent studies of speech as a multimodal phenomenon (e.g., Rosenblum, 2005).

Information

Type
Research Timeline
Information
Language Teaching , Volume 54 , Issue 2 , April 2021 , pp. 206 - 220
Check for updates
Copyright
Copyright © The Author(s), 2020. Published by Cambridge University Press

Introduction

This timeline provides an update on research since 2009 involving auditory-visual (AV) input in spoken language processing (see Hardison, Reference Hardison2010 for an earlier timeline on this topic). A brief background is presented here as a foundation for the more recent studies of speech as a multimodal phenomenon (e.g., Rosenblum, Reference Rosenblum, Pisoni and Remez2005).

In the 1950s, some researchers suggested that the prevailing view of speech as an auditory-only (A-only) event had overlooked an important source of input. Sumby and Pollack (Reference Sumby and Pollack1954) argued that speech intelligibility could be enhanced by observation of the speaker; specifically, lip movements could be helpful for disambiguating consonant sounds (Miller & Nicely, Reference Miller and Nicely1955). Subsequent studies demonstrated that visual cues from a speaker's face offered an advantage in the accurate identification of speech sounds for a variety of listener populations, languages, and stimulus conditions. These included the hearing impaired (e.g., Walden, Prosek, Montgomery, Scherr, & Jones, Reference Walden, Prosek, Montgomery, Scherr and Jones1977; Bergeson, Pisoni, & Davis, Reference Bergeson, Pisoni and Davis2003), non-impaired listeners trying to comprehend conceptually difficult messages or accented speech (Reisberg, McLean, & Goldfield, Reference Reisberg, McLean, Goldfield, Dodd and Campbell1987), non-impaired listeners in ambient noise (e.g., Benoît, Mohamadi, & Kandel, Reference Benoît, Mohamadi and Kandel1994 for French; MacLeod & Summerfield, Reference MacLeod and Summerfield1990 for English), listeners of speech presented in the clear (e.g., McGrath & Summerfield, Reference McGrath and Summerfield1985), infants in their first language (L1) development (e.g., Meltzoff & Moore, Reference Meltzoff, Moore, de Boysson-Bardies, de Schonen, Jusczyk, McNeilage and Morton1993), and individuals responding to the McGurk Effect (e.g., McGurk & MacDonald, Reference McGurk and MacDonald1976; Hardison, Reference Hardison1996) – a perceptual effect in which discrepant AV cues may result in an illusory percept (e.g., a combination of auditory /ba/ and visual /ga/ may produce the percept /da/). For second-language (L2) learners of English (L1 Japanese and Korean), training with visual cues from a native speaker's face improved their identification accuracy of /r/ and /l/ which transferred to production improvement (Hardison, Reference Hardison2003) and earlier identification of words beginning with those sounds (Hardison, Reference Hardison2005). Articulatory gestures often precede the associated acoustic signal, essentially giving the listener/observer a head start in reducing the set of potential candidates in the speech recognition process (e.g., Munhall & Tohkura, Reference Munhall and Tohkura1998). In addition to the information value of lip movements in face-to-face interactions, head and eyebrow movements, which are correlated with changes in vocal pitch, also improve speech perception (e.g., Munhall, Jones, Callan, Kuratate, & Vatikiotis-Bateson, Reference Munhall, Jones, Callan, Kuratate and Vatikiotis-Bateson2004). Among other nonverbal cues, hand-arm gestures contribute to language comprehension (Sueyoshi & Hardison, Reference Sueyoshi and Hardison2005; Gullberg, Reference Gullberg2006); in particular, beat gestures can focus perceivers’ attention on certain elements in multimodal discourse (Dimitrova, Chu, Wang, Özyürek, & Hagoort, Reference Dimitrova, Chu, Wang, Özyürek and Hagoort2016).

Early demonstrations of the advantage of computer-based visual displays of acoustic information in L2 training involved L1 Dutch speakers learning Chinese lexical tones (Leather, Reference Leather, Leather and James1990) and English sentence-level intonation (de Bot, Reference de Bot1983). Further advances in computer-based sources of visual feedback on one's own speech or that of a model speaker led to widespread use of the acronym CAPT (Computer-Assisted Pronunciation Training; see Hincks, Reference Hincks, Reed and Levis2015 for a review). This feedback includes displays of waveforms, which can visually represent the duration of sounds (e.g., Motohashi-Saigo & Hardison, Reference Motohashi-Saigo and Hardison2009); spectrograms, which show the internal structure of a sound's acoustic energy (e.g., Hardison, Reference Hardison, Peters and Heraud2019); and pitch tracking for visualizing the rise and fall of vocal pitch (e.g., Hardison, Reference Hardison2004). Other displays of articulatory information, such as ultrasound imaging with its history in clinical applications, have been adapted to provide feedback in language learning (e.g., Gick, Bernhardt, Bacsfalvi, & Wilson, Reference Gick, Bernhardt, Bacsfalvi, Wilson, Hansen and Zampini2008). Automatic Speech Recognition (ASR) systems that rely on machine learning can be used for feedback through dictation programs or segmental error detection in a learner's utterance (e.g., O'Brien et al., Reference O'Brien, Derwing, Cucchiarini, Hardison, Mixdorff, Thomson and Levis2018). Computer-animated faces or talking heads (e.g., Massaro & Light, Reference Massaro and Light2003) have the potential to serve as virtual pronunciation teachers with the goal of showing the articulatory movements necessary to produce a target sound.

The importance and contribution of multimodal input is supported by the findings of neuroimaging studies which have revealed a relationship between visual input and activity in the auditory cortex during speech perception (e.g., Sams et al., Reference Sams, Aulanko, Hämäläinen, Hari, Lounasmaa, Lu and Simola1991; Hickok, Buchsbau, Humphries, & Muftuler, Reference Hickok, Buchsbau, Humphries and Muftuler2003). The developmental role of multimodal input is captured well in the following statement from Shams and Seitz (Reference Shams and Seitz2008), ‘It is likely that the human brain has evolved to develop, learn and operate optimally in multisensory environments.’ (p. 411).

The updated timeline on spoken language processing is constructed around the following themes based on the studies that have informed the directions that multimodal speech research has been taking.

  1. 1. Input from the human face

  2. 2. Electronic displays

    1. A. Displays of acoustic information (e.g., pitch, waveforms)

    2. B. Displays of articulatory position and movement (e.g., ultrasound)

    3. C. Computer-animations or talking heads

    4. D. ASR

  3. 3. Hand-arm gestures

  4. 4. Neurophysiological studies of AV processing

Debra M. Hardison is affiliated with the Department of Linguistics and Languages at Michigan State University. She conducts research on multimodal integration in spoken language processing, co-speech gesture, factors affecting oral skills development for L2 learners, and the applications of technology in L2 perception and production training involving segmental and suprasegmental features. She has given plenaries at New Sounds: International Symposium on the Acquisition of Second Language Speech and Pronunciation in Second Language Learning & Teaching and publishes in numerous journals and edited collections. She teaches graduate courses in L2 phonetics, and research methods in language learning and teaching.

Footnotes

* denotes the full reference appears in the references list above.

Note. Authors' names are shown in small capitals when the study referred to appears in this timeline.

References

Benoît, C., Mohamadi, T., & Kandel, S. (1994). Effects of phonetic context on audio-visual intelligibility of French. Journal of Speech and Hearing Research, 37(5), 1195–1203.CrossRefGoogle ScholarPubMed
Bergeson, T. R., Pisoni, D. B., & Davis, R. A. O. (2003). A longitudinal study of audiovisual speech perception by children with hearing loss who have cochlear implants. Volta Review, 103(4), 347–370.Google ScholarPubMed
de Bot, K. (1983). Visual feedback of intonation I: Effectiveness and induced practice behaviour. Language and Speech, 26(4), 331–350.CrossRefGoogle Scholar
Dimitrova, D., Chu, M., Wang, L., Özyürek, A., & Hagoort, P. (2016). Beat that word: How listeners integrate beat gesture and focus in multimodal speech discourse. Journal of Cognitive Neuroscience, 28(9), 1255–1269.CrossRefGoogle ScholarPubMed
Gick, B., Bernhardt, B. M., Bacsfalvi, P., & Wilson, I. (2008). Ultrasound imaging applications in second language acquisition. In Hansen, J. & Zampini, M. (Eds.), Phonology and Second Language Acquisition (pp. 309–322). Amsterdam, Netherlands: John Benjamins.CrossRefGoogle Scholar
Gullberg, M. (2006). Some reasons for studying gesture and second language acquisition (Hommage à Adam Kendon). IRAL, 44(2), 103–124.CrossRefGoogle Scholar
Hardison, D. M. (1996). Bimodal speech perception by native and non-native speakers of English: Factors influencing the McGurk effect. Language Learning, 49(1), 213–283.CrossRefGoogle Scholar
Hardison, D. M. (2003). Acquisition of second-language speech: Effects of visual cues, context, and talker variability. Applied Psycholinguistics, 24(4), 495–522.CrossRefGoogle Scholar
Hardison, D. M. (2004). Generalization of computer-assisted prosody training: Quantitative and qualitative findings. Language Learning & Technology, 8(1), 34–52.Google Scholar
Hardison, D. M. (2005). Second-language spoken word identification: Effects of perception training, visual cues, and phonetic environment. Applied Psycholinguistics, 26(4), 579–596.CrossRefGoogle Scholar
Hardison, D. M. (2010). Visual and auditory input in second-language speech processing. Language Teaching, 43(1), 84–95.CrossRefGoogle Scholar
Hardison, D. M. (2019). Technology-based communication success for second-language learners. In Peters, M. A. & Heraud, R. (Eds.), Encyclopedia of Educational Innovation. Singapore: Springer: doi.org/10.1007/978-981-13-2262-4_85-2Google Scholar
Hickok, G. B., Buchsbau, B., Humphries, C., & Muftuler, T. (2003). Auditory-motor interaction revealed by fMRI: Speech, music, and working memory in area Spt. Journal of Cognitive Neuroscience, 15(5), 673–682.CrossRefGoogle ScholarPubMed
Hincks, R. (2015). Technology and learning pronunciation. In Reed, M. & Levis, J. M. (Eds.), The handbook of English pronunciation (pp. 505–519). Hoboken, NJ: Wiley.CrossRefGoogle Scholar
Leather, J. (1990). Perceptual and productive learning of Chinese lexical tone by Dutch and English speakers. In Leather, J. & James, A. (Eds.), New Sounds 90 (pp. 72–97). Amsterdam, Netherlands: University of Amsterdam.Google Scholar
MacLeod, A., & Summerfield, Q. (1990). A procedure for measuring auditory and audio-visual speech-reception thresholds for sentences in noise: Rationale, evaluation, and recommendations for use. British Journal of Audiology, 24(1), 29–43.CrossRefGoogle ScholarPubMed
Massaro, D. W., & Light, J. (2003). Read my tongue movements: Bimodal learning to perceive and produce non-native speech /r/ and /l/. In EUROSPEECH-2003, pp. 2249–2252.Google Scholar
McGrath, M., & Summerfield, Q. (1985). Intermodal timing relations and audio-visual speech recognition by normal-hearing adults. Journal of the Acoustical Society of America, 77(2), 678–685.CrossRefGoogle ScholarPubMed
McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–748.CrossRefGoogle ScholarPubMed
Meltzoff, A., & Moore, M. K. (1993). Why faces are special to infants – on connecting the attraction of faces and infants’ ability for imitation and cross-modal processing. In de Boysson-Bardies, B., de Schonen, S., Jusczyk, P., McNeilage, P., & Morton, J. (Eds.), Developmental neurocognition: Speech and face processing in the first year of life (pp. 211–225). Dordrecht, Netherlands: Kluwer Academic.CrossRefGoogle Scholar
Miller, G. A., & Nicely, P. E. (1955). An analysis of perceptual confusions among some English consonants. Journal of the Acoustical Society of America, 27, 338–352.CrossRefGoogle Scholar
Motohashi-Saigo, M., & Hardison, D. M. (2009). Acquisition of L2 Japanese geminates: Training with waveform displays. Language Learning & Technology, 13(2), 29–47.Google Scholar
Munhall, K. G., Jones, J. A., Callan, D. E., Kuratate, T., & Vatikiotis-Bateson, E. (2004). Visual prosody and speech intelligibility: Head movement improves auditory speech perception. Psychological Science, 15(2), 133–137.CrossRefGoogle ScholarPubMed
Munhall, K. G., & Tohkura, Y. (1998). Audiovisual gating and the time course of speech perception. Journal of the Acoustical Society of America, 104, 530–539.CrossRefGoogle ScholarPubMed
O'Brien, M. G., Derwing, T. M., Cucchiarini, C., Hardison, D. M., Mixdorff, H., Thomson, R. L., … Levis, G. M. (2018). Directions for the future of technology in pronunciation research and teaching. Journal of Second Language Pronunciation, 4(2), 182–206.CrossRefGoogle Scholar
Reisberg, D., McLean, J., & Goldfield, A. (1987). Easy to hear but hard to understand: A lip-reading advantage with intact auditory stimuli. In Dodd, B. & Campbell, R. (Eds.), Hearing by eye: The psychology of lip-reading (pp. 97–113). London, UK: Erlbaum.Google Scholar
Rosenblum, L. (2005). The primacy of multimodal speech perception. In Pisoni, D. & Remez, R. (Eds.), Handbook of Speech Perception (pp. 51–78). Malden, MA: Blackwell.CrossRefGoogle Scholar
Sams, M., Aulanko, R., Hämäläinen, M., Hari, R., Lounasmaa, O. V., Lu, S.-T., & Simola, J. (1991). Seeing speech: Visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters, 127(1), 141–145.CrossRefGoogle ScholarPubMed
Shams, L. A., & Seitz, A. R. (2008). Benefits of multisensory learning. Trends in Cognitive Sciences, 12(11), 411–417.CrossRefGoogle ScholarPubMed
Sueyoshi, A., & Hardison, D. M. (2005). The role of gestures and facial cues in second-language listening comprehension. Language Learning, 55(4), 661–699.CrossRefGoogle Scholar
Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212–215.CrossRefGoogle Scholar
Walden, B. E., Prosek, R. A., Montgomery, A. A., Scherr, C. K., & Jones, C. J. (1977). Effects of training on the visual recognition of consonants. Journal of Speech and Hearing Research, 20(1), 130–145.CrossRefGoogle ScholarPubMed