The majority of studies in second-language (L2) speech processing have involved unimodal (i.e., auditory) input; however, in many instances, speech communication involves both visual and auditory sources of information. Some researchers have argued that multimodal speech is the primary mode of speech perception (e.g., Rosenblum Reference Rosenblum, Pisoni and Remez2005). Research on auditory-visual (AV) input has been conducted more extensively in the fields of infant speech development (e.g., Meltzoff & Kuhl Reference Meltzoff, Kuhl, Lewkowitz and Lickliter1994), adult monolingual processing (e.g., McGurk & MacDonald Reference McGurk and MacDonald1976; see reference in this timeline), and the treatment of the hearing impaired (e.g., Owens & Blazek Reference Owens and Blazek1985) than in L2 speech processing (Hardison Reference Hardison and Pennington2007). In these fields, the earliest visual input was a human face on which lip movements contributed linguistic information. Subsequent research expanded the types of visual sources to include computer-animated faces or talking heads (e.g., Massaro Reference Massaro1998), hand-arm gestures (Gullberg Reference Gullberg2006), and various types of electronic visual displays such as those for pitch (Chun, Hardison & Pennington Reference Chun, Hardison, Pennington, Edwards and Zampini2008). Recently, neurophysiological research has shed light on the neural processing of language input, providing another direction researchers have begun to explore in L2 processing (Perani & Abutalebi Reference Perani and Abutalebi2005).
This timeline on AV input in speech processing is constructed around the following themes based on the studies that have informed the directions this relatively new area of research is taking in L2 acquisition.
A L2 auditory perception studies establishing the groundwork for AV research
B Input from the human face
1 Studies assessing the discernibility of articulatory gestures
2 Native speaker perception
a Eye-tracking studies
3 L2 learner perception
C Input from computer-animations or talking heads
1 Native speaker perception
2 L2 learner perception
D Electronic visual input (e.g., pitch displays, waveforms) in L2 learning
E Input involving hand-arm gestures in language comprehension
F Neurophysiological studies of AV processing
Debra Hardison is Associate Professor in the Department of Linguistics and Languages at Michigan State University. Her research program focuses on auditory-visual integration in spoken language processing, applications of technology in perception and production studies, coordination of gesture and prosody, and relationships among learner affect and language use factors, acoustic measures, and rater assessments of the components of speech involving different tasks. She teaches courses in second language acquisition, research methods, and second-language speech. Publications appear in various journals and edited collections. She is guest editor of the October 2009 issue of Language Learning & Technology.
1 fMRI is a method of capturing and creating an image of the activity in the brain through measurement of the magnetic fields created by the functioning nerve cells in the brain associated with a given task (Hickok et al. 2003).
2ERPs measure the timing of electrical brain responses to a stimulus. The brainwaves correspond to different types of neurocognitive processes (Kelly et al. 2009).
