INTRODUCTION
Today’s L2 learners have easy access to ample auditory training materials that come with wider speaker variability than traditional classroom materials. Computer-assisted language learning programs have an increasing number of mobile phone applications (a.k.a. m-learning) that provide language learners with more spontaneous or contextualized conversational input than traditional textbooks (Chinney, Reference Chinney2006; Kukulska-Hulme et al., Reference Kukulska-Hulme, Pettit, Bradley, Carvalho, Herrington, Kennedy and Walker2011; O’Malley et al., 2005; Rodríguez-Arancón, Arús, & Calle, Reference Rodríguez-Arancón, Arús and Calle2013). Despite the accessibility and amount of auditory input in today’s L2 learning environment, systematic experimental investigations of the effects of input modality and of voice familiarity on L2 syntax acquisition are sparse. The present study examines the effects of auditory modality and training voice on the efficacy of processing instruction (PI), which has been demonstrated to facilitate improvement in both comprehension and production of novel grammatical forms in L2 learners (Benati, Reference Benati2001; Cadierno, Reference Cadierno1995; Comer & DeBenedette, Reference Comer and DeBenedette2010; VanPatten & Cadierno, Reference VanPatten and Cadierno1993; VanPatten & Wong, Reference VanPatten, Wong and VanPatten2004). In particular, we use the causative construction in French to test (a) whether auditory PI training leads to faster (or slower) and more accurate (or inaccurate) processing of spoken sentences than PI training with written input, and (b) whether receiving PI training with a voice that is different from the voice for testing benefits or hinders L2 acquisition of the French causative construction.
BACKGROUND AND MOTIVATION
PREVIOUS RESEARCH ON PI TRAINING: WONG AND ITO (2018)
PI is a well-attested pedagogical intervention that focuses on input that is structured to help L2 learners acquire a novel grammatical construction without requiring production of the target construction (Benati, Reference Benati2001; Cadierno, Reference Cadierno1995; Comer & DeBenedette, Reference Comer and DeBenedette2010; Dracos, Reference Dracos2012; Fernandez, Reference Fernandez2008; Henry, Reference Henry2015; Henry, Culman, & VanPatten, Reference Henry, Culman and VanPatten2009; McManus & Marsden, Reference McManus and Marsden2016; VanPatten & Cadierno, Reference VanPatten and Cadierno1993; VanPatten & Wong, Reference VanPatten, Wong and VanPatten2004). A recent study by Wong and Ito (Reference Wong and Ito2018) tested how PI training changes the way L2 learners of French process spoken input using eye-tracking. The eye-movements of intermediate French learners were examined before and after they went through either traditional instruction (TI) with drill exercises or PI training with structured input activities. In the TI training, participants read prompts such as “Claude/ nettoyer la chambre à Richard” and mechanically produced the target causative structure: “Claude fait nettoyer la chambre à Richard” (Claude makes Richard clean the room). In the PI training, participants read sentences and were asked to identify the person who executes the action (e.g., Claude fait nettoyer la chambre à Richard. Who cleans the room? a. Claude b. Richard). Critically, participants in the PI group never produced the target causative structure during training. Both trainings were given in written formats: Participants either typed the sentence (TI) or selected their answer (PI) using a computer keyboard.
In the two eye-tracking blocks, participants saw two pictures of the quasi-identical scene, which differed only in the roles of two people (Pierre and Marie). For example, one picture showed Pierre making sushi for Marie while the other picture showed Marie making sushi for Pierre. For those critical picture sets, participants heard either (a) a causative or (b) a noncausative sentence:
(1) Pierre fait faire du sushi à Marie.
Pierre makes to make sushi to Marie
‘Pierre makes Marie make sushi.’
(2) Pierre fait du sushi pour Marie.
Pierre makes sushi for Marie
‘Pierre makes sushi for Marie.”
Participants were told to listen till the end of each sentence and click on the picture that matched the sentence they just heard.
The eye-movement patterns before training were very similar between the TI and PI groups, who had never been taught the causative structure. They confirmed the well-documented tendency to interpret the first animate noun as the agent of the action (previously termed the First Noun Principle [FNP] by VanPatten, Reference VanPatten and VanPatten2004). This is a common strategy of typically developing children and individuals with a developmental disorder or aphasia for comprehending sentences with a noncanonical word order structure such as passive forms, scrambled sentences, object clefts, and object-relative clauses (Grodzinsky, Reference Grodzinsky2000; Leonard, Wong, Deevy, Stokes, & Fletcher, Reference Leonard, Wong, Deevy, Stokes and Fletcher2006; c.f., Meyer, Mack, & Thompson, Reference Meyer, Mack and Thompson2012; Swinney & Zurif, Reference Swinney and Zurif1995). Guided by this strategy, shortly after they heard the sentence-initial noun (e.g., Pierre in [a] and [b]), both TI and PI groups started looking at the picture in which the first noun was the agent of the action, and the looks to the (first noun = agent) picture kept increasing throughout the sentence regardless of the structure. Accordingly, participants clicked on the incorrect picture for the causative sentences and the correct picture for the noncausative sentences more than 95% of the time.
After a relatively short training with 48 items (half causative and half noncausative), the response patterns diverged drastically between the groups: The accuracy of picture selection improved from 3.9% to 11% in the TI group, and from 4.6% to 62% in the PI group. This difference in interpretation accuracy due to training type was in accordance with the previous studies that reported the effect of PI with offline measures (Benati, Reference Benati2001; Cadierno, Reference Cadierno1995; VanPatten & Cadierno, Reference VanPatten and Cadierno1993; VanPatten & Wong, Reference VanPatten, Wong and VanPatten2004, inter alia).
Importantly, the eye-tracking data of Wong and Ito (Reference Wong and Ito2018) also demonstrated the change in online responses to the input in the PI group but not in the TI group. Despite the small but significant increase in the picture-selection accuracy, the TI group maintained the same pattern of eye-movements, that is, the bias toward the (first noun = agent) picture, till the end of the sentence. In the PI group, looks to the incorrect picture for the causative structures started decreasing, that is, the gazes were shifted to the correct picture, during the sentence-final prepositional phrase (à Marie).
Although this change in the eye-movements provided novel evidence for the effect of PI on online sentence processing, Wong and Ito (Reference Wong and Ito2018) reported that the timing of the shift was not as early as predicted. The infinitive verb that follows the main verb (e.g., faire after fait in example [1]) is the first critical grammatical cue that differentiates the causative structure from the noncausative structure. Thus, if learners had mastered the causative structure after the short training, they should have been able to swiftly respond to the infinitive verb and shift their gazes much earlier in the sentence. The second experiment of Wong and Ito (Reference Wong and Ito2018) gave explicit written explanation of the grammatical structure of French causative sentences at the beginning of the TI and PI training. This explicit instruction (EI) led to the gaze shift in both groups, but again, the reduction of the biased looks was observed during the sentence-final prepositional phrase (à Marie). Thus, Wong and Ito concluded that EI did not have an additive effect to facilitate online processing of the causative structure in L2 French learners. In both experiments, the modality of training was written while the modality of testing was auditory. The present study uses the exact same experimental design as Wong and Ito (Reference Wong and Ito2018), except that the PI training was given with auditory input. The stimuli are identical to Wong and Ito (Reference Wong and Ito2018) such that the picture-selection accuracy data and the eye-tracking data can be directly compared across studies.
The present study aims to replicate the gaze shift with PI training reported by Wong and Ito (Reference Wong and Ito2018), and further tests whether the shift can appear earlier, that is, whether L2 learners can process the target structure quicker, with PI training using auditory input instead of written input.
EFFECTS OF MODALITY
There is evidence from cognitive psychology that the modality of presentation of stimuli affects how that stimuli is recalled (Penney, Reference Penney1980, Reference Penney1989; Penney & Burr, Reference Penney and Burr1986). Penney (Reference Penney1980) showed that when different types of stimuli (numbers, words in different languages, words of different semantic categories, etc.) are presented both auditorily and visually in an alternating manner, participants better recall the stimuli organized by modality of presentation. Ronnberg and colleagues (Reference Ronnberg, Nilsson and Ohlsson1982) examined how participants organized words in lists that contrasted two modalities (auditory and visual), two languages (Swedish and English), and two semantic categories. They found that organization by modality was consistently more frequent than organization by language or by semantic category. Similar effects of modality on recall were reported also by Hede (Reference Hede1973), Madsen et al. (Reference Madsen, Rollins and Senf1970), and Penney and Burr (Reference Penney and Burr1986). These studies suggest a possible advantage of using the same (auditory) modality input for training before testing the changes in the L2 learners’ responses to the target causative structure with auditory stimuli.
Another set of studies, however, suggest that implicitly learned rules or patterns may or may not surface depending on the modality of testing, and demonstrate that the effect of learning may be more difficult to capture with auditory stimuli than with written stimuli. Hama and Leow (Reference Hama and Leow2010), Kerz, Wiechmann, and Riedel (Reference Kerz, Wiechmann and Riedel2017), and Williams (Reference Williams2005) investigated whether novel grammatical forms in a semi-artificial language can be learned in the absence of awareness. Williams (Reference Williams2005) created target structures with artificial determiners that encoded both distance (near vs. far) and animacy (animate vs. inanimate). Participants were told which determiners marked near or far objects while receiving no instruction on the animacy rules. During training, they listened to sentences that contained a determiner and indicated whether the determiner meant “near” or “far.” During testing, participants read the sentence fragment (e.g., The lady spent many hours sewing…) and selected between two written noun phrases (gi cushions/ro cushions) to complete the sentence. Later, they were asked what criteria they used to make their decisions. Results revealed that a good number of “unaware” participants who never mentioned animacy as their criteria selected the correct noun phrase significantly above chance level. Hama and Leow (Reference Hama and Leow2010) revised Williams’s experimental design and had the participants listen to sentences (e.g., I had to read the manual to find out how to adjust [beep] clock up on the wall.) and choose the missing determiner from four written choices (2 distance × 2 animacy). Unlike Williams (Reference Williams2005), Hama and Leow’s “unaware” participants did not select the correct determiner for animacy above chance level. They also failed to demonstrate their implicit knowledge of animacy-marking determiners in their sentence productions. A study by Kerz et al. (Reference Kerz, Wiechmann and Riedel2017) used a similar design but trained and tested participants with text only. Kerz et al. showed that unaware participants chose the correct determiner for animacy above chance level (although not as highly as those who received EI), indicating that written input may lead to better learning of implicit rules than auditory input.
Another important factor that may underlie the modality-related difference in learning is the function of attention and memory. L2 learners tend to better recall main ideas after listening to a narrative than after reading it (Absalom & Rizzi, Reference Absalom and Rizzi2008; Lund, Reference Lund1991; Park, Reference Park2004), but they tend to remember more details after reading than listening (Lund, Reference Lund1991; Park, Reference Park2004). Leow (Reference Leow1993, Reference Leow1995) investigated whether simplification of input (through reduction of number of words per sentence and occurrences of embedded clauses) affects the intake of target linguistic forms (present perfect or subjunctive) in L2 Spanish learners. When tested with written input (Leow, Reference Leow1993), performances on multiple-choice sentence recognition task did not show any effect of input simplification regardless of the target form. However, with auditory input (Leow, Reference Leow1995), participants better recognized sentences that were presented in present perfect forms than in subjunctive forms. Leow (Reference Leow1995) argues that present perfect forms may have been more salient in auditory input (as it included dual cues from the auxiliary verb “haber” [“to have”] and the past participle form ending in –ado/–ido) than the subjunctive forms that had a single inflectional cue in the verb. The target structure of the present study has both an earlier cue (faire + infinitive) and a later cue (à + agent). Thus, if these cues are sufficiently salient in auditory input, they should facilitate the representation of target syntactic form. In contrast, if reading makes L2 learners attend to the details of the forms, written input may be more effective for learning a syntactic frame as they provide consistent visual cues across training sentences.
Finally, we must consider the availability of cognitive resources for predicting the timing of cue processing that should be revealed in eye-tracking data. Wong (Reference Wong2001) showed that auditory input and written input may differently impact the processing of forms and meaning using a task of free recall of idea units with L2 English learners. Participants had to either read or listen to a passage for content only; read or listen to a passage for content while attending to the content word “inflation”; or read or listen to a passage for content while attending to the definite article “the.” Results showed that the group that received auditory input had difficulty in simultaneously attending to the content of the passage and the word “the,” while the group that received written input did not show such difference across conditions. This finding echoed the results of VanPatten (Reference VanPatten1990), who tested L2 learners of Spanish with auditory stimuli that required division of attention between two levels of information (form and message). These studies indicate that tasks that demand attention to both the form and meaning of auditory input may present a challenge to L2 learners. As stated in VanPatten’s (Reference VanPatten and VanPatten2004) primacy of meaning principle, when processing resources are limited, L2 learners tend to process input for meaning before form. Given that auditory L2 processing is more taxing on cognitive resources (e.g., short-term memory: Brown & Hulme, Reference Brown, Hulme and Harris1992), learners may miss the early syntactic cue more when they are trained with auditory than with written input. If this is the case, the early shift in the gaze may not be observed after auditory PI training.
To summarize, the literature suggests that input modality impacts the outcome of L2 learning but the direction of the effect may depend on multiple factors. While the consistency of auditory modality across training and testing may benefit learners, implicit grammar learning may be more effective with written input that directs learners’ attention to the details (i.e., forms) than auditory input that may make them attend more to the main idea. Additionally, when limited L2 cognitive resources are primarily allocated for processing sentential meaning during PI training, surface grammatical forms may not be quickly acquired unless they are highly salient. Importantly, unlike many of the aforementioned studies, the present study does not test participants with explicit form judgment tasks. Instead, we examine changes in participants’ spontaneous responses to grammatical cues as a result of PI training using eye-tracking. The novelty of this approach is in the evaluation of the effect of input modality with the measure of spontaneous online processing of speech input.
EFFECT OF VOICE FAMILIARITY IN L2 ACQUISITION
Past research on the effect of voice variability on L2 acquisition has suggested both advantages and disadvantages of multitalker input on novel category formation. Studies on the acquisition of the English /l/ versus /ɹ/ distinction by Japanese listeners demonstrated that training with multitalker (as compared to single-talker) input not only facilitates target category formations but also allows for generalization of categorical features to a new talker (Bradlow, Pisoni, Akahane-Yamada, & Tohkura, Reference Bradlow, Pisoni, Akahane-Yamada and Tohkura1997; Lively, Logan, & Pisoni, Reference Lively, Logan and Pisoni1993; Logan, Lively, & Pisoni, Reference Logan, Lively and Pisoni1991). Bradlow et al. (Reference Bradlow, Pisoni, Akahane-Yamada and Tohkura1997) additionally showed that learners who received multitalker input improved their pronunciations of /ɹ/ and /l/ after three weeks of training, that is, variability in the input enabled the transfer of perceptual knowledge to production of target categories. These studies claim that variability in L2 input is effective because learners are exposed to a wider range of acoustic realizations of target categories, while single-talker input may lead to robust representations of the speaker-specific traits that may hinder the generalization to input from novel speakers.
While similar effect of talker variability has been demonstrated for Mandarin tone perception by English speakers (Wang, Spence, Jongman, & Sereno, Reference Wang, Spence, Jongman and Sereno1999), Japanese geminate perception by Dutch speakers (Sadakata & McQueen, Reference Sadakata and McQueen2014), and Spanish word learning by English speakers (Barcroft & Sommers, Reference Barcroft and Sommers2005), other studies suggest that the effect of input variability may depend on the types of target categories (Wade, Jongman, & Sereno, Reference Wade, Jongman and Sereno2007), and it is also susceptive to listeners’ perceptual aptitudes (Perrachione, Lee, Ha, & Wong, 2011; Sadakata & McQueen, Reference Sadakata and McQueen2014) and their L1 phonetic inventories (Barcroft & Sommers, Reference Barcroft and Sommers2014). Perrachione et al. (Reference Perrachione, Lee, Ha and Wong2011) assessed participants’ perception of pitch contours before they learned a set of monosyllabic pseudowords that carried level, rising, and falling pitch contours. Only those who performed well in the pitch contour perception test (PCPT) benefited from multitalker input and better identified the objects associated with each pseudoword spoken by a novel talker. Those who had lower scores in PCPT performed worse with the multitalker input, failing to identify words spoken by a novel talker. Sadakata and McQueen (Reference Sadakata and McQueen2014) replicated the hindering effect of talker variability in Dutch listeners with weaker aptitude for learning tones, who exhibited more difficulty in learning disyllabic pseudo-Mandarin words with higher-variability input than with lower-variability input during the training. Barcroft and Sommers (Reference Barcroft and Sommers2014) demonstrated that the higher variability in fundamental frequency (f0) introduced by multiple talkers facilitated novel word learning (in Russian) only in learners whose L1 has lexical tones (Zapotec) but not in those whose L1 does not use f0 as lexical cues (Spanish).
Note that the majority of past work on talker variability in L2 processing investigated acquisition of foreign phonemic/prosodic categories and lexical items and thus the stimuli were mostly isolated words and short phrases. Other studies, however, suggest that talker variability in sentential processing may have both facilitative and hindering effects. Bradlow and Bent (Reference Bradlow and Bent2008) tested L1 English word recognition accuracy with Chinese-accented speech, and showed that training with input from five different speakers led to significantly higher improvement for the speech from a novel speaker of the same accent than the same amount of training with input from a single speaker. In contrast, Sadakaa and McQueen (Reference Sadakata and McQueen2014) showed that higher variability in the training input did not facilitate the correct identification of target words embedded in sentences, regardless of learners’ perceptual aptitude.
In sum, past research on talker variability in L2 processing suggests a general merit of exposure to a variation of novel categories, while the degree of success may depend on target categories, learners’ aptitude, and their L1 background. To our knowledge, no study has reported the effect of training voice on the acquisition of novel syntactic constructions, especially within the PI framework. The present study tests whether the voice of training (which is either the same or different from the testing voice) makes any difference in the learning of the causative structure for L2 French learners. On the one hand, we may hypothesize that receiving training with a novel voice may benefit learners due to the variation-to-generalization principle (e.g., Bradlow et al., Reference Bradlow, Pisoni, Akahane-Yamada and Tohkura1997). Critically, the present stimuli sentences are made of words that are known to the participants and thus the task does not involve processing any acoustically novel categories, which should not impose much cognitive load and may ease the extraction of target causative construction. On the other hand, training with a different voice may hinder the acquisition of target construction. Studies on speech comprehension with unfamiliar/nonnative/foreign-accented voices demonstrate the cost of unfamiliarity to the speaker-specific phonetic characteristics during word recognition in both adults (Bent, Reference Bent2014; Bent & Bradlow, Reference Bent and Bradlow2003; Bent & Holt, Reference Bent and Holt2013; Bradlow & Bent, Reference Bradlow and Bent2008; Clarke & Garrett, Reference Clarke and Garrett2004; Munro, Reference Munro1998) and children (Barker & Meyer Turner, Reference Barker and Meyer Turner2015; Bent, Reference Bent2014; Creel, Reference Creel2012; Nathan & Wells, Reference Nathan and Wells2001; Nathan, Wells, & Donlan, Reference Nathan, Wells and Donlan1998). If the general effect of unfamiliar voice also leads to slower, less accurate L2 speech processing, learners may be hindered by a voice switch and benefit from the same-speaker input, especially while they learn a novel grammatical construction with brief training.
EXPERIMENT 1
The first experiment traced eye-movements while learners heard the stimuli produced by a female native French speaker for the dichotomous scene selection task before and after the auditory PI training. The auditory PI training was given with either the same female voice or with a different male voice.
RESEARCH QUESTIONS
RQ1. To what extent does auditory PI training, as opposed to written PI training, modulate the improvement in picture selection? Do learners better learn the target causative structure with auditory than with written input?
RQ2. To what extent does auditory PI training, as compared to written PI training, modulate the eye-fixation patterns? Do participants more quickly respond to the earlier syntactic cues (i.e., infinitive verb after the main verb fait) when they are trained with auditory than with written input?
RQ3. To what extent do learners benefit from training with the same voice than from training with an unfamiliar, different voice? Does the overall familiarity with the input voice affect the degree of the effect of PI?
METHOD
PARTICIPANTS
Fifty-four French learners were recruited from a third-semester college French course (exposed to standard French) at Ohio State University. Participants received extra credit for their participation. They were recruited from the equivalent courses as in Wong and Ito (Reference Wong and Ito2018) to control overall L2 proficiency level across studies. Importantly, the target causative structure was not taught in these courses. Participants were randomly assigned to the same-voice or the different-voice training groups. Within each group, they were further randomly assigned to one of the four presentation lists. Sixteen participants spoke a language other than English and French with higher than middle-level proficiency, but their data did not change the overall group performance and thus were included in the analyses.
MATERIALS
EYE-TRACKING STIMULI
Ninety-six picture pairs that depict quasi-identical scenes were co-presented with pre-recorded auditory sentences. The visual and the auditory stimuli were identical to those used in Wong and Ito (Reference Wong and Ito2018). Each picture pair had the same two characters: one male (Pierre) and one female (Marie). Across the two quasi-identical scenes, the actions of the two characters (and thus their positions) were swapped: for example, Marie is doing the dishes for Pierre on one side while Pierre is doing the dishes for Marie on the other side (Figure 1).
Example slide.
Twenty-four picture pairs were presented as critical items in the causative and noncausative conditions. Two causative sentences (e.g., Pierre/Marie fait faire la vaisselle à Marie. “Pierre/Marie makes Marie/Pierre do the dishes.”) and two noncausative sentences (e.g., Pierre/Marie fait la vaisselle pour Marie/Pierre. “Pierre/Marie does the dishes for Marie/Pierre.”) were recorded to be presented with the same picture pair across four presentation lists. Among the 72 filler items, 24 items were presented with active (e.g., Pierre/Marie embrasse Marie/Pierre. “Pierre/Marie kisses Marie/Pierre.”) and passive (e.g., Pierre/Marie est embrassée par Marie/Pierre. “Pierre is kissed by Marie/Pierre.”) sentences. The remaining 48 items presented two contrastive actions (e.g., Pierre/Marie lit un roman, et Marie/Pierre fait un dessin. “Pierre/Marie reads a book, and Marie/Pierre draws a picture.”). To introduce variation in syntactic complexity among the stimuli, one third of these items presented an object-relative clause (e.g., Pierre/Marie regarde le film que Marie/Pierre a loué. “Pierre/Marie watches the film that Marie/Pierre rented.”).
A female native speaker from Rennes recorded a total of 288 sentences (48 causative, 48 noncausative, 48 active, 48 passive, 64 conjoined, 32 object relative clause) at 44.1 KHz sampling rate using Praat (Boersma & Weenink, Reference Boersma and Weenink2016). The results of duration measures for the critical causative items are summarized in Table 1 (see Experiment 2 for the voice comparison).
Experiments 1 and 2: Duration analysis of causative test stimuli (n = 96)
TRAINING MATERIALS
The training materials consisted of 48 sentences (24 causative and 24 noncausative sentences) that were used as written input in Wong and Ito (Reference Wong and Ito2018) for the PI training. Those sentences were recorded by the same female speaker who had recorded the eye-tracking stimuli, and also by a male native French speaker from Marseille, at 44.1 KHz sampling rate using Praat (Boersma & Weenink, Reference Boersma and Weenink2016). The amplitude of the recorded sentences was equalized.
While both speakers produced the training sentences clearly and three research assistants confirmed the intelligibility of their contents, the two speakers showed different speech rates for the critical causative sentences. The female speaker produced significantly longer subject noun phrase (NP), main verb fait, infinitive verb, and sentence-final prepositional phrase (PP) (à somebody) than the male speaker (Table 2). Thus, both the earlier and later cues to the target structure had longer duration in the female than in the male speech. If this durational difference enhances the clarity of critical grammatical cues, the voice familiarity effect can be potentially confounded with the durational advantage.
Experiments 1 and 2: Duration analysis of training causative training stimuli (n = 48)
TASKS AND PROCEDURE
All sessions consisted of (a) before-training eye-tracking, (b) auditory PI training (with either the same [female] or different [male] voice), and (c) after-training eye-tracking blocks. After signing a written consent and answering a background questionnaire, each participant’s gazes were quickly calibrated using the Clearview program. Both eye-tracking blocks presented 48 items (6 causatives, 6 noncausatives, and 36 filler items) using EPrime 1.0 (Schneider, Eschman, & Zuccolotto, Reference Schneider, Eschman and Zuccolotto2002). Each trial first presented a pair of pictures, and after 1.5 seconds, an auditory sentence in the female voice was played through a headphone set. Participants were asked to listen till the end of each sentence and click on the picture that matched the sentence. No feedback was given after the response. Each mouse click automatically advanced to the next trial. Participants’ gaze locations were continuously recorded at 50 Hz by Tobii 1750. Fixations on the two rectangular areas of interest (Figure1) were coded as correct or incorrect gazes.
After the first eye-tracking block, participants were given a brief break and were instructed on the next task. They were told to listen to each sentence carefully and answer the question that appeared on the screen by pressing a key on a keyboard. Each trial first showed a word “Listen” for one second, and then played a sentence (e.g., Simon fait faire du shopping à Henri.) with a blank screen. After the sentence was played, a question asking for the agent of the described action appeared on the screen (e.g., Who shops?). Participants responded by pressing a key that corresponded to choices (a. Henri) or (b. Simon). After they responded, the correct answer was shown at the bottom of the screen, highlighting the answer in red (e.g., Figure 2). A total of 48 training items were presented in four blocks of 12 trials. Roughly half of the participants heard the training items in the same female voice as in the preceding eye-tracking block. The other half heard them in a male voice. The order of item presentation was identical to the written presentation in Wong and Ito (Reference Wong and Ito2018).
Example PI-training slide.
Participants were given a quick break after the training block and went through a quick calibration and another set of 48 picture-selection items. The after-training eye-tracking stimuli (none of which were repeated from the before-training block) were all given in the same female voice as the before-training block. Throughout the experiment, participants received no EI on the target causative structure.
RESULTS
Data from one participant was excluded from the analyses due to task incompletion.
The following analyses include data from 26 participants in the same-voice group and 27 participants in the different-voice group. The average accuracy of auditory PI training was 54.6% for the same-voice group and 62.7% for the different-voice group, which did not differ statistically (t = 1.37, p = 0.18). Upon debriefing, one participant in the same-voice group and three participants in the different-voice group reported that they noticed the study had to do with the interpretation of the agent. Because these participants performances did not affect group performances, they were analyzed with the rest of the groups. Picture accuracy responses were coded as 1 (accurate) or 0 (inaccurate) and submitted to mixed-effects logistic regressions that tested the effects of block, PI-training voice (same vs. different), and PI-training modality (written vs. auditory). The fixation data were coded as log of the ratio between the looks to the correct picture and those to the incorrect picture for each analysis window and submitted to mixed-effects regressions with the same predictor factors as the accuracy analyses. The regression models were tested using R 3.4.1 lmer package. The effects of particular predictor factors that showed the coefficients higher than 1.5 were tested using model comparisons (Baayen, Reference Baayen2008).
Picture-Selection Accuracy
The previously reported effect of PI training in Wong and Ito (Reference Wong and Ito2018) was replicated with auditory training. Before training, both groups exhibited very low accuracy in picture selection for the causative sentences (same: 5.6%; different: 4.9%): that is, they mostly clicked on the incorrect (first noun = agent) picture, confirming the prediction of the FNP. After auditory PI training, both groups showed a large improvement, with the same-voice group showing a numerically higher average score than the different-voice group (same: 54.3%; different: 47.4%: see Figure 3) but did not differ statistically. Mixed-effects logistic regression model in (3)
(3) Response ∼ Training + Block + Training: Block + (1 + Block|Subj) + (1|Item), family = binomial
Experiment 1 picture-selection accuracy.
confirmed no group difference before training (z = .27, p = .78), a significant effect of block (after vs. before: z = 4.91, p < .001), and no effect of training voice (same vs. different: z = 0.62, p = .54) and no interaction between the two (z = 0.28, p = .78). Thus, auditory PI training was equally effective for learning the target structure regardless of training voice (despite the difference in speech rate).
To test the effect of modality of training, the performances were directly compared between the data from Wong and Ito (Reference Wong and Ito2018) Experiment 1’s PI group (n = 30) and the present different-voice group (n = 27), who had the same amount of exposure to the same female voice across the eye-tracking blocks with no EI on the target structure. The mixed-effects logistic regressions with training modality and block as predictor factors and the same random effect terms as in (3) in the preceding text confirmed no group difference before training (z = .49, p = .62) and revealed a robust effect of block (z = 4.32, p < .001), no effect of training modality (z = -.45, p = .65), and no interaction between the two (z = 1.03, p = .3). While the group that received written input made a numerically larger improvement (written: 4.6% → 62%), the auditory input led to a statistically equivalent effect of PI training (auditory: 4.9% → 47.4%) to improve picture-selection accuracy.
Eye-Movements
Eye-movements before training confirmed the tendency to interpret the first noun as the agent in both groups. Figure 4 shows the shift in fixation likelihood during the target causative sentences for both groups (left: same-voice; middle: different-voice). The fixation functions are expressed in the log of ratio between looks to the correct picture and looks to the incorrect picture such that an upward deviation from the center “0” line indicates more looks toward the correct picture while a downward deviation indicates more looks to the incorrect picture.
Experiment 1 (same vs. different voice) and written PI (Wong & Ito, Reference Wong and Ito2018) fixation patterns (upper = before, lower = after PI training): n = participants; N = observations.
Before training (upper panels of Figure 4), both groups steadily showed more looks to the incorrect (fist noun = agent) picture after the main verb fait till the end of the sentence. To our surprise, the eye-movements in the block after the training (bottom panels of Figure 4) also showed persistent looks to the incorrect picture. A visible reduction of looks to the incorrect picture (i.e., the upward shift toward the center line) did not appear until after the sentence offset in both groups. Even for the same-voice group, which heard the critical grammatical cues with longer duration, gaze responses to the target structure did not become faster after training. For the statistical analysis, an empirical logit (i.e., log of the ratio between the correct and incorrect fixations) was calculated for each window. Mixed-effects models revealed that there was no reduction of incorrect looks (or increase in correct looks) after training throughout the target sentences (Table 3). As shown in the rightmost column of Figure 4, the reduction of incorrect looks had been observed more clearly during the last PP (à Marie/Pierre) for the written PI training (z = 1.91, p = .06) in Wong and Ito (Reference Wong and Ito2018). Thus, although auditory PI training led to equivalent improvement in picture-selection accuracy, it did not have an impact on how participants responded to the auditory cues to the target causative structure. In fact, the shift of looks to the correct picture happens even later with auditory PI training than with written PI training.
Experiment 1: Summary of mixed-effect regressions on the fixation logits: (n = 53) Elogit ∼ Training-Voice + Block + Training-Voice: Block + (1 + Block|subj) + (1 + Training-Voice|item)
Note: p values are indicated only for the factors for which the effect was tested using a model fit comparison.
DISCUSSION
While Experiment 1 largely replicated the effect of PI training of Wong and Ito (Reference Wong and Ito2018) with auditory input, the consistency in stimuli presentation modality was not particularly beneficial to learners (RQ1). Instead, the auditory training led to a statistically equivalent improvement in the picture-selection accuracy as compared to the written training.
To our surprise, the eye-tracking data showed resilient FNP-driven fixations even after training with auditory input. A clear shift toward the correct picture was not observed even during the sentence-final PP (à Marie) (RQ2). As compared to the written-input group, learners trained with auditory input seemed to take longer to overcome the FNP bias. This finding could perhaps be explained by the position of the critical grammatical elements in the sentence. As stated in VanPatten’s (Reference VanPatten, Wong and VanPatten2004) sentence location principle, “[L]earners tend to process items in sentence initial position before those in final position and those in medial position” (p. 14). In other words, grammatical elements are least salient position in medial position followed by those in final position. In the French causative construction, the first critical element is in medial position and the second critical element is in final position. In a study on partial word form learning in lexical input processing, Barcroft (Reference Barcroft2008) found that L2 learners recalled more word fragments in word-initial position. They had more difficulty with fragments in word-final position and the most difficulty with fragments in word-medial position. Therefore, the fact that participants in Wong and Ito (Reference Wong and Ito2018) (using written input) showed a shift to the correct picture around the point of the second critical grammatical cue (which occurred at the end of the sentence) but not during the first cue (which occurred in the middle of the sentence) may in part be due to the position of the two critical cues that had an impact on their saliency. It is possible that in the current experiment with auditory input, participants had even more difficulty processing both critical cues, even the cue that occurred at the end of the sentence because the auditory input taxed their processing resources even more.
These results indicate that auditory PI training can be equally efficient as written PI training to improve offline interpretational judgments, while it may not immediately enhance sensitivity to novel grammatical cues in speech. The current results rather suggest that written input may be more efficient for building sensitivity to grammatical forms. This difference may be attributed to either the general effect of written input to draw attention to the details (Kerz et al., Reference Kerz, Wiechmann and Riedel2017; Lund, Reference Lund1991; Park, Reference Park2004; Williams, Reference Williams2005) or to the general difficulty of processing details in auditory L2 input due to learners’ overall attention to main ideas (Absalom & Rizzi, Reference Absalom and Rizzi2008; Lund, Reference Lund1991; Park, Reference Park2004). While the design of the present study does not allow us to determine which of these two accounts better explains the difference in the effect of input modality on online eye-movement responses, our results clearly demonstrate that a brief PI training, with either written or auditory input, does not lead to immediate changes in responses to early grammatical cues (faire + infinitive verb).
As for the effect of training voice, the results hinted at a numerical advantage for hearing the familiar voice throughout the experiment (which may be partially due to the overall slower speech rate of the female voice in the training), yet this trend did not bear statistical significance (RQ3). Despite the consistency in input modality and the higher familiarity with the testing voice with slower speech rate for the critical grammatical cues than the other voice during the PI training, the same-voice group’s performance was not better than that of the written-input group. This may appear surprising given the past report on swift adaptation to novel accents (Bradlow & Bent, Reference Bradlow and Bent2008; Dahan, Tanenhaus, & Chambers, Reference Dahan, Tanenhaus and Chambers2002), yet may add to the fact that adaptation to spoken L2 input generally takes days of training (Bradlow et al., Reference Bradlow, Pisoni, Akahane-Yamada and Tohkura1997; Lively et al., Reference Lively, Logan and Pisoni1993; Logan et al., Reference Logan, Lively and Pisoni1991). Given the general difficulty to learn novel L2 grammar solely with auditory input, Experiment 2 tests how much a change in test voice modulates the degree of difficulty.
EXPERIMENT 2
The second experiment was conducted with the identical design and procedure as Experiment 1. The only change was the voice of testing: The male voice used for different-voice training in Experiment 1 was used here as the test voice for the picture-selection tasks. Thus, in Experiment 2, participants in the same-voice group heard the same male voice throughout the study while those in the different-voice group heard the female voice only during the training.
RESEARCH QUESTIONS
RQ4. To what degree does the change in the test stimuli to the male voice modulate the improvement in the picture-selection task as compared to Experiment 1?
RQ5. Does the eye-tracking task with the male voice stimuli confirm a slow shift in gaze after auditory PI training?
PARTICIPANTS
Sixty-three French learners were recruited from the same third-semester college French course as in Experiment 1 for extra credit. None of them participated in any of the preceding experiments and had no previous formal instruction on the target structure. They were randomly assigned to the same-voice or different-voice groups: Within each group, they were randomly assigned to one of the four presentation lists. The data includes those from 14 participants who spoke languages other than English and French with higher than middle-level proficiency.
EYE-TRACKING STIMULI AND TRAINING MATERIALS
The visual stimuli for the picture-selection task were identical to those in Experiment 1. The exact same 288 sentences that were copresented with the pictures were recorded by the male speaker from Marseille whose voice was used for the different-voice training in Experiment 1. The training materials were identical to those in Experiment 1.
The duration analysis of the test stimuli revealed differences for different parts of speech as compared to the female speech used in Experiment 1. The male speaker produced significantly longer subject NP and later part of sentences after the main verb—critically the sentence-final PP (à somebody), as compared to the female speaker; whereas the earlier part of the sentence—fait + infinitive verb that provide the first cue to the causative structure—was significantly shorter than the female speaker (Table 1). Although these differences between speakers were not intended and the durational measures alone cannot specify the whole acoustic characteristics of the input, the observed contrast in test stimuli duration may impact the outcome of the experiment. Experiment 1 showed that the durational difference (i.e., longer grammatical cues in the female voice) in the PI-training material did not lead to group differences in either picture-selection accuracy or eye-movements. However, the durational difference in the test voice may have a direct impact on the responses to the picture-selection task. For example, longer later cue (à somebody) in the male voice may be easier to process than the longer earlier cue (fait infinitive verb) in the female voice for the L2 learners. If the location of the slower speech rate in the test voice robustly impacts processing of the target structure, participants of Experiment 2 may perform better than those of Experiment 1, regardless of the training voice. In contrast, if the durational salience of grammatical cues in the test voice interacts with the familiarity with the voice that is acquired through PI training, then the same-voice group may perform better than the different-voice group in the after-training block in Experiment 2.
TASKS AND PROCEDURE
The picture-selection task and the training task were identical to those in Experiment 1. Again, participants never received an EI on the target causative construction.
RESULTS
Sufficient data from three participants could not be obtained due to technical failure. The following analyses include data from 60 participants (same voice: 30; different voice 30). The average accuracy of auditory PI training was 65% for the same-voice group and 60% for the different-voice group, and they did not differ statistically (t = 0.88, p = 0.38). Two participants from the same-voice group and one participant from the different-voice group noticed that the testing had to do with the interpretation of the agent, but their responses did not change the group performances so were included in the analyses.
Picture-Selection Accuracy
As in Experiment 1, both groups were mostly incorrect in selecting pictures for the causative sentences before training (same: 1.7%; different: 0.5%) and showed a large improvement after training (same: 75%; different: 60%: see Figure 5).
Experiment 2 picture-selection accuracy.
A logistic regression on the picture-selections confirmed no group difference before training (z = -.98, p = .33), a main effect of block (after vs. before: z = 7.97, p < .001), a marginal effect of training voice (same vs. different: z = 1.96, p = .05), and no interaction between the two (z = -.52, p = .61).
Because participants in Experiment 2 exhibited numerically higher performances than those in Experiment 1 in the after-training block despite their numerically lower performances in the before-training block, we compared the performances between Experiment 1’s Different-Voice group (female→male→female) and Experiment 2’s Same-Voice group (male→male→male), which differed only in the test voice. The logistic regressions revealed a main effect of block (z = 5.37, p < .001), null effect of test voice (z = .86, p = .39), and a significant interaction between the two (z = 2.48, p = .01). Thus, while the male test voice did not have an overall effect of leading to higher picture-selection accuracies as compared to the female test voice, it led to a larger improvement after training with the same voice. We also tested the effect of voice between Experiment 1’s and Experiment 2’s Same-Voice groups (i.e., female→female→female vs. male→male→male) and found a robust effect of block (z = 8.142, p < .001), a null effect of test voice (z = 1.3, p = .19), and a significant interaction between the two (z = 3.13, p < .01): Participants improved more from hearing the same voice throughout the experiment with the male than with the female voice. Interestingly, when the Different-Voice groups of the two experiments were compared (i.e., female→male→female vs. male→female→male), there was no interaction between the test voice and block (z = 1.63, p = .1): Switching voices led to equivalent, relatively moderate improvement in performance regardless of voice.
Although both the test voice and the training modality differed, we also compared Experiment 2’s Same-Voice group performances (average 75% after training) to those of Wong and Ito (Reference Wong and Ito2018) Experiment 1’s written PI group (average 62% after training) to examine whether the magnitude of improvement with auditory training was still comparable to that with written input. A robust effect of block (z = 4.73, p < .001), no effect of experiment (z = .38, p = .7), and no interaction between the two (z = 1.63, p = .13) were confirmed, suggesting that the effect of the auditory PI training with the same male voice was equivalent to the written PI training with the female test voice.
Eye-Movements
As in Experiment 1, a resilient FNP bias after the main verb fait was confirmed before training in both groups (Figure 6, upper panels).
Experiment 2 same vs. different voice fixation patterns (upper = before, lower = after PI training): n = participants; N = observations.
Additionally, as in Experiment 1, the strong FNP bias remained after training in both groups, as shown in the large downward shift during the object NP after the infinitive verb. However, the upward shift to the correct picture was visible in the sentence-final PP (à Marie/Pierre) in both groups (Figure 6, lower panels). The statistical analysis confirmed the stronger FNP bias (i.e., more looks to incorrect picture) in the object NP window for the after-training block (as compared to the before-training block), and an effect of training voice for the final PP (Table 4).
Experiment 2: Summary of mixed-effect regressions on the fixation logits: (n = 60) Elogit ∼ Training-Voice + Block + Training-Voice: Block + (1 + Block|subj) + (1 + Training-Voice|item)
Note: p values are indicated only for the factors for which the effect was tested using a model fit comparison.
Subsequent analyses confirmed a stronger FNP bias (i.e., more looks to incorrect picture) after the PI training for the object NP window in Different-Voice group (t = -2.07, p < .05) but not in Same-Voice group (t = -1.25, p = .18). The two groups’ FNP bias during the final PP did not differ for the before-training block (t = -1.38, p = .16), while it was marginally weaker for Same-Voice group than for Different-Voice group in the after-training block (t = 1.68, p = .09).
DISCUSSION
While Experiment 2 largely replicated the results of Experiment 1 in showing the effect of auditory PI training on picture selection, there are some informative differences across experiments. Although the before-training performance was equally low, participants who were trained and tested with the same male voice made larger improvement in picture selection than those who were trained and tested with the same female voice, despite the female’s overall slower speech rate for the training materials (RQ4). Critically, the male voice had shorter duration for the earlier cue (fait + infinitive verb) and longer duration for the later cue (à Marie/Pierre) than the female voice in the test materials. Assuming that the longer duration of words eases auditory processing of L2 input, the larger improvement after the auditory PI seemed to stem from the clarity of the later grammatical cue in the test speech, rather than from the durational property of the training speech. Our findings partially support VanPatten’s (Reference VanPatten and VanPatten2004) sentence location principle that postulates that grammatical elements at the end of sentences are easier to process than those in the middle. The present data suggests that L2 learners benefit from the longer duration (leading to greater clarity) of the sentence-final cues than from the longer sentence-medial cues. Additionally, because the relative durational clarity of the later grammatical cue in the male speech did not lead to a higher picture-selection accuracy in the before-training block, the later cue must have guided learners’ picture selections only after they received the PI training. Critically, because switching voices between training and testing (i.e., female→male→ female; male→female→ male) led to equivalent moderate improvements, the clarity (i.e., longer duration) of the later grammatical cue in the male voice seemed to have an additive effect only when the learners were highly familiar with the voice (male→male→ male).
As for the online eye-tracking data, Experiment 2 replicated the resilient FNP bias even after PI training (which was even larger in the same voice group in the object NP window) (RQ5). The reduction of the bias in the sentence-final PP (à Marie/Pierre) was visually easier to identify than in Experiment 1, and this may have been led by the relative durational clarity of the later grammatical cue in the male voice. However, the data generally confirmed the weak impact of auditory PI training on L2 speech processing: even with relatively clear acoustic cues, consistency in input modality did not lead to an immediate change in how learners responded to cues to the causative structure after a short training such as the one employed in the present study.
GENERAL DISCUSSION AND CONCLUSION
The two experiments in the present study tested (a) whether consistency in input modality across auditory PI training and testing benefits L2 French learners for the acquisition of a novel causative structure, and (b) whether familiarity with the voice of testing enhances the effect of PI training. The effects of input modality and voice familiarity were measured by the accuracy in dichotomous picture selections and the eye-movements of participants while they listened to spoken stimuli for picture selections. Compared to those who received written PI training (Wong & Ito, Reference Wong and Ito2018), learners who received auditory PI training in the present study made an equivalent improvement in picture selections, yet their performances were not better than those of the written PI group. The eye-movement patterns revealed a resilient FNP bias toward (first noun = agent) pictures, while the increase in looks to the correct picture was overall slower as compared to the written PI groups’ responses. Thus, while auditory PI training can be equally effective as written PI training to improve offline interpretational judgments, the consistency in input modality does not seem to have an additive, immediate effect to change online responses to the auditory form of a novel structure. This finding may in part be due to the overall difficulty of processing grammatical elements that occur in the middle of sentences as postulated by VanPatten’s (Reference VanPatten and VanPatten2004) sentence location principle. The increased cognitive load of processing auditory input in our present experiment may have made processing elements in final position more challenging as well. Our eye-tracking data echo the findings of Hama and Leow (Reference Hama and Leow2010) that showed the difficulty of acquiring a novel grammatical structure with auditory input and support the findings of VanPatten (Reference VanPatten1990) and Wong (Reference Wong2001) who found it challenging to attend to surface grammatical forms while processing auditory L2 input. The discrepancy between the gaze responses and the picture selections confirms the general delay between auditory perception and sentential interpretation in L2 processing. Future studies may need to explore whether and how the quality (e.g., type of feedback) or quantity (e.g., number of items) of PI training enhances online sensitivity to auditory input.
As for the effect of voice familiarity, the present results suggest that the degree to which L2 learners benefit from hearing the same voice for acquiring a novel grammatical construction may depend on the clarity of the cues to the target structure. Although the durational differences between the two speakers were not part of the intended design, the contrast between the male and the female voices revealed that the learners made better use of the later cue (à Marie/Pierre) than the earlier cue (fait + infinitive verb) in the testing stimuli. Because the female speaker consistently produced longer (fait + infinitive verb) for both the training and the test sentences, participants who continuously heard her voice could have become more sensitive to the earlier cue, and such a change in the sensitivity to the auditory input would have led to an earlier gaze shift after the infinitive verb. The data instead showed no sign of detection of the earlier cue: In fact, the gaze shift was slower for the same (female) voice group as compared to the written PI group. In contrast, those who were trained and tested with the male voice showed the highest after-training accuracy in picture selections, and they also made larger improvement than those who consistently heard the female voice. Although it was statistically marginal, those who continuously heard the male voice exhibited a reduction in incorrect looks to (first noun = agent) pictures in the sentence-final PP (à Marie/Pierre) as compared to those who were trained with the female voice.
We identify a few factors that may have helped learners process the later cue better. First, the sentence-final phrase may have been generally more salient than the sentence-medial words (e.g., with Intonation Phrase boundary tones that mark finality: Jun & Fougeron, 2010; VanPatten’s [Reference VanPatten and VanPatten2004] sentence location principle). Second, the form of sentence-final PP (à Marie/Pierre) had much less variability than the predicate elements after (fait + faire) that varied across items and thus served as a more reliable cue to the structure of the input. Third, given the more taxing nature of auditory input on processing, L2 learners generally may not have paid close attention to the form during the PI training and instead focused more on processing the meaning of input (i.e., the primacy of meaning principle, VanPatten, Reference VanPatten and VanPatten2004, p. 14). Because elements in final position are more salient than those in medial position, they processed the sentence-final information with more weight (the sentence location principle, VanPatten, Reference VanPatten and VanPatten2004). With auditory L2 input, the surface grammatical forms—especially the earlier cue that could tax limited L2 phonological memory (e.g., Brown & Hulme, Reference Brown, Hulme and Harris1992)—may not have been represented in detail (Absalom & Rizzi, Reference Absalom and Rizzi2008; Lund, Reference Lund1991; Park, Reference Park2004). The current experimental design does not enable us to further test the relative contributions of these factors, while they are all likely to have played some role in driving the advantage of listening to the male voice. Future investigations that aim to clarify how learners allocate their attention to various cues during L2 spoken sentence processing may require controls over the phonetic properties and the lexical variability of the stimuli, and also a measure of learners’ L2 cognitive capacities.
In summary, the present data suggest that the salience of grammatical cues, which is conditioned by their position and phonetic clarity, may influence the acquisition of a novel construction with a short training. However, the interpretations of the present data are limited, especially with respect to the general effect of voice variability on the outcome of PI training. While the present study has made an initial step toward the exploration of factors that may influence the efficacy of auditory PI training, many important questions remain open: Is multiple speaker input beneficial for PI training as in novel phonemic acquisition (Bradlow et al., Reference Bradlow, Pisoni, Akahane-Yamada and Tohkura1997; Lively et al., Reference Lively, Logan and Pisoni1993; Logan et al., Reference Logan, Lively and Pisoni1991)? Does the size of PI training modulate online responses to auditory input and, if so, how much training is necessary for learners to become sensitive to the cues to a novel construction? How stable is the effect of auditory PI training as compared to written PI training? How much do learners’ aptitudes (Perrachione et al., Reference Perrachione, Lee, Ha and Wong2011; Sadakata & McQueen, Reference Sadakata and McQueen2014) or cognitive capacities predict the effect of PI training and its stability? We plan to address these questions in future studies, and hope that the findings of this line of research will contribute to advancing the development of pedagogical materials and foreign language learning technologies.
