1. Cross-modal priming: the method
The Cross-Modal Priming Task (CMPT) is a psycholinguistic method developed by David Swinney (Swinney, Reference Swinney1979) that measures activation of lexical and syntactic information during sentence comprehension (see also, Roberts, Reference Roberts, Jegerski and VanPatten2014). It is an online method that measures the activation of lexical and syntactic information as participants listen to sentences in real-time. It contrasts with offline sentence comprehension tasks that measure the outcome of sentence comprehension after participants have heard the sentence and have had time to think about its meaning. Therefore, it is an implicit measure that taps into the participants’ automatic response to lexical and syntactic information in contrast to offline comprehension tasks that may be affected by the participants’ metalinguistic awareness (Marinis, Reference Marinis, Blom and Unsworth2010).
The CMPT is a dual task involving auditory and visual modalities; this is why it is called cross-modal. In each trial, participants start to listen to a sentence. Before the end of the sentence they see a word (cross-modal lexical priming) or a picture (cross-modal picture priming) on the computer screen that is either related (or identical) to a word they heard in the sentence before or it is completely unrelated. As soon as they see the word/picture, they have to press a button as fast as they can to make a lexical decision (word/non-word) or a picture classification (e.g., an animacy task). Reaction times to a word/picture that is related (or identical) to a word they have heard before are shorter than reaction times to an unrelated word/picture because in the first case there is facilitation by the appearance of a related (or identical) word prior to the word/picture. This is why it is called a priming task.
The instructions on how to perform the task are given prior to the start of the experiment and a practice session is required to ensure that the participants familiarize themselves with the task. After the end of the sentence, a comprehension question can be used to ensure that participants pay attention to the meaning of the sentence and researchers also have a measure of their off-line comprehension.
The CMPT has often been used to investigate the processing of lexical ambiguity (e.g., Klepousniotou, Reference Klepousniotou2002; Onifer & Swinney, Reference Onifer and Swinney1981; Swinney, Reference Swinney1979; Swinney, Love, Walenski & Smith, Reference Swinney, Love, Walenski and Smith2007; Tabossi, Reference Tabossi1988) and syntactic dependencies, e.g., filler-gap dependencies (Love & Swinney, Reference Love and Swinney1996; Love & Swinney, Reference Love and Swinney2007; Marinis & van der Lely, Reference Marinis and van der Lely2007; Nicol, Reference Nicol, Gerry and Shillcock1993; Felser & Roberts, Reference Felser and Roberts2007; Roberts, Marinis, Felser & Clahsen, Reference Roberts, Marinis, Felser and Clahsen2007); object scrambling (Clahsen & Featherston, Reference Clahsen and Featherston1999; Nakano, Felser & Clahsen, Reference Nakano, Felser and Clahsen2002); and reference of pronouns and reflexives (McKee, Nicol & McDaniel, Reference McKee, Nicol and McDaniel1993; Nicol & Swinney, Reference Nicol and Swinney1989; Schwartz, Hestvik, Seiger-Gardner & Almodovar, Reference Schwartz, Hestvik, Seiger-Gardner and Almodovar2016).
A good example of a lexical decision version of the CMPT addressing access of lexical information is the original study by Swinney (Reference Swinney1979). This study addressed context effects on lexical access using two CMPTs (Experiment 1, Experiment 2). Participants heard in each trial an introductory sentence, for example: ‘Rumor had it that, for years, the government building had been plagued with problems’. This was followed by a critical sentence in one of the four conditions that included the factors Ambiguity (lexical ambiguity) and Context, as shown in Table 1 below.
Conditions in the Swinney (Reference Swinney1979) study.
At the position indicated by [*] (Experiment 1) or three syllables after that position (Experiment 2), one of three words (ANT: contextually related, SPY: contextually inappropriate, SEW: unrelated) appeared on the computer screen and participants had to press a button to judge whether they saw a word or a non-word. The ambiguous sentences included the ambiguous word ‘bugs’, whereas the unambiguous sentences included the unambiguous word ‘insects’. The sentences with the biasing context biased participants to interpret ‘bugs’ as insects, whereas in the sentences without context, ‘bugs’ was completely ambiguous. The three words were matched on frequency and length. Therefore, differences in reaction times (RTs) between the three words in the lexical decision task could be interpreted as a result of the context they were used in. The results revealed that in the two ambiguous conditions, adult monolingual participants had shorter RTs for the two words related to the two meanings of the ambiguity (ANT, SPY) compared to the unrelated word (SEW) when the words were presented at the offset of the ambiguity (bugs). In contrast, when the words were presented three syllables after the ambiguity, only the word with the appropriate meaning (ANT) had shorter RTs than the unrelated word (SEW). This did not differ from the contextually inappropriate word (SPY). This demonstrated that at the offset of an ambiguous word, all possible meanings of the word are activated irrespective of the context but the previous biasing context rapidly affects post access lexical processing, and, thus, one of the two meanings is selected/available later on in the sentence.
A good example of a picture decision version of the CMPT addressing access of syntactical information is the CMPT used in the study by Roberts et al. (Reference Roberts, Marinis, Felser and Clahsen2007) with monolingual children. This study addressed the processing of filler-gap dependencies. Participants listened to a sentence, as shown in Table 2.
Conditions in the Roberts, et al. (Reference Roberts, Marinis, Felser and Clahsen2007) study.
At the position indicated by [*] one of two pictures (squirrel, toothbrush) appeared on the computer screen and participants had to press a button to judge whether what they saw was an animate or an inanimate object. The picture was presented either at the position of the trace or at a control position in order to address the Trace Reactivation Hypothesis. According to the Trace Reactivation Hypothesis, the parser holds a filler (in this case to which that refers to the squirrel) temporarily in short term memory and at the position of the gap it sets up a filler-gap dependency by reconstructing the grammatical and semantic features of the filler (e.g., Swinney, Ford, Frauenfelder & Bresnan, Reference Swinney, Ford, Frauenfelder, Bresnan, Grosz, Kaplan, Macken and Sag1988). This predicts shorter RTs for the picture of the antecedent (squirrel) than the picture of the unrelated referent (toothbrush) at the trace position, but predicts no such difference in RTs at the control condition. The results revealed that monolingual children and adults with high working memory show shorter RTs for the picture of the antecedent compared to the unrelated picture at the trace but not at the control position. This demonstrated that they process filler-gap dependencies by reconstructing the grammatical and semantic features of the filler at the gap.
The two studies above have demonstrated that the CMPT can be used to measure activation of lexical and syntactic information during sentence comprehension. A further motivation for using the CMPT is to provide evidence for the psycholinguistic reality of competing syntactic analyses when more than one have been proposed on theoretical grounds. A case in point is the study by Paspali and Marinis (Reference Paspali and Marinis2017) on the processing of double object constructions in Greek. Greek has two word orders for double object constructions, as shown in the examples below.
1.
2.
Currently several analyses have been proposed about the base and derived word order of Greek double object constructions (Anagnostopoulou, Reference Anagnostopoulou, Stavrou and Terzi2005). The first analysis suggests that the Direct Object (DO) dominates the Indirect Object (IO) and, thus, the DO-IO order is base generated and the IO-DO is derived. In the IO-DO word order, there is syntactic movement of the IO that leaves a trace behind. The second analysis proposes the opposite scenario, where the IO-DO order is base generated and the DO-IO derived (Georgala, Reference Georgala, Özsoy and Gürel2012, Bowers & Georgala, Reference Bowers, Georgala and Alexiadou2007). In this analysis, the DO-IO word order involves syntactic movement of the DO that leaves a trace behind. Paspali and Marinis (Reference Paspali and Marinis2017) tested these hypotheses in adult native speakers of Greek by using two CMPTs and two Probe Classification During Reading (PCDR) tasksFootnote 1 in a design similar to Roberts et al. (Reference Roberts, Marinis, Felser and Clahsen2007). Table 3 shows the conditions used in the experiments and shows where the trace is in each one of the two analyses.
Conditions in the Paspali & Marinis (Reference Paspali and Marinis2017) study.
In the CMPTs, at the position indicated by [*], the picture of the antecedent of the relative clause (camel: identical picture) or an unrelated picture (umbrella: unrelated picture) appeared on the computer screen and participants had to press a button to judge whether the picture showed an animate or inanimate character. The results indicated a priming effect only at the offset of the direct object and as a result provided evidence for the psychological reality of the analysis by Georgala (Reference Georgala, Özsoy and Gürel2012) and Bowers and Georgala (Reference Bowers, Georgala and Alexiadou2007), according to which the base word order is IO-DO. This demonstrates that the CMPT is also useful in testing competing theoretical analyses.
2. Cross-modal priming in bilingualism research
Despite its high accuracy and sensitivity in revealing effects of lexical and syntactic processing, the CMPT so far has not been used as widely as other methods (e.g., self-paced reading) to address sentence processing in bilingualism research. Two studies are presented below as examples to illustrate how this methodology has been used to address syntactic processing in bilinguals, Felser and Roberts (Reference Felser and Roberts2007) and Miller (Reference Miller2015a).Footnote 2
Felser and Roberts (Reference Felser and Roberts2007) used the task from Roberts et al. (Reference Roberts, Marinis, Felser and Clahsen2007) with Greek adult second language learners of English. The results were very different from the pattern attested in monolingual children and adults (Roberts et al., Reference Roberts, Marinis, Felser and Clahsen2007). Whereas the monolingual adults only showed priming at the gap, the adult second language learners showed priming in both the gap and control positions. This demonstrates that they kept the filler in working memory but did not reactivate it at the gap. If they did, the priming effect would then have been larger at the gap compared to the control position. This provided evidence that adult second language learners process filler-gap dependencies qualitatively differently than monolingual adults.
Miller (Reference Miller2015a) used a very similar CMPT to Felser and Roberts (Reference Felser and Roberts2007) to test the processing of filler-gap dependencies in French indirect object constructions. Miller (Reference Miller2015a) used very similar material to the material used in Felser and Roberts (Reference Felser and Roberts2007); a picture appeared either at the gap or at the offset of the previous word, which was the control position, as shown in Table 4.
Conditions in the Miller (2015) study.
However, there was an important difference in the pictures used in these two studies. In Felser and Roberts (Reference Felser and Roberts2007) the control picture depicted an inanimate object that was not introduced in the sentence, whereas in Miller (Reference Miller2015a) the control picture depicted a character that was introduced in the sentence and was closer to the gap than the antecedent. In the example above, the control picture depicted a kangaroo. Since both the antecedent and the control character were introduced in the sentence, both were activated. Moreover, the control character was closer to the gap than the antecedent, which predicts high activation. Unsurprisingly, adult native speakers did not show any difference in RTs between the picture of the antecedent and the control picture in either the trace or the control position. The second language learners showed a similar pattern to the one attested in Felser and Roberts (Reference Felser and Roberts2007).
3. Methodological considerations
The studies presented above illustrate several methodological issues that need to be carefully considered when designing CMPTs across the board and for studies in bilingual populations in particular:
1) Familiarisation
This is a dual task, because participants have to process the sentences for comprehension, whilst at the same time they have to categorise words/pictures. It requires a long familiarization phase for the participants to understand how to do the task and to practice before moving to the experimental phase.
2) Processing capacity and cognitive flexibility
Due to the dual nature of the task, high levels of processing capacity and cognitive flexibility are required. Participants who have processing limitations may have difficulties doing the task and may focus more on one of the two tasks (comprehension, word/picture categorization). This will be evident if they show a higher success rate in the comprehension questions and a lower success rate in the categorization, or the other way around.
3) Lexical access
The priming effect measured in this task results from the activation of lexical and syntactic features of words that were introduced earlier in the sentence (or semantic associates of the words). This involves lexical access, activation, decay of activation, and reactivation. Lexical access may be slower and more effortful in bilinguals. The use of semantic associates involves an additional process to establish a semantic association between the word/picture and the antecedent (Clahsen & Featherston, Reference Clahsen and Featherston1999). Therefore, tasks with semantic associates are more effortful than tasks with identical words/pictures. Moreover, it is unclear as to whether bilinguals have the same semantic associations as monolinguals. This will largely depend on their vocabulary size and density, proficiency in the language, language dominance, and language use. These factors can lead to considerable individual variability that may affect the results.
4) Psycholinguistic properties of material
Factors that relate to the lexical properties of the words, such as word frequency, length in terms of number of letters, syllables, and neighborhood density, as well as age of acquisition may affect the level of activation of words. Therefore, these factors have to be carefully controlled. When pictures are used, these should also be controlled for factors, such as visual complexity and association of the picture to the target word. The pre-testing of pictures through a naming task is necessary in order to ensure that they correspond to the related word in the sentence. The picture pairs should also be matched on imageability, and pre-testing should ensure that the pictures in each pair have similar speed of naming.
5) Working memory
The task puts high demands on working memory when it measures syntactic dependencies. This was evident in the study by Roberts et al. (Reference Roberts, Marinis, Felser and Clahsen2007) that showed effects of working memory in participants whose working memory is still developing (primary school children). This issue is also relevant for bilingual children and other populations whose working memory is still developing. Therefore, it is wise to use a measure of working memory in order to be able to control for the participants’ working memory. Importantly, when the working memory task used is a verbal task, such as reading or listening span tasks, the participants’ language proficiency may affect the results of the working memory task. Therefore, the results of such working memory tasks are predicted to correlate with language proficiency tasks. This is important to keep in mind when interpreting the results of the study.
6) Effect sizes
Effect sizes in CMPTs are usually small (less than 100 millisecond). If the number of participants is small and/or there is large individual variability in speed between the participants, this may lead to null results. The null results in the monolingual group included in Miller (Reference Miller2015a) may be the outcome of recruiting only 12 participants.
7) Speed of processing
The priming effect is affected by the participants’ speed of processing. If there is large individual variability in the participants’ speed of processing and some tend to process sentences at a slower rate than others, reactivation may occur after the critical point in the sentence where the word/picture is presented. This may lead to a null result when participants with difference speed of processing are grouped together. The individual variability in processing speed may be larger in bilinguals compared to monolinguals because of the larger individual variability in their language history. Strict selection criteria and pre-testing of the participants’ processing speed can avoid null results due to speed of processing variability.
4. Advantages & disadvantages
Cross-modal priming has many important advantages compared to other tasks. It is an online task that measures the participants’ automatic reaction to verbal stimuli. It is a measure of the ongoing process of language comprehension and can provide information about how participants process sentences in real-time. This is unlike many offline tasks, such as picture selection and grammaticality judgment tasks. The CMPT measures the participants’ implicit knowledge rather than their explicit knowledge about language. Therefore, its main advantage compared to offline tasks is that it is immune to participants’ response strategies and metalinguistic knowledge (Marinis, Reference Marinis, Blom and Unsworth2010).
The CMPT also has many advantages in comparison to other online tasks. Compared to self-paced-listening tasks that use sentences segmented in words or phrases, the CMPT uses unsegmented sentences. Therefore, it is closer to the typical listening experience people have when they listen to sentences in their everyday life. Although it is an experimental task, it has better ecological validity than self-paced listening that allows participants to listen to sentences at their own pace, which does not happen in real life. As a result, the processes measured in the CMPT reflect better the real life process of language comprehension. The cross-modal nature of the CMPT provides an advantage against tasks that use only the visual or auditory modality, e.g., self-paced reading and self-paced listening. By using both the visual and auditory modalities, the CMPT is minimally affected by form overlap.
Some further advantages of the CMPT are that it can be used with preliterate children and adults with low literacy skills, because, if pictures are used, it does not require reading. Finally, in terms of cost, it is relatively inexpensive to run compared to other online methods, like eye-tracking and ERP, because it only requires a computer and it is portable, because it can be implemented on a laptop computer.
As with all tasks, the CMPT also has some disadvantages. It is a dual task – participants have to listen to sentences for comprehension and at the same time make a lexical or animacy decision. This requires high levels of attention and working memory. Therefore, it is more demanding than tasks like self-paced reading or listening that are single tasks (reading or listening for comprehension). Participants with attention difficulties or working memory limitations may have difficulties to perform the task; this may not be due to their comprehension ability but their attention and working memory limitations. Including a training session for the task and tasks that measure attention and working memory can help to tease apart effects of attention and working memory from effects of processing of lexical or syntactic information.
A second disadvantage, compared to tasks such as self-paced reading and listening, is that it does not measure sentence processing in a continuous manner across the whole sentence but focuses on a single point in the sentence. Participants who have slower processing speed may not show a priming effect. This can be rectified if the design includes control positions for the presentation of a word/picture not only before but also after the critical word, as in the study by Paspali and Marinis (Reference Paspali and Marinis2017).
5. Conclusions
The CMPT provides a window into the way people process sentences in real-time and their automatic response to lexical and syntactic information. This is very useful, as it allows us to measure their implicit knowledge, which is difficult to capture using off-line tasks that can be affected by metalinguistic knowledge.
