1. Introduction
There has been a long-term discussion about how people generate inferences that ‘fill the gaps’ between what has been explicitly stated and what the ‘fully filled-in’ message was intended to convey, in reading (Calvo & Castillo, Reference Calvo and Castillo2001; Campion, Reference Campion2004; Casteel, Reference Casteel2007; Cook, Limber, & O’Brien, Reference Cook, Limber and O’Brien2001) and in auditory sentence comprehension (Cook & Guéraud, Reference Cook and Guéraud2005; Cook, Halleran, & O’Brien, Reference Cook, Halleran and O’Brien1998; Cook & O’Brien, Reference Cook and O’Brien2014; Hald, Steenbeek-Planting, & Hagoort, Reference Hald, Steenbeek-Planting and Hagoort2007; Hare, Jones, Thomson, Kelly, & McRae, Reference Hare, Jones, Thomson, Kelly and McRae2009; Kamide, Altmann, & Haywood, Reference Kamide, Altmann and Haywood2003; Kintsch, Reference Kintsch1998; Metusalem et al., Reference Metusalem, MKutas, Urbach, Harb, MacRas and Elman2012). This discussion has mainly concerned information retrieval in sentence comprehension. Two types of information that are retrieved in sentence comprehension are temporary information provided by the given text, and world knowledge retrieved from long-term memory.
It the early literature, researchers focused on how comprehenders quickly and accurately process (e.g., comprehend and retrieve) temporary information, such as syntax and different facts mentioned in a given sentence (Campion, Reference Campion2004; Hald, Reference Hald2002; Hald et al., Reference Hald, Steenbeek-Planting and Hagoort2007; Kamide et al., Reference Kamide, Altmann and Haywood2003; Lindsay, Scheepers, & Kamide, Reference Lindsay, Scheepers and Kamide2013; Liversedge et al., Reference Liversedge, Drieghe, Li, Yan, Bai and Hyönä2016; Rayner, Reference Rayner1998; Villata, Tabor, & Franck, Reference Villata, Tabor and Franck2018). The effects of retrieval interference and encoding interference were tested using garden-path sentences, or sentences with ambiguous syntax. Syntactic information was used to create retrieval interference in some studies, and encoding interference in others.
In a retrieval interference model, accomplishment of memory tasks is driven by retrieval cues that “allow the parser to access the intended element based on its content, rather than scanning all the elements in memory in sequence” (Villata et al., Reference Villata, Tabor and Franck2018, p. 2). Therefore, there would be a lower probability of successful retrieval during sentence comprehension once the retrieval cues were not unique to the to-be-retrieved element. For example, when comprehenders were given a relative clause such as “The pilot remembered that the lady who was sitting near the smelly seat/man moaned about a refund”, they showed lower comprehension accuracy under the ‘man’ condition than ‘seat’ condition (Van Dyke, Reference Van Dyke2007). In other words, comprehension was facilitated by the knowledge that an inanimate subject ‘seat’ was not plausible for the verb ‘moan’ (i.e., unique retrieval cue condition).
By contrast, in an encoding model, fast and accurate sentence comprehension cannot always be based on a cue-based retrieval process. For example, in Gordon and colleagues’ study (Gordon, Hendrick, & Johnson, Reference Gordon, Hendrick and Johnson2001, Reference Gordon, Hendrick and Johnson2004), syntactic type was controlled between subjects in subject relatives, and between objects in object relatives (i.e., a mismatch condition such as “The barber that you admired climbed the mountain” vs. a match condition such as “The barber that the lawyer admired climbed the mountain”). Readers showed faster reading times at the verb, and higher sentence comprehension accuracy, under the mismatch condition compared to the match condition. However, in the above examples, the distinction between definite description and pronoun could not be cued by the verb. Therefore, the facilitation effect of a mismatch does not appear to lie in the cue-based retrieval process directed at satisfying the constraints of the verb. Researchers have claimed that interference should arise during the encoding of elements in memory (encoding interference), even though the effect was detected at the critical retrieval region (i.e., the verb). Recently, Villata and colleagues (2018) also reported strong evidence for encoding interference, and weaker evidence for retrieval interference, in two self-paced reading experiments. Based on these results, they proposed a self-organizing sentence processing model, accounting for retrieval and encoding interference with a single mechanism.
Recently, some researchers claimed that not only temporary information but information from long-term memory affects sentence processing. Therefore, researchers began to focus on the interaction between temporary information and world knowledge, a kind of information retrieved from comprehenders’ long-term memory, in sentence comprehension. That is, world knowledge helps facilitate the processing of temporary information, if the facts given in a sentence are consistent with the frame of world knowledge, but it inhibits processing if the facts are inconsistent with the frame of world knowledge (Casteel, Reference Casteel2007; Chen, Yang, Ma, & Li, Reference Chen, Yang, Ma and Li2018; Cook & O’Brien, Reference Cook and O’Brien2014; Cozijn, Noordman, & Vonk, Reference Cozijn, Noordman and Vonk2011; Hald, Reference Hald2002; Hald et al., Reference Hald, Steenbeek-Planting and Hagoort2007; Hare et al., Reference Hare, Jones, Thomson, Kelly and McRae2009). For example, a glass, or a jar, is the typical location for milk, whereas an orange is not for a syringe – it is in an atypical location that is inconsistent with world knowledge. In Chen and colleagues’ (2018) research, participants were instructed to comprehend typical sentences such as “The boy will pour the milk from the jar to the glass (i.e., Target), take the dumpling from the stove to the plate, and then taste the milk (i.e., referring to Target figure ‘glass’)”, or atypical target sentences such as “The woman will pick up the syringe from the drawer and insert it into an orange (i.e., Target), take the cotton ball from the bottle to the bowl, and then push the syringe (i.e., referring to Target figure ‘orange’)”. In Experiment 1, the results showed that participants fixated earlier on the photograph ‘glass’ (i.e., typical location for ‘milk’) when listening to the final word ‘milk’ than on the final word ‘orange’ (i.e., atypical location for ‘syringe’) when listening to the final word ‘syringe’.
The findings from Experiment 1 in Chen et al.’s (Reference Chen, Yang, Ma and Li2018) research indicated that activation of world knowledge facilitates the accessing of typical information, but inhibits the accessing of atypical information. More importantly, in Experiment 2 the researchers found that the effect of the critical location’s typicality (i.e., effect of world knowledge) became temporarily decreased, once the appropriate antecedent context was given. That is, fixation on the atypical location (e.g., orange) started much earlier under the sentences with appropriate antecedent context than without, while the participant was listening to the final target word (e.g., syringe). It seems that participants were sensitive to atypical contexts that were not consistent with world knowledge, and deactivation of world knowledge occurred before the processing of final target words. Storage and retrieval of information in long-term memory were fast and reliable in processing a series of sentences in which regular cases were equal to or more typical than atypical ones (Chen et al., Reference Chen, Yang, Ma and Li2018; Delaney & Ericsson, Reference Delaney and Ericsson2016; Ericsson & Delaney, Reference Ericsson, Delaney, Miyake and Shah1999), but might not be fast and reliable in processing a series of sentences in which irregular cases were more common than regular ones.
This adaptation can be seen as experience-based processing, which is efficient in sentence comprehension. Cook and colleagues, and some other researchers as well, asserted that people use strategy to help generate important information and to keep it available for use (Campitelli, Reference Campitelli2015; Cook et al., Reference Cook, Limber and O’Brien2001; Cook & O’Brien, Reference Cook and O’Brien2014). Using strategy reduces the need to retrieve large amounts of information, some of which might be superfluous for the given task. Three components are required to successful retrieve appropriate information in sentence comprehension: (a) retrieval rules for organized cues that are generated before encoding and then regenerated as retrieval cues on a later test; (b) temporal and recency cues for the most recent encoding of a given type; and (c) elaborative, skilled encoding methods that integrate preexisting knowledge and new knowledge (Delaney & Ericsson, Reference Delaney and Ericsson2016). These arguments are consistent with the Adaptive Control of Thought-Rational (ACT-R) theory proposed by Anderson and colleagues (Anderson, Reference Anderson1974; Anderson, Bothell, & Byrne, Reference Anderson, Bothell and Byrne2004; Anderson, Bothell, Lebiere, & Matessa, Reference Anderson, Bothell, Lebiere and Matessa1998; Anderson & Reder, Reference Anderson and Reder1999), which has been used to explain how different types of memory information could be efficiently integrated in cognitive processing.
ACT-R maintains that the mechanism of memorizing information is made up of multiple modules, including at least a declarative module (retrieval buffer), a visual module (visual buffer), a manual module (manual buffer), and an intentional module (goal buffer) (Anderson et al., Reference Anderson, Bothell and Byrne2004). One of the key features of ACT-R is that it helps the listener to adjust the weight of given memory cues. According to this theory, the latency in retrieving any fact from memory is determined by the fact’s activation level (Anderson & Reder, Reference Anderson and Reder1999). Activation or deactivation of the retrieved fact should be determined by retrieval rules, the function of which is to update the buffers in the ACT-R architecture (Anderson et al., Reference Anderson, Bothell and Byrne2004), and to make adjustments in the weights of cues. Therefore, the activation level of the retrieved fact (e.g., world knowledge in long-term memory) is stronger when information stored in long-term memory and temporary information in working memory are matched rather than mismatched (Anderson et al., Reference Anderson, Bothell and Byrne2004).
Based on Chen and colleagues’ (2018) finding, it seems that activation of world knowledge could be reduced or eliminated with a 50% mismatch in experimental items, which is partially consistent with the idea of adjustment of the retrieval buffer. However, it is not clear how sensitive the retrieval buffer is. In other words, there is an open question concerning whether retrieval rules could be sensitively adapted to different mismatch conditions. In Chen and colleagues’ research, though world knowledge was claimed to be deactivated (i.e., adjustment of retrieval buffer) under the atypical sentence condition (i.e., trials that mismatched the typical world knowledge), there was still an effect of world knowledge on sentence processing.
Chen et al.’s findings (2018) indicated that the remaining activation of world knowledge and keeping world knowledge in mind could always be efficient when typical cases are no less than 50% of all sentences. If the retrieval rules are sensitively adaptable in processing a series of sentences with different proportions of typical cases (e.g., low: 25% typical and 75% atypical cases; high: 75% typical and 25% atypical cases), we expected to detect different activation levels of world knowledge according to these proportions. In other words, we were interested in whether adjustment level of world knowledge could be changed according to different proportions of world knowledge-inconsistent items. This issue is worth discussing because it could help clarify how people learn to flexibly and efficiently integrate different memory information in cognitive processing. The findings could be used as evidence of the adaptation of retrieval rules during auditory sentence comprehension.
2. The present study
In the present study, we aimed to strengthen the argument for the possibility that there is flexible adjustment of world knowledge’s activation level based on adjustment of the retrieval rule in sentence comprehension. There were two main concepts in the present study: proportions of typical sentences and retrieval rules. Proportions of typical sentences in the present study referred to the proportion of sentences, in a given series of sentences, where there was a match between world knowledge in long-term memory and temporary information provided by the current context. Retrieval rules refer to the retrieval strategies that might help people adjust activation of world knowledge. Therefore, the main assumptions was that, in processing a series of atypical sentences, participants would adjust the activation of world knowledge according to the proportion of typical sentences that referred to typical locations: the more world knowledge-inconsistent items there are, the weaker the activation of world knowledge will be. That is, they would adapt the retrieval rule.
In the present study, a listen-and-look paradigm was used to test the above assumption. In the literature, one (Kamide, Lindsay, Scheepers, & Kukona, Reference Kamide, Lindsay, Scheepers and Kukona2016; Lindsay et al., Reference Lindsay, Scheepers and Kamide2013; Speed & Vigliocco, Reference Speed and Vigliocco2014), two (Speed & Vigliocco, Reference Speed and Vigliocco2014; Zhang, Wang, & He, Reference Zhang, Wang and He2015), or four locations (Cook & Guéraud, Reference Cook and Guéraud2005; Cook et al., Reference Cook, Halleran and O’Brien1998; Cook & O’Brien, Reference Cook and O’Brien2014; Hald et al., Reference Hald, Steenbeek-Planting and Hagoort2007; Hare et al., Reference Hare, Jones, Thomson, Kelly and McRae2009; Kamide et al., Reference Kamide, Altmann and Haywood2003; Kintsch, Reference Kintsch1998; Metusalem et al., Reference Metusalem, MKutas, Urbach, Harb, MacRas and Elman2012) have usually been used in the listen-and-look paradigm to discuss integration of information in sentence comprehension. In the present study, an auditory sentence was given two candidate locations. A one-location paradigm was excluded because there would be no appropriate index to detect changes in the activation of world knowledge. A four-location paradigm was also excluded because each auditory sentence presents four facts and it might take too much effort to retrieve all information. For example, in some of the studies with four locations (Cook & Guéraud, Reference Cook and Guéraud2005; Cook et al., Reference Cook, Halleran and O’Brien1998; Cook & O’Brien, Reference Cook and O’Brien2014; Hald et al., Reference Hald, Steenbeek-Planting and Hagoort2007; Hare et al., Reference Hare, Jones, Thomson, Kelly and McRae2009; Kamide et al., Reference Kamide, Altmann and Haywood2003; Kintsch, Reference Kintsch1998; Metusalem et al., Reference Metusalem, MKutas, Urbach, Harb, MacRas and Elman2012), two objects were mentioned in a sentence, each of which was paired with two locations. During sentence comprehension, participants might have to retrieve four typical locations for the given objects from long-term memory, and the mismatch location/locations will interfere with the retrieval of the given location/locations in working memory. Therefore, a two-location paradigm was finally chosen for the present study. Two kinds of materials (one for each condition) were presented to the participants. Examples of these materials are listed below.
(A) Typical final location: 准备借书, 他沿着小路走向图书馆。
‘To borrow books, he followed the path to go to the library.’
(B) Atypical final location: 参加会议, 他沿着小路走向游泳池。
‘To take part in the meeting, he followed the path to go to the swimming pool.’
Sentences within a condition were arranged with the same wording, with the only difference being the appropriateness of the pairing between the antecedent context (i.e., the first part of the sentence before the comma) and the critical location (i.e., the final word of the sentence). For example, ‘library’, in sentence (A), is a typical final location to the motive ‘to borrow books’, and thus sentence (A) is called a typical sentence. However, ‘swimming pool’, in sentence (B) is an atypical final location to the motive ‘to take part in the meeting’, and thus sentence (B) is called an atypical sentence. In a series of sentences, the proportions of typical sentences, according to the appropriateness of the antecedent context and the processing goal under world knowledge, were manipulated as one of the experimental factors (i.e., 50% typical vs. 50% atypical; 75% typical vs. 25% atypical; 25% typical vs. 75% atypical). The main question was whether participants were sensitive to the proportion of typical sentences during the given task using a listen-and-look paradigm (Altmann & Kamide, Reference Altmann and Kamide2009; Chen et al., Reference Chen, Yang, Ma and Li2018; Kukona, Altmann, & Kamide, Reference Kukona, Altmann and Kamide2014), thus facilitating sentence comprehension. The main assumption tested was that the proportion of typical sentences should be captured when there are more atypical cases than typical cases throughout the text, thus causing a change in the activation of world knowledge. The change in activation would be used as an indication of the flexible adaptation of retrieval rules.
Two experiments were conducted. Experiment 1 and Experiment 2 both manipulated the appropriateness of critical context-location pairs, and the fixation patterns on the critical locations were considered as the dependent variable. The effect of the proportion of typical sentences (in Experiment 1, 50% typical vs. 50% atypical; in Experiment 2, either 75% typical vs. 25% atypical or 25% typical vs. 75% atypical) was tested. Listeners viewed visual arrays depicting photographs of two possible final locations (see Figure 1). While they viewed the visual array, they heard “he followed the path to go to” and engaged in a priori processing before the critical location, then respectively heard either a typical final location in sentence (A) (e.g., ‘library’) or an atypical final location in sentence (B) (e.g., ‘swimming pool’) and engaged in real-time processing of the targets.
Procedure and sample materials used in Experiments 1 and 2.
First, in Experiment 1 we expected higher fixations on typical final locations (e.g., ‘library’ in (A)) than atypical final locations (e.g., ‘swimming pool’ in (B)) during the before-critical-word stage (i.e., hearing “he followed the path to go to”), when sentences with atypical final locations were no more frequent than sentences with typical final locations, indicating the effects of world knowledge. That is, higher fixations on typical final locations would be found not only in Experiment 1, in which half of the sentences had typical final locations and the other half had atypical final locations (i.e., equal proportion of typical final locations, 50% typical vs. 50% atypical), but also in Experiment 2, in which more sentences had typical final locations than atypical final locations (i.e., a high proportion of typical final locations, 75% typical vs. 25% atypical).
Second, according to the findings in Chen et al.’s (Reference Chen, Yang, Ma and Li2018) report, we expected high fixations on the location corresponding to the auditory location being heard in real time (Experiment 1), and under a high proportion of typical final locations (Experiment 2). That is, when listening to the name of a typical location, the typical location would be more fixated on than the atypical location when they were displayed as a pair; when listening to the name of an atypical location, the atypical location would be more fixated on than the typical location when they were displayed as a pair.
In Experiment 2 we expected to find one of two fixation tendencies under the low proportion of typical final locations condition, in which 20 sentences had a typical final location and the other 60 had an atypical final location (i.e., 25% typical vs. 75% atypical). There are two possible tendencies that would indicate the adaptation of retrieval rules in sentence comprehension: (1) atypical final locations are reliably more fixated on than typical final locations, or (2) typical and atypical final locations are equally fixated on until the auditory name of the corresponding location has been processed. Otherwise, world knowledge could be seen as the main factor in sentence comprehension under the current experimental manipulations.
3. Experiment 1
The purpose of Experiment 1 was to examine the effects of the appropriateness of auditory context-location information in sentence comprehension. In Experiment 1, we controlled the proportion of typical final locations given the context of the sentence: 50% of the sentences ended with the auditory location that typically coexists with the given context, according to world knowledge, whereas the other 50% of the sentences did not (i.e., 50% typical vs. 50% atypical). It was expected that sentence comprehension would be mainly affected by the match in appropriateness between context and location. Once the context-location information conflicts with world knowledge (i.e., in the atypical condition), sentence comprehension should become more difficult.
3.1. method
3.1.1. Participants
A total of 31 students (6 male, 25 female; ages 18 to 24, M = 19.55, SD = 2.30) from South China Normal University voluntarily participated in the present study. All were native Mandarin Chinese speakers, had no auditory or visual problems, and were given a small payment after participation. The study was reviewed and approved by the Human Research Ethics Committee for Non-Clinical Faculties (ethics committee of the School of Psychology at South China Normal University) before it was conducted, and all participants provided their written informed consent to participate.
3.1.2. Design
Stimuli in a listen-and-look task were manipulated in a single factorial design. The within-group factor was final location (typical or atypical), and the dependent variables were fixation count, fixation duration during time-window 1, and the first fixation time during time-window 2. Listeners received 40 out of 80 target trials with typical final locations, and the other 40 target trials with atypical final locations.
3.1.3. Materials
We created 120 target sentences (i.e., “To do _____ , he/she followed the path to go to_____.”), each of which had two different possible names for the final location (i.e., 240 locations in total). One of these locations was the typical location that matched the context according to general world knowledge (e.g., “To borrow books, he followed the path to go to the library /图书馆/.”), whereas the other location was an atypical location that mismatched the context according to general world knowledge (e.g., “To take part in the meeting, he followed the path to go to the swimming pool /游泳池/.).
To generate the 120 target sentences, we first created 160 sentences, with 320 location names, as material candidates. Twenty college students who did not take part in the main study were asked to rate the familiarity of each final location’s name using a scale from 1 (absolutely unfamiliar) to 5 (absolutely familiar). In addition, these students were asked to rate how appropriate the sentence contexts and the corresponding final locations were, according to world knowledge, using a scale from 1 (absolutely not appropriate) to 5 (very appropriate). According to the ratings for the familiarity of location names and the typicality of sentences (see details in Table 1), 120 sentences, each of which had two different final locations, were chosen as the target sentences.
Information on ratings of locations’ names and sentences in Experiments 1 and 2
notes: *** = p < .001, + = p > .10.
A male native speaker of Mandarin Chinese recorded all of the target sentences as the auditory stimuli on DAT tapes (16-bit, 44.1 KHz). These stimuli were then digitized and stored as individual computer files with an average length of around 5650 ms. The length of time-window 1 was always the same (i.e., the same piece of auditory segment “他沿着小路走向” was used in all materials) in the typical and atypical sentences. The length of key words (i.e., the names of the final locations) in the typical and atypical sentences was carefully controlled, ranging from 450 to 550 ms, and paired. Visual probes were color photographs (1024 × 768 pixels) on a white background that represented the names of the final locations. For each trial, two locations (the typical and atypical locations for the given sentence) were displayed as shown in Figure 1, and the positions of the typical and atypical locations, on the upper or lower part of the screen, were counterbalanced across trials.
Of the 120 target sentences, we chose 40 sentences with typical final locations (i.e., typical sentences) and 40 sentences with atypical final locations (i.e., atypical sentences) as the auditory materials for Experiment 1, based on the ratings of familiarity and appropriateness. In addition, 6 surprise tests, which were not included in the analyses, were embedded in the list, so that participants had to listen and look carefully during the whole task. Sentences in the surprise test were chosen from the 40 of 120 candidates who had not been chosen for use in the final target trials. Participants were given a question about the sentence they had listened to, and they judged whether the given contexts and locations were matched or not, according to the previous sentence (samples are provided in the ‘Appendix’).
3.1.4. Procedure
We used an SR Research Eye-Link 1000 head-mounted eye-tracker sampling at 500Hz to assess fixation in a look-and-listen task. Control files were constructed to display stimuli on a 17-inch IBM (9512-AB1) monitor (screen resolution: 1024 × 768 pixels). Visual stimuli stayed on the monitor for 9000 ms, and spoken stimuli were played 1000 ms after onset of the visual stimuli. Spoken stimuli lasted for 5650 ms on average. Calibrations were made after every eighth trial.
Before the main task, participants completed a photograph-naming task where they were instructed to learn all of the study materials (i.e., the photographs of the final locations and their corresponding names) until they reached 100% accuracy. During the main task, following six practice trials, participants heard sentences in a pseudo-random order, so that there were at least three other sentences between sentences that had the same antecedent context. The main experiment, including 6 practice trials, 80 main trials, and 6 surprise test trials, lasted 30 min (Chen et al., Reference Chen, Yang, Ma and Li2018; Zhang et al., Reference Zhang, Wang and He2015). Items were rotated across eight lists in a Latin square design, so that participants saw the same 80 sentences but in different orders.
3.2. results and discussion
One participant made three out of six wrong judgments in the six surprise test trials and was excluded from the final sample, leaving a sample of 30 participants. Trials in which fixation time was either too short (i.e., < 60 ms) or too long (i.e., > 800 ms), possibly indicating abnormal processing, were excluded following guides in the literature (Liversedge et al., Reference Liversedge, Drieghe, Li, Yan, Bai and Hyönä2016; Rayner, Reference Rayner1998), These exclusions resulted in approximately 7.4% of the data being omitted from the analyses.
Mean values for first fixation time, fixation count, fixation duration, and proportion of fixation to each final location (i.e., typical location or atypical location) recorded during time-window 1 (“he followed the path to go to the”) and time-window 2 (“library”) are summarized in Table 2. Note that we use this example item throughout, although our results include all items.
Means (SE) of fixation indexes in Experiment 1
notes: Time-window 1= listening to “he followed the path to go to the”; time-window 2= listening to “library”.
We tested linear mixed-effects models using the lmer programme of the lme4 package (version lme 4_1.1_6) for R (2012 <https://www.r-project.org/>). There were two parts of the analysis in Experiment 1. First, we submitted the data on eye-movements to 1-factor analyses, which were pairwise comparisons between the typical and atypical final locations for “he followed the path to go to the”, for all sentences, so that there were no context cues for identifying whether the given sentence was typical or atypical in time-window 1. That is, we set fixation location (i.e., typical location or atypical location visually displayed in each trial; Deviation: typical location = 1; atypical location = 2) as a fixed effect, and both participants and items were entered in the models as random effects (Barr, Levy, Scheepers, & Tily, Reference Barr, Levy, Scheepers and Tily2013). Second, we set the sentences (i.e., typical or atypical sentences given auditorily; Deviation: typical sentence = 1; atypical sentence = 2), fixation locations (Deviation: typical location = 1; atypical location = 2) and their interaction as fixed effects. Participants and items were entered in the models as random effects for the time-window 2 (e.g., ‘library’). The random effects model was determined by starting with the maximal random effects structure, which was trimmed if the model did not converge. Results are reported in Table 3.
Fixed effect estimates in Experiment 1
notes: F-location, fixation location on typical and atypical locations; S, sentence; *** = p < .001, ** = p < .01, * = p < .05, + = p < .10.
The results of Experiment 1 indicated different activation patterns of work knowledge under different conditions during sentence comprehension. On the one hand, by analyzing fixation duration, fixation counts, and proportions of fixation, we found that listeners fixated on typical locations longer/more than on atypical locations in listening to all sentences during time-window 1, and in listening to typical sentences during time-window 2. In addition, when listening to the final word under the atypical-sentence condition, listeners still posted a significantly longer first fixation time on the typical location. These findings suggested that world knowledge guides listeners’ location expectations and decisions. See Tables 2 and 3.
On the other hand, the typical locations were fixated on more than atypical locations were while the name of the final location was auditorily presented, based on fixation duration, fixation counts, and proportion of fixations. World knowledge and information in working memory appear to be co-activated during sentence comprehension, creating interference (Chen et al., Reference Chen, Yang, Ma and Li2018). In the present study, according to the sentence context and typical world knowledge, participants had difficulties processing the final atypical locations.
4. Experiment 2
In Experiment 2, we examined how world knowledge and retrieval rules are integrated during different stages of auditory sentence comprehension. Unlike the experimental manipulations in Experiment 1, in which sentences under typical and atypical location conditions were displayed in equal proportions, two unequal proportions of typical final locations were manipulated in Experiment 2: 60 sentences with typical final locations vs. 20 sentences with atypical final locations (i.e., high proportion of typical final locations), and 20 sentences with typical final locations vs. 60 sentences with atypical final locations (i.e., low proportion of typical final locations). Eye-tracking patterns identified in Experiment 1 were used as the basis of comparison for Experiment 2, allowing the examination of how world knowledge and retrieval rules are integrated during different stages of sentence comprehension. We expected to find that the eye-tracking patterns would differ between the low proportion of the typical final locations condition and those captured in Experiment 1, in which the proportions were equal. We also expected that fixation differences between typical and atypical locations under low proportion of the typical final location condition would be much smaller than in Experiment 1, due to adaptation of retrieval rules.
4.1. method
4.1.1. Participants
A total of 60 students (19 male, 41 female; ages 18 to 24, M = 19.88, SD = 2.22) from South China Normal University voluntarily participated in the present study. All were native Mandarin Chinese speakers, had no auditory or visual problems, and were given a small payment after participation. The study was reviewed and approved by the Human Research Ethics Committee for Non-Clinical Faculties (ethics committee of the School of Psychology at South China Normal University) before it was conducted, and all participants provided their written informed consent to participate.
4.1.2. Design
Stimuli in a listen-and-look task were manipulated in a 2 × 2 repeated-measures factorial design with appropriateness of final location (typical or atypical) as a within-groups factor, proportion of typical final locations as a between-groups factor, and first fixation time, fixation duration, fixation count, proportion of fixations during time-window 1 and during time-window 2 as the dependent variables.
There were two kinds of proportions of typical final locations: high proportion of typical final locations (75% typical vs. 25% atypical) and low proportion of typical final locations (25% typical vs. 75% atypical). In the high proportion of typical final locations condition, of the 80 target stimuli, 60 were sentences with typical final locations (e.g., “To borrow books, he followed the path to go to the library /图书馆/.”), and the remaining 20 were sentences with atypical final locations (e.g., “To take part in the meeting, he followed the path to go to the swimming pool /游泳池/.). By contrast, in the low proportion of typical final locations condition, 20 target stimuli were sentences with typical final locations, and the remaining 60 were sentences with atypical final locations.
4.1.3. Materials
In Experiment 2, two experimental lists were created (i.e., proportion of typical sentences: high proportion of typical final locations and low proportion of typical final locations). The materials used in Experiment 2 were selected from the same set of 120 possible sentences, using the same selection method as in Experiment 1.
4.1.4. Procedure
The procedure and experimental equipment used in Experiment 2 were the same as those used in Experiment 1.
4.2. results and discussion
Participants’ mean fixation counts, fixation duration, proportions of fixations to the same areas of interest during time-window 1 and during time-window 2, under different proportion-of-the-typical-final-location conditions, are summarized in Table 4.
Means (SE) of fixation indexes in Experiment 2
notes: Time-window 1= listening to “he followed the path to go to the”; time-window 2= listening to “library”.
We tested linear mixed-effects models using the lmer program of the lme4 package (version lme 4_1.1_6) for R (2012) specifying three crossed random factors: participant, final location that was being fixated on, and proportion of typical final locations (i.e., high or low). Significance values reflect both participant and trial. In time-window 1, there were no cues for identifying whether the given sentence was typical or atypical, so a new model was fitted to the data. The new model included fixation location (Deviation: typical location = 1; atypical location = 2) and proportion of typical final locations (Deviation: high proportion of typical final locations = 1; low proportion of typical final locations = 2) as categorical predictors. Their interaction was also entered as a fixed effect. Random intercepts were included for participants and items. In time-window 2, the model included the fixed effects, F-Auditory location (i.e., fixation location corresponding to the final auditory word; Deviation: typical sentences’ typical location = 1; atypical sentences’ atypical location = 2), and proportion of typical final locations (Deviation: high proportion of typical final locations = 1; low proportion of typical final locations = 2), as well as their interaction. In addition, participants and items were entered as random effects. The random effects model was determined by starting with the maximal random effects structure, which was trimmed if the model did not converge. Results are reported in Table 5.
Fixed effect estimates in Experiment 2. Models for different appropriateness of final locations.
notes: F-location, fixation location on typical and atypical locations, same as those in Table 3; F-auditory location, fixation location consisted with final auditory words; P, proportions of typical final locations; F×P, interaction between fixation location and proportions of typical final locations; *** = p < .001, ** = p < .01, * = p < .05, + = p < 0.10.
Because we did not have strong a priori predictions concerning two-way interactions between our fixed factors, we adhered to a three-step algorithm when constructing the models. First, our initial model contained the fixed factors described above without any interactions. Second, model comparisons examined whether the addition of a two-way interaction resulted in a significantly better fit compared to the simpler model without the interaction. This was done for every potential two-way interaction (and also for three-way interactions, though none of them contributed significant variance). Third, if one of the fixed factors was not significant either by itself or within a two-way interaction, we conducted an additional model comparison to see whether a model without the factor provided a similar fit to the data. If this was the case, the most parsimonious model (i.e., without the fixed factor) was selected for further model comparisons. As a result, the models presented in this paper sometimes differ in terms of the listed predictors for different measures.
Similar to the findings in Experiment 1, the results of Experiment 2 with a high proportion of typical final locations showed that world knowledge impelled listeners’ expectations of typical locations (i.e., typical locations triggered longer fixation duration, higher fixation count, and higher proportion of fixations than atypical locations did). In addition, when participants listened to the final word (i.e., the name of the final location), atypical locations were generally processed with more difficulty than typical locations were (see Tables 4 and 5). Further simple effects analysis on interaction between appropriateness and proportion of typical final locations showed that the main effects came mainly under the high proportion of typical final locations (see Table 6).
Fixed effect estimates in Experiment 2. Effects for different appropriateness of final locations under different proportions of typical locations.
In addition, we specified three crossed random factors: participant, sentence (i.e., typical or atypical sentences), and final location (i.e., typical location or atypical location visually displayed in each trial) for the final word (e.g., ‘library’) under different proportions of the typical final location, separately (see Table 6). In time-window 1, we set fixation location (Deviation: typical location = 1; atypical location = 2) as a fixed effect, and both participant and item were entered in the models as random effects. In time-window 2, we set sentence (Deviation: typical sentence = 1; atypical sentence = 2), fixation location (Deviation: typical location = 1; atypical location = 2), and their interaction as fixed effects. Participant and item were entered in the models as random effects. The random effects model was determined by starting with the maximal random effects structure, which was trimmed if the model did not converge.
The results of Experiment 2 based on a high proportion of typical final locations showed that longer first-fixation-time was recorded on the atypical location than on the typical location, but there were longer/higher fixations on typical locations than atypical locations. By contrast, the results based on a low proportion of typical final locations showed that there were no significant differences between the typical and atypical locations in first fixation time and fixation duration. In addition, the interaction between interest area (i.e., typical location vs. atypical location) and sentence (i.e., typical sentence vs. atypical sentence) reached significance. Further simple effects analysis showed that the main effects came mainly under the typical sentence condition. Also, the interaction effects were much smaller when there was a low proportion of typical final locations than when there was a high proportion (Tables 4 and 5). These findings indicate strong adaptation of retrieval rules during sentence comprehension, which adjusted the effects of world knowledge.
5. General discussion
Many existing studies have proved that both long-term world knowledge and temporal cues can affect language comprehension. However, proportions of typical and atypical conditions were always equal (i.e., 50% vs. 50%). In the present study, we further tested whether participants were sensitive to varying proportions of typical sentences in the given task and whether this sensitivity led them to adapt retrieval rules, namely, to adjust the activation level of world knowledge, in sentence comprehension. The main assumption was that participants under conditions with more atypical sentences might learn that atypical conditions could be highly possible in the given task, and then activation of long-term world knowledge might be weakened before the sentence has been fully comprehended. According to the present findings, we could partially conclude that flexible adjustment of activation of world knowledge existed during task fulfillment.
One of the main manipulations in this research was the proportion of typical sentences. In each trial, participants were instructed to listen to the given sentence and look at the displayed photographs, in which typical and atypical final locations were shown. The basic assumption in Experiment 2 was that participants might be able to capture the proportion of typical sentences, and use this information to adapt their activation strategy for world knowledge. In other words, participants should retrieve world knowledge according to the context and keep that knowledge available during the task to facilitate sentence comprehension, if the proportion of typical trials is no less than the proportion of atypical trials (i.e., 75% typical vs. 25% atypical, 50% typical vs. 50% atypical) throughout the task. By contrast, world knowledge should be strongly deactivated if more atypical trials than typical trials (i.e., 25% typical vs. 75% atypical) are displayed. Using the fixation data as the key index for the activation of world knowledge, this assumption is supported. That is, the present findings indicated that the proportion of typical sentences was successfully captured and used as the guide to adapt retrieval rules during sentence comprehension.
These findings are an important alternative interpretation of how people perform complex sentence comprehension tasks with a large amount of information, when they are limited by working memory capacity. During sentence comprehension, an individual reading a sentence in a text must have access to previously mentioned information and resolve references to the critical words (Ericsson & Kintsch, Reference Ericsson and Kintsch1995). This process is termed ‘retrieval of temporary information’ (Chen et al., Reference Chen, Yang, Ma and Li2018; Kukona et al., Reference Kukona, Altmann and Kamide2014). In addition, readers integrate prior knowledge (i.e., world knowledge) into the given temporary information (Chen et al., Reference Chen, Yang, Ma and Li2018; Hald et al., Reference Hald, Steenbeek-Planting and Hagoort2007), facilitating comprehension. However, this process is not efficient if all kinds of information are kept throughout sentence comprehension. Therefore, information, including world knowledge and temporary information, is likely to be dynamically activated and deactivated during sentence comprehension, so that working memory will not be overloaded (Chen et al., Reference Chen, Yang, Ma and Li2018; Hald et al., Reference Hald, Steenbeek-Planting and Hagoort2007). According to the ACT-R theory (Anderson et al., Reference Anderson, Bothell and Byrne2004), the retrieval buffer in the ACT-R architecture is adjusted.
The standard for activation or deactivation of information during sentence completion is likely built according to retrieval rules. Researchers have claimed that domain-relevant skills, knowledge, and procedures for the task should be tightly integrated to make use of the extended capacity of working memory (Ericsson & Delaney, Reference Ericsson, Delaney, Miyake and Shah1999). More importantly, retrieval strategies, strategies for activating different information based on different given cues, should be modifiable throughout the task. In the literature, the choice of retrieval strategies has been mainly discussed in terms of explicit cues, referring to domain-relevant skills (e.g., linguistic rules) or specific knowledge (e.g., chess playing) (Ericsson & Kintsch, Reference Ericsson and Kintsch1995; Gobet & Simon, Reference Gobet and Simon1996). Experts use specific knowledge to help solve complex cognitive tasks, so that the temporary information provided by the current task can be quickly and accurately processed.
By extension, in the present study we examined effects of proportion of typical sentences, a more implicit cue as compared to the syntax information. In long-term memory, there exist different kinds of information or scenes from everyday experience, some of which are typical and some of which are atypical. All information is stored under a specific structure, waiting to be accessed. For example, a book is often on a table, in a schoolbag, or in a bookcase (i.e., typical scenes), but seldom on the floor (i.e., atypical scene). In most cases, typical information stored in long-term memory is automatically and dynamically activated, and atypical information is deactivated. However, world knowledge keeps changing. Therefore, people need the available retrieval rules to access the most needed information. With different retrieval rules, participants can decide whether world knowledge should be strongly activated before the target is displayed. For example, it should be more efficient to activate and keep world knowledge in mind during sentence comprehension if typical conditions are more frequent than atypical conditions (i.e., 75% typical vs. 25% atypical in the present research). By contrast, the process becomes inefficient if participants activate and keep world knowledge in mind before accessing the target word when there are fewer typical conditions than atypical conditions (i.e., 25% typical vs. 75% atypical in the present research).
Under the present listen-and-look paradigm, fixations on the given object before hearing the final word in a sentence (the comprehension target) could be seen as one of the main indexes of activation of the relevant information, from either world knowledge or temporary information provided by the sentence context (Altmann & Kamide, Reference Altmann and Kamide2009; Chen et al., Reference Chen, Yang, Ma and Li2018; Kukona et al., Reference Kukona, Altmann and Kamide2014). Results indicated that participants were sensitive to the proportion of typical sentences, and used them as standards to choose information efficiently during sentence comprehension (i.e., adapt retrieval rules). World knowledge was deactivated before the critical target word had been fully accessed, when more atypical items than typical items were displayed in the given task. In addition, fixation tendencies under the equal condition (i.e., 50% typical vs. 50% atypical) were similar to those under the more typical condition (i.e., 75% typical vs. 25% atypical). It seemed that people have a bias to activate world knowledge ahead of auditory sentence comprehension, until they find a significant disadvantage of this strategy (e.g., under 25% typical vs. 75% atypical conditions).
It should be noted that sentence structure was the same during the whole task, both in the present study and in most of the reports in which a listen-and-look paradigm was used (Chen et al., Reference Chen, Yang, Ma and Li2018; Cook & O’Brien, Reference Cook and O’Brien2014; Zhang et al., Reference Zhang, Wang and He2015). It is undeniable that single sentence structure might facilitate learning of the rules conveyed by the materials. Though we believe that this facilitation is not a reason to discount the current results, it is best to draw careful conclusions. That is, the present results should not be generalized to natural sentence comprehension before we have accumulated more evidence with other materials or paradigms.
6. Conclusion
World knowledge plays an important role in sentence comprehension (Chen et al., Reference Chen, Yang, Ma and Li2018; Cook et al., Reference Cook, Limber and O’Brien2001; Cook & O’Brien, Reference Cook and O’Brien2014; Hald, Reference Hald2002; Kukona et al., Reference Kukona, Altmann and Kamide2014). Combining findings in the literature with those of the present study, it can be concluded that people show a bias to use world knowledge to facilitate accessing temporary information provided by sentence context (Chen et al., Reference Chen, Yang, Ma and Li2018). In addition, retrieval rules guide people to manage whether world knowledge should be activated or deactivated before the critical target words are accessed. This was called adjustment of world knowledge in the present study. Without being explicitly told about the proportion of typical sentences, participants were still sensitive to the specific given rule, and used this rule to adjust the timecourse and strength of world knowledge activation during sentence comprehension. The present experiments contribute to our understanding of the flexible adaptation of retrieval rules in sentence comprehension, helping people activate the most appropriate information at the right time.
Appendix
Samples of surprise test trials.
(1) Listen and look: Early in the morning, the doctor followed the path to go to the theater.
Correct or Incorrect: The doctor went to the theater.
(2) Listen and look: Hearing the class bell, the student followed the path to go to the bathroom.
Correct or Incorrect: The student went to the classroom.
(3) Listen and look: To attend the meeting, the members followed the path to go to the council chamber.
Correct or Incorrect: The members went to the council chamber.
Participants were instructed to judge whether the ‘Question’ sentence was correct (i.e., A. correct or B. incorrect) according to the given ‘Listen and Look’ sentence. No photographs were displayed while the ‘Question’ sentence was displayed.
