The working memory is a broad theoretical construct, used to refer the set of cognitive functions that enables the maintenance and manipulation of information in the short term (Baddeley, Reference Baddeley2012). The storage of the serial order in which visual and spatial events happen is considered in the literature as a complex function performed by working memory, and is the central theme of this study. This ability is related to many cognitive domains, such as how to direct the behavior towards a goal, perceive and recognize an action, predict environmental events, and generate thoughts about the future (Parmentier, Andrés, Elford, & Jones, Reference Parmentier, Andrés, Elford and Jones2006).
The studies on the short-term representation of the order of events in the visual-spatial memory have suggested dissociation between the memory of the event’s characteristics (the identity of the stimulus) and of the serial position of events (Mammarella et al., Reference Mammarella, Cornoldi, Pazzaglia, Toso, Grimildi and Vio2006; Pickering, Gathercole, Hall, & Lloyd, Reference Pickering, Gathercole, Hall and Lloyd2001). The evidences came from studies of spatial (Mammarella et al., Reference Mammarella, Cornoldi, Pazzaglia, Toso, Grimildi and Vio2006) and visual information (Pickering et al., Reference Pickering, Gathercole, Hall and Lloyd2001). In combination, behavioral (Rudkin, Pearson, & Logie, Reference Rudkin, Pearson and Logie2007) and neuropsychological studies (Elvevag, Fisher, & Goldberg, Reference Elvevag, Fisher and Goldberg2003; Price & Schin, Reference Price and Shin2009) indicate that the memory for the serial order of visual and spatial events is a process that consumes more attentive resources than the memory of the identity of stimuli.
In addition, there are evidences from dual-task paradigm (Zimmer, Speiser, & Seidler, Reference Zimmer, Speiser and Seidler2003), recognition tasks (Lecerf & Ribaupierre, Reference Lecerf and Ribaupierre2005) and serial recall (Jalbert, Saint-Aubin, & Tremblay, Reference Jalbert, Saint-Aubin and Tremblay2008) that the information about the order of events is encoded incidentally in working memory, i.e., without voluntary effort, when the experimental condition requires the storing of only one stimulus dimension. For example, Lecerf and Ribaupierre (Reference Lecerf and Ribaupierre2005) showed that in a task in which participants should keep spatial information of stimuli sequentially presented, the irrelevant change in the order of items in the testing sequence have a detrimental effect on the performance of the spatial task. Moreover, Parmentier et al. (Reference Parmentier, Andrés, Elford and Jones2006) showed that a form of organization of stimuli – the temporal grouping – confirms the occurrence of the information’s integration for order with the characteristics of the events.
In a conciliatory approach, Gmeindl, Walsh, and Courtney (Reference Gmeindl, Walsh and Courtney2011) studied memory for spatial sequences in a task in which the response was given in relation to the spatial location of events, or contain both location and order of events. The results showed that the participants had more difficulty in retrieving the spatial information when to store the order was an explicit demand on the task. This was interpreted as evidence that integration between serial order and spatial representation in working memory is an active process (controlled) only when the specific instruction to respond in relation to the serial order occurs. However, the storage of serial order occurs in an incidental way when it represents relevant information to the task.
Parmentier (Reference Parmentier, Vandierendonck and Szmalec2011) discusses evidences about the serial order of spatial sequences in terms of what he called transitional information. For him, in addition to the specific storage to the stimulus modality, there is another mechanism that stores the association of stimuli, given by the abstract information originated by the sequential presentation of events. Thus, it is possible to suggest that the maintenance of information about serial order collaborates with the registration of identity of the object in working memory, i.e., a strategy in which information about the object is recited serially and undertaken according to the spatial trajectory formed; in using this information, the serial position becomes a hint to the correct recovery of information (Cowan, Saults, & Morey, Reference Cowan, Saults and Morey2006). The possibility that this process is stored by a general domain system can be considered as the binding component, the episodic buffer in Baddeley’s model (2012) or, executed by the serial distribution of the attention focus in the sequence (Cowan, Reference Cowan2005; Morey, Reference Morey2009).
Considering the controversial evidences about the short-term memory of visual-spatial serial order and admitting that short-term visual-spatial memory is a function performed by a more complex system – working memory (Allen, Baddeley, & Hitch, Reference Allen, Baddeley and Hitch2006), the present study investigated if the order of events in a sequence is encoded in an incidental way towards visual and spatial information of events in working memory. An experimental paradigm for change detection was used, which consists in determining the interference of changes in an irrelevant dimension of a stimulus on memory task performance based on a relevant dimension (Treisman & Zhang, Reference Treisman and Zhang2006).
In this proposal, the short-term storage of sequences defined by visual appearance, spatial location, or visual-spatial conjunction was evaluated. It was considered that, if an irrelevant change in the stimuli serial position would undermine the task performance, it could be concluded that serial order and other characteristics of the event (visual appearance and spatial location) were encoded in an integrated manner in working memory (Treisman & Zhang, Reference Treisman and Zhang2006). The employment of this paradigm can help to clarify if the order of events is incidentally encoded in working memory and contribute new information to the body of knowledge on visual-spatial working memory.
Method
Participants
Sixty University of São Paulo-Brazil students, between the ages of 18 and 40 years (M = 23 years, SD = 3.47), from both genders, with normal or corrected visual acuity participated in this study. The participants were randomly assigned to three groups according to the experimental tasks to be performed: visual, spatial, or visual-spatial.
Materials and Stimuli
The visual stimuli were letters (21 consonants in the size of 40 x 40 pixels) in 14 different graphic fonts (visual patterns with shape), selected from the Microsoft Office fonts’ file. These stimuli were presented in scattered spatial locations on the computer screen. The fixation point was a diamond of 14 x 17 pixels, located in the center of the screen. The stimuli were presented in a 15’’ monitor with a resolution of 1024 x 768, and in black (0.92 cd/m2) on a white background (70 cd/m2). The presentation of stimuli and recording of responses were executed with the E-Prime software, version 1.2 (Schneider, Eschman, & Zuccolotto, Reference Schneider, Eschman and Zuccolotto2002).
Two hundred and ninety four visual stimuli were selected from the combination between 21 consonants and 14 types of graphic fonts. Each visual sequence presented for memorization was formed by the same consonant presented in four different types of fonts. The spatial location of the stimuli was determined from sixteen possible spatial locations uniformly distributed in two concentric circles (hidden during the presentation) with radii of 13.5 cm and 9.2 cm, respectively.
Procedure
Each trial (Figure 1) began with the presentation of a warning sign during 900 ms. Four stimuli were presented after the disappearance of this sign in different positions on the monitor screen, one after the other, during 500 ms each, and with an interval of 100 ms in between. The presentation of the four stimuli in the testing sequence started after a retention interval of 6 sec. The participant’s task was to state if the two sequences were equal or different depending on the relevant dimension in the task. The response was given using the numeric keypad after the presentation of the testing sequence and followed by a feedback signal presented during 100 ms in the center of the screen. The correct response was flagged by a blue square (5 x 5 pixels) and the wrong responses by a red square (5 x 5 pixels). The interval between this sign and the next test was of 400 ms.
Example of a trial defined by visual appearance of stimuli (relevant dimension; positive probe). The serial order is the irrelevant dimension. Depiction is not to scale.
The relevant dimension for the task was manipulated between the groups of participants. One group was instructed to memorize the visual appearance of the stimuli, another to memorize the spatial location of the stimuli, and the third group to memorize the appearance and location of each stimulus (appearance-location conjunction). At the beginning of the session, the participants were briefed on the relevant dimension of the task and instructed to explicitly ignore the order in which the stimuli were presented (irrelevant dimension). The sequences were equal in the relevant dimension in half of the tests and different in the other half. In both cases, the irrelevant dimension (order of stimuli) was equal in half of the tests and different in the other half. The order of presentation was manipulated following seven possibilities of changes between positions of the series during the presentation: (A) without unchanged serial positions (equal serial order between the sequences); (B) changed between the first and second stimuli (1-2); (C) between the first and third (1-3); (D) between the first and fourth (1-4); (E) between the second and third (2-3); (F) between the second and fourth (2-4), and (G) between the third and fourth (3-4).
Each performed trial was drawn from a set of 128 trial created from random combinations between spatial locations, visual appearance, and serial order. Both spatial location and visual appearance were also randomized in the tests with different relevant dimension. There were three possibilities for changes in the relevant dimension in the appearance-location conjunction task: equal visual appearance and different spatial location, different visual appearance and equal spatial location, and different appearance and location.
In each session, each participant performed 100 tests, however, the first 4 were considered to practicing. The relevant dimension was equal in both sequences in half of the valid tests (48) and different in the other half (48 tests). In the equal sequences, half (24) presented the same order and half (24) a different order. In the different sequences, half (24) presented the same order and half (24) a different order. The participants were randomly assigned to groups according to the type of task: visual, spatial, or visual-spatial conjunction.
The participants answered the following question after the sessions: “Which strategy did you use to perform the task just presented?” The answers were recorded in writing by the researcher.
Results
General Analysis
An analysis of variance (ANOVA) with repeated measures was performed considering the type of relevant dimension (visual, spatial, and visual-spatial conjunction) as a manipulated factor between participants, and repeated measurements of equality factors of sequences in relevant and irrelevant dimensions. The answers to the question concerning the strategies used for execution of the task were transcribed, read, and categorized in accordance with the terms used. The Anova test showed that the participants who memorized spatial location presented the best performance (81% of hits on average, SEM = 3%) compared to those in the visual-spatial (M = 74%, SEM = 4%) and visual group, with an average of 64% of hits (SEM = 3%), F(2, 54) = 24.30, p = .001, η2 P = .47. The performance was better when the sequences were equal with regard to relevant dimension (M = 79%, SEM = 2%) than when they were different (M = 67%, SEM = 2%), F(1, 54) = 35.58, p = .001, η2 P = .40. The performance was also affected by the change in the irrelevant dimension: sequences with equal order were best memorized (M = 75%, SEM = 2%) than sequences with different order (M = 71%, SEM = 2%), F(1, 54) = 14.04, p = .001, η2 P= .21.
The performance was also affected by the interaction between changes in the relevant and irrelevant dimensions, F(1, 54) = 14.32, p = .001, η2 P = .21. According to this interaction, the change in the irrelevant dimension affected the performance only in the tests where the relevant dimension remained identical to the memorized set; in this case, the change in order caused an average loss of 9% in the percentage of correct responses. The effect of the change in the irrelevant dimension of around 2% is not significant in the tests in which the relevant dimension changed.
The changes in the order of presentation of stimuli seemed to have an enhanced effect on the group that memorized the spatial location of events (Figure 2). Although the interaction between groups and equality, in the relevant and irrelevant dimensions, were not significant, this difference is confirmed by the post-hoc HSD test by Duncan (p = .001). In order to evaluate in more detail the effect of changes in the irrelevant dimension, the obtained results for each of the groups were subjected to separate analysis of variances taking into account repeated measurements in the equality of the relevant (equal or different sequences) and irrelevant (equal or different order) dimensions.
Correct responses in memory tasks defined by relevant dimension (same, different) in irrelevant dimension conditions (same, different order). (*) Interaction between relevant and irrelevant dimensions; p < .05.
Visual relevant dimension
The performance was better in the tests with equal sequences (M = 72%, SEM = 3%) than with different sequences (M = 56%, SEM = 3%), F(1, 19) = 21.65, p = .001, η2 P = .53, in the group that evaluated the equality of sequences in terms of visual appearance of stimuli. Moreover, the effect of changes in the irrelevant dimension (order) was also observed on the rate of correct responses, F(1, 19) = 10.44, p = .004, η2 P = .35. The performance in the tests with unchanged order of presentation was 67% (SEM = 4%), whereas in the tests with changed order of presentation was 62% (SEM = 3%).
Two strategies were outstanding among the answers to the question prompted to the participants at the end of testing: the distinction of stimuli through visual similarity, and the nomination of stimuli; the first was cited by 15 participants and the second by 9, both out of 20 participants.
Spatial relevant dimension
The analysis of the proportion of correct responses revealed that the performance was better (M = 86%, SEM = 3%) when the sequences were equal than when they were different (M = 76%, SEM = 4%), F(1, 19) = 5.63, p = .03, η2 P = .23, in the group evaluating the equality of sequences in terms of spatial location of events. The performance of the participants was also affected by changes in the irrelevant dimension (order). In this case, memory for sequences with equal order was better (M = 84%, SEM = 3%) than for sequences with different order (M = 78%, SEM = 3%), F(1, 19) = 7.75, p = .012, η2 P = .29.
A statistically significant interaction between the factors in the relevant and irrelevant dimensions was also observed for this group, F(1, 19) = 14.68, p = .001, η2 P = .44. The performance was worse when the order was changed (M = 79%, SEM = 3%) than when unchanged (M = 92%, SEM = 2%) in the tests with equal relevant dimension. On the other hand, when the relevant dimension changed, there was no difference in performance between the conditions with changed order (M = 78%, SEM = 3%) and unchanged (M = 75%, SEM = 4%).
With regard to cognitive strategies, 15 out of 20 participants mentioned drawing an imaginary line between the presented spatial locations, and thus, drawing a trajectory that would be subsequently compared with the testing sequence. Some participants (n = 6) stated establishing a relative distance between the spatial locations, however reported that this strategy was only successful when the spatial locations drawn were considerably far in between.
Visual-spatial relevant dimension
The data analysis revealed that there was a difference between the averages of correct responses for the conditions of relevant dimension (equal vs. different appearance-location conjunction) in the group evaluating the equality of sequences in terms of the conjunction between visual appearance and spatial location. The performance in the tests with equal sequences with regard to relevant dimension was better (M = 80%, SEM = 4%) compared to the tests with different sequences (M = 68%, SEM = 3%), F(1, 19) = 9.07, p = .007, η2 P = .32. In addition, a significant interaction between changes in the relevant and irrelevant dimension was observed. The performance of the participants in the condition of change occurring only in the relevant dimension (M = 65%, SEM = 3%) was inferior compared with the conditions of unchanged dimension between presentation and test (M = 84%, SEM = 3%), and only when the order was changed (M = 77%, SEM = 4%), F(1, 19) = 4.90, p = .04, η2 P = .20.
As explained previously, the condition of memory for visual-spatial conjunction was designed according to four possibilities of manipulation of relevant dimensions between the memorized and tested sequence: equal visual and spatial, different visual and equal spatial, equal visual and different spatial and, different visual and different spatial. These conditions were analyzed based on the irrelevant dimension (equal vs. different order) in order to deepen the study of memory for sequences based on visual-spatial conjunction. The analysis showed a difference in the results for the two conditions of visual dimensions (equal vs. different), F(1, 19) = 12.68, p = .002, η2 P = .40, as well as between the two conditions of spatial location (equal vs. different), F(1, 19) = 22.15, p = .001, η2 P = .53. An interaction between visual dimensions and order was identified demonstrating that the performance in the condition with different visual and equal order (M = 60%, SEM = 6%) was lower than in the other conditions, as for example, when the two dimensions were equal between the sequences (M = 80%, SEM = 4%), F(1, 19) = 4.84, p =. 04, η2 P = .20.
An interaction between visual dimensions and position was observed suggesting that the performance was lower (M = 46%, SEM = 5%) in the condition with different visual and equal location between the sequences if compared with other conditions, for example, when the two dimensions were equal between the sequences (M = 80%, SEM = 3%), F(1, 19) = 32.01, p = .001, η2 P = .63. This interaction raised the need for an analysis of the sequences that were different in regards to the relevant dimension. The analysis of the responses for the condition with different sequences, in regards to the relevant dimension (visual-spatial conjunction), revealed differences among the three possibilities of this condition. The performance in the situation with equal spatial location between the two sequences, but with different visual was lower (M = 46%, SEM = 5%) than in the situation with different spatial location and equal visual (M = 75%, SEM = 4%), and with different location and visual dimensions (M = 82%, SEM = 3%), F(2, 38) = 28.72, p = .001, η2 P = .60. This data corroborate the participants’ statements regarding the strategies used to perform the task: only 8 out of 20 participants used strategies for both dimensions, visual and spatial, to distinguish between equal and different sequences in the visual-spatial conjunction. Twelve participants used only one spatial strategy to perform the task, considered by them easier than trying to identify visual equality too. The most employed spatial strategy (cited by 14 participants) was to imagine a trajectory line between the spatial locations.
Discussion
This study evaluated if the serial position of items in a sequence is incidentally encoded with the items in working memory in a task that requires the storing of visual appearance, spatial location, or the conjunction of visual-spatial events. The performance of the memory task was negatively affected when the serial position of items was modified between the memorized and testing sequences. This result is in agreement with previous studies (Jalbert et al., Reference Jalbert, Saint-Aubin and Tremblay2008; Lecerf & Ribaupierre, Reference Lecerf and Ribaupierre2005; Parmentier et al., Reference Parmentier, Andrés, Elford and Jones2006) and additionally demonstrated the specific effects of irrelevant changes on serial order in the different visual-spatial memory tasks. This denotes that the serial order of events is encoded in an initial stage in the cognitive processing. Because the serial order was encoded in visual and spatial conditions, it can be assumed that there was a loss in the selection of relevant information for the performance of the task.
In the task in which the participants should store visual appearance the performance was affected by both changes in the relevant and irrelevant dimension however, these effects were independent. Considering the Logie’s working memory model for storage of visual information (Logie, Reference Logie2003), only the relevant dimension for the task – visual appearance – received attentive resources and guided the pattern of results, nevertheless, this pattern was affected by the order of events that were possibly stored in memory without requiring additional attentive resources.
There is an alternative possibility that considers that the visual storer is limited in capacity. The integration and maintenance of different dimensions (visual and order) would be functions performed by another component – the episodic buffer – that would be responsible for forming agglomerates of information (chunks), which together would form episodes (Baddeley, Reference Baddeley2012). It should be assumed that the sequence of events is stored as a whole, integrating the visual information and the order in which the events were presented. These statements require further investigations.
In the case of the spatial task, the participants’ performance was also affected by the change in the serial order, i.e., it can be considered that the order was incidentally encoded with the location of events. According to Logie (Reference Logie2003), a specific memory component – the inner scribe – would recite the sequence of movements formed by the presentation of locations allowing the stored information in the visual cache (a passive component) to remain active.
In an alternative interpretation, it is possible that the sequential presentation generated an organization of stimuli in two levels (space and time) (Parmentier, Reference Parmentier, Vandierendonck and Szmalec2011; Parmentier et al., Reference Parmentier, Andrés, Elford and Jones2006). The interaction order-space allows assuming that when the testing and memorized sequences were identical according to the location of the stimuli, but changed for serial order of events, the response to the task switches from spatial to another level – the serial order. In the condition where the second sequence was different from the first, as for spatial location, there would be a change in a superior hierarchical level of information recording – the spatial configuration –, element sufficient to activate the response without the need to trigger the next level, that is, the serial order. A hierarchical relationship in the sequence’s properties occurred: the serial organization at a lower level and spatial structure at a higher organization level.
The hierarchical organization of information mode suggested by the data could be discussed using some current models of visual-spatial working memory (Baddeley, Reference Baddeley2012; Logie, Reference Logie2003). According to Logie’s perspective (2003), the contents previously stored in long-term memory, that contained a hierarchical structure (serial order in a lower level in relation to spatial location), had been activated when the information was encoded. In an alternative way, it is possible that both spatial location and serial position had direct access to working memory and that these information were associated with and maintained in the consciousness by the episodic buffer component (Rossi-Arnaud, Pieroni, & Baddeley, Reference Rossi-Arnaud, Pieroni and Baddeley2006). The maintenance of the associations of characteristics, rather than of a single multidimensional code, would explain a possible hierarchy of dimensions (Morey, Reference Morey2009). This statement is based on the hypothesis that the attentive resources that maintained spatial information are of a serial nature (Cowan, Reference Cowan2005). Future studies could examine how this integrated information was generated and maintained in memory.
Final considerations
The results in this study demonstrated that the negative impact in the performance of the spatial task, due to irrelevant changes in the order, was different from the observed in the visual and visual-spatial conjunction tasks. While there was interaction between the relevant and irrelevant dimensions in the spatial and visual-spatial conditions, the same was not verified in visual task. These results suggested that the information about the object and the position it occupies in a sequence are stored in an integrated manner in memory and recited in a sequential way. In spatial condition, a hierarchical organization of the sequence properties is suggested, in which the serial organization would be subjected to spatial structure. It is not clear whether this operation would be executed with the aid of attentive general resources, through a separate or dependent system (episodic buffer), or simply from the serial displacement of the focus of attention.
The acceptance of the hypothesis that the order of visual and spatial events is encoded in an incidental way - effortlessly - leads to the belief that this function has important repercussions on behavior control, for example, to guide the attention to events that follow a specific serial order, previously encoded in the form of episodes. These considerations have been addressed in studies about the cognitive process in the generation of thoughts about the future, which assume that the storing and retrieving of sequences are functions needed to make predictions, recognize context-related temporal patterns, and generate behaviors (Hawkins, George, & Niemasik, Reference Hawkins, George and Niemasik2009).
