Introduction
Each year, many people are affected by moderate to severe traumatic brain injury (TBI); it is well established that these individuals often experience residual cognitive impairments especially in the areas of attention, memory, and executive functioning (Whyte, Hart, Laborde, & Rosenthal, Reference Whyte, Hart, Laborde, Rosenthal, DeLisa, Gans and Walsh2005). Changes in processes of self-appraisal after TBI have since emerged as an area of interest. Documented relationships between self awareness after TBI and patient outcome highlight the importance of examining and understanding these processes after injury (Evans, Sherer, Nick, Nakase-Richardson, & Yablon, Reference Evans, Sherer, Nick, Nakase-Richardson and Yablon2005; Flashman & McAllister, Reference Flashman and McAllister2002; Ownsworth & Fleming, Reference Ownsworth and Fleming2005).
In the late 1970s, Flavell used the term metacognition to describe awareness of one's own cognitive processing (Flavell, Reference Flavell1979). More recently, work has drawn distinctions between metacognitive knowledge (awareness of how one performs in a certain cognitive domain in general) and metacognitive experience (ability to track “online” how one is performing during cognitive processing). Metacognitive knowledge following TBI is assessed by using questionnaires of self report and provides useful information regarding the individual's ability to gauge functioning on a global scale; however, there are also important advantages afforded by in-the-moment measurements of metacognitive experience (Cosentino & Stern, Reference Cosentino and Stern2005; also see Chiou, Carlson, Arnett, Cosentino, & Hillary, Reference Chiou, Carlson, Arnett, Cosentino and Hillary2011, for discussion of the distinction between metacognitive knowledge and experience, and the use of self-report questionnaires). Measurements of metacognitive experience are made by collecting judgments of performance while engaging in a cognitive activity. Judgments that predict performance and that occur immediately before engaging in a cognitive task are known as prospective monitoring judgments. The focus of this study is on reflective judgments collected immediately after engagement in a cognitive task, or retrospective confidence judgments (RCJs) (see Nelson & Narens, Reference Nelson and Narens1990, for a review of this methodology).
Studies examining the metacognitive experience of individuals with TBI remain inconclusive. Within the domain of metamemory (online tracking of performance during a memory task), individuals with TBI have been found to be less accurate in their predictions of memory performance compared to healthy adults (Kennedy, Carney, & Peters, Reference Kennedy, Carney and Peters2003); however, other researchers found that adults with TBI were able to make accurate predictions of their performance on memory tasks (Anderson & Schmitter-Edgecombe, Reference Anderson and Schmitter-Edgecombe2009). When examining retrospective monitoring judgments, Kennedy (Reference Kennedy2001) observed that adults with TBI and healthy controls did not differ in their accuracy of RCJs made during a metamemory task; however, adults with TBI were overconfident in their judgments, while healthy adults were underconfident. In our work, adults with TBI were found to have significantly less accurate RCJs on a memory task compared to healthy individuals (Chiou et al., Reference Chiou, Carlson, Arnett, Cosentino and Hillary2011). Interestingly, such differences in accuracy did not appear when measuring awareness of performance during tasks of executive functioning, suggesting that metacognitive functioning may depend on the cognitive domain being reflected upon or may be influenced by other variables inherent in the methodology of assessment (Chiou et al., Reference Chiou, Carlson, Arnett, Cosentino and Hillary2011).
Indeed, studies have found that individual and/or task characteristics can influence metacognitive performance (for review, see Lin & Zabrucky, Reference Lin and Zabrucky1998). In the literature examining metacomprehension (e.g., being able to assess one's ability to comprehend a reading passage), the difficulty of the item is found to have an effect on the accuracy of one's metacognitive judgments. Thus far, two competing findings have emerged; some researchers have found that metacognitive judgments were most accurate when task items were easy (Lin, Zabrucky, & Moore, Reference Lin, Zabrucky and Moore2002). This finding had been previously observed during a procedure testing metamemory (Maki & Swett, Reference Maki and Swett1987). In contrast, Weaver and Bryant (Reference Weaver and Bryant1995) argue for the “optimum effort hypothesis,” where highest metacognitive accuracy occurs when individuals engage in cognitive tasks that match their level of functioning.
A separate, implicit manipulation that can be made to cognitive tests is the presentation sequence of item difficulty. In neuropsychological assessment, several standardized tests are designed so that item difficulty progresses from least to most difficult to increase efficiency of test administration and provide potential process observations—an example of this is found in the subtests of Wechsler's Intelligence Scales (Boone & Kaplan, Reference Boone and Kaplan1993; Wechsler, Reference Wechsler1981). Although this sequence of progressive difficulty is standard in neuropsychology, it remains unknown whether metacognitive accuracy is affected by this type of task demand variable. It is possible that individuals engaging in cognitive testing bring with them expectations regarding task procedures based upon schooling, previous testing, and even pop culture; in particular, they may expect that tasks “should” increase in difficulty as they progress. If such an expectation does exist, then it may influence the reported metacognitive confidence judgment.
There is reason to believe that an individual's expectations do indeed influence judgments of cognitive performance. A review of the literature suggests that healthy individuals hold pre-conceived expectations of performance that can ultimately affect their metacognitive abilities (Zhao & Linderholm, Reference Zhao and Linderholm2008). Specifically, an “anchoring and adjustment” model has been posited whereby metacognitive judgments are made based upon previous experiences and expectations (serves to provide an initial “anchor” judgment), as well as current experiences of the task (allows for “adjustments” to be made to the anchor judgment) (Zhao & Linderholm, Reference Zhao and Linderholm2008). If this is the case, we would anticipate that expectations about the sequence of item difficulty establish the anchors for future metacognitive judgments.
It has been established that metacognitive judgments can be influenced by aspects of the individual and of the test items. While an effect of item difficulty on metacognitive performance has been documented, the influence of a manipulation in the sequence of how items with varying difficulty levels are presented on metacognition remains unexamined. Given the extensive clinical use of cognitive assessments that are structured with a particular progression in item difficulty, it would be beneficial to determine the potential effects of this assessment variable on metacognitive functioning in clinical populations. This study investigated the effect of item sequence on metacognitive confidence judgments made by adults with moderate to severe TBI. We hypothesize that item sequencing will affect metacognitive accuracy. Specifically, we anticipate that participants will demonstrate better metacognitive accuracy when items are presented in order of increasing difficulty compared to when the items are in random order based upon difficulty.
Methods
Participants
Twenty-two adults with moderate and severe TBI and 22 age and education matched healthy adults participated in the study. Injury severity was determined by a Glasgow Coma Scale score at the time of admission into the hospital. All enrolled subjects with TBI demonstrated a GCS score between 3 and 12 and/or the presence of positive findings on neuroimaging scans. Demographic information for both groups is presented in Table 1. Individuals with a history of colorblindness, psychiatric illness, or substance abuse that required inpatient treatment were excluded from the study. All participants received and provided written consent to participate in the study as approved by the Institutional Review Board at the institution.
Means and standard deviations of demographic information, cognitive performance, and metacognitive accuracy for adults with TBI and healthy adults
‡Denotes significant between group (TBI and healthy) difference at p < .005.
*Denotes significant between group (TBI and healthy) difference at p = .001.
†Denotes significant within group difference at p < .001.
Materials and Procedure
All participants completed the Matrix Reasoning subtest of the Wechsler's Adult Intelligence Scales-III (WAIS-III) (Wechsler, Reference Wechsler1997), a task involving pattern recognition in a series of stimuli and is traditionally used to measure abstract reasoning. Tasks assessing memory, visual attention, crystallized intelligence, inhibition and switching abilities were also administered (listed in Table 1). The stimuli in the Matrix Reasoning task were manipulated to examine the effects of item sequencing. The original 26 stimuli of the subtest were divided into two sets of 13. The same level of difficulty was maintained in both sets by separating every other item of the stimuli from the original subtest. For “ordered stimuli,” the original sequence of difficulty in the published version of the test (all even numbered items from easy to difficult) was maintained. However, in a second set of stimuli, the “random stimuli” (or all odd numbered items), the order of presentation was pseudo-random, so that the level of difficulty no longer progressed from easy to difficult. The standard rule for discontinuation was not used; each participant received all items in both sets of the stimuli but was not informed of the manipulation. The presentation order of the sets (ordered sequence first or random sequence first) was counter-balanced between the participants.
To collect RCJs, participants were first instructed to determine the correct answer and then after each response they were asked how certain they were that their answer was correct. A 6-point Likert scale was provided for participants to rate their confidence; they could be “completely certain, certain, somewhat certain, somewhat uncertain, uncertain, or completely uncertain.” The item responses and RCJs were coded so that a Goodman and Kruskal's gamma coefficient could be calculated (Goodman & Kruskal, Reference Goodman and Kruskal1954). The coefficient ranges between negative one and one, with higher values denoting higher metacognitive accuracy.
Results
The mean performance scores of both groups of participants while performing all cognitive tasks are reported in Table 1. Independent samples t tests indicated that adults with TBI performed worse than healthy adults on tasks of visual attention (t(22.1) = 3.6; p = .002), switching and inhibition (Stroop t(32.2) = −3.1; p = .004; Trails B t(22.8) = 3.5; p = .002), and crystallized intelligence (t(39) = −3.3; p = .002). No statistically significant difference was found on the task of working memory (t(42) = −1.7; p = .1). Using a mixed factorial analysis of variance (ANOVA), the healthy adults were found to perform better in both sets of the ordered and random sequence stimuli than the adults with TBI (F(1,42) = 14.2; p = .001). A significant main effect of item sequence on actual performance was also found (F(1, 42) = 14.3; p < .001), where participants performed better when the items progressed in difficulty irrespective of group. No significant interaction was found between the effect of item sequencing on actual performance and group (TBI or healthy adult) membership (F(1,42) = 0.8; p = .38). The mean and standard deviations of raw performance scores on the Matrix Reasoning task are shown in Table 1.
Mixed factorial ANOVA was also used to examine the effect of item sequence on metacognitive accuracy in the two sample groups. Participants were shown to have higher gamma coefficients (more accurate metacognitive judgments) when items were administered in order; a main effect of item sequence on metacognitve accuracy was found to be significant (F(1,42) = 29.5; p < .001). There was no significant effect of group membership on metacognitive accuracy (F(1,42) = 0.17, p = .68). No significant interaction was found between the effect of item sequencing on metacognition and group membership (F(1,42) = 1.13; p = .30). The mean and standard deviations the gamma coefficients are presented in Table 1.
Discussion
The findings here reveal that the order of task items over the course of this problem solving task influenced both performance and evaluation of performance. There was a significant main effect of item sequence on cognitive performance and metacognition, and participants from both groups benefitted (reported more accurate metacognitive judgments) when the stimuli were presented in order of increasing difficulty. This finding suggests that both individuals with TBI and healthy adults draw from implicit task characteristics, ultimately influencing task performance and enhancing metacognitive accuracy.
Certainly it appears that there are implicit task demand characteristics that influence both task and metacognitive performance, but the consistency in the findings between groups was unexpected. One explanation is that the participants maintained a latent expectation that performance should deteriorate as the task progresses. When examining mean responses, however, we observe a negative slope in the confidence ratings as the ordered sequence task progresses, but not for the random sequence task (see Fig. 1). This suggests that it is not simply an “expectation” of increasing task difficulty over the course of the task that is disrupting confidence ratings in the random sequence. According to the “anchor and adjust” model, it may be that presenting the items in order of difficulty offers the participant an accurate “anchor” for judgment. However, incorrect expectations regarding item sequence may result in unreliable anchors that either misinform the judgment or are abandoned altogether resulting in either erroneous or reduced information for making metacognitive judgments.
Mean confidence ratings reported for Ordered and Random sequence tasks. Graphs tracking the mean confidence rating reported by participants through the progression of both Order and Random sequenced metacognitive tasks.
This account does not, however, explain why all participants answered more items correctly when completing the ordered task compared to the randomly sequenced task. Given that the level of difficulty for both tasks should have been the same, the difference in performance was a surprising finding. One potential explanation for this is that encountering a difficult item early on during the task may cause the participant to mis-calibrate the difficulty of future items. The expectation that future items will be more difficult (and thus have a higher likelihood of eliciting an incorrect response) may change one's problem-solving approach.
The findings here may have broader implications for clinical assessment and patient rehabilitation. For example, the difference in performance on tasks that can occur based on this manipulation highlights the importance in establishing assessment methodology to measure the most accurate representation of cognitive performance. The performance differences observed here serve as a reminder that task performance is not immune to nuances in the assessment procedure; by changing the order of the items, we essentially made the task more difficult without changing task load or content. Secondly, the ability for individuals to perform better when the task structure is clear or when task items progress in difficulty may be important information for patient rehabilitation efforts. In particular, it may be beneficial to examine whether prospective monitoring judgments are affected by such manipulations, as patients may be able to use compensation strategies if they are able to accurately recognize their performance.
The findings reported from this study serve as a preliminary point from which future studies should expand; the data here reveal an effect of item sequencing and a logical next step would be to explore why or by which mechanisms this effect occurs. For instance, future work could investigate whether conscious awareness of task structure exists and how this influences individuals’ task performance and metacognitive judgments. Also, it is noted that this study specifically examined participant judgment in an abstract reasoning task where answer choices were presented and remained present for evaluation during the Likert rating which may have distinct consequences for performance ratings. Self-generated responses have been shown to elicit distinct confidence ratings (see Chiou et al., Reference Chiou, Carlson, Arnett, Cosentino and Hillary2011, for a discussion of such findings); thus, future studies should explore whether the item sequencing effects found here extend to other types of tasks and in other metacognitive domains.
Despite performing poorer on the task, the metacognitive accuracy of adults with TBI did not differ from healthy adults. Of note, this group of individuals with TBI did demonstrate metcognitive deficits on tests of memory (see Chiou et al., Reference Chiou, Carlson, Arnett, Cosentino and Hillary2011). This finding suggests that metacognition may require different processes and may be differentially affected following TBI. These data demonstrate that performance and metacognition are not necessarily coupled and that even after moderate and severe TBI, individuals are able to detect subtle task characteristics that influence performance even if those demand characteristics are not made explicit.
Conclusions
Studies in cognitive and educational psychology have established that individual and assessment factors are capable of influencing the accuracy of metacognitive experience. This study manipulated the sequencing of item difficulty to examine its effect on metacognitive monitoring abilities following moderate and severe TBI. A significant effect of item order was found as participants in both groups performed better and had more accurate metacognitive judgments when completing items sequenced in a manner of increasing difficulty. These data indicate that implicit task characteristics create expectations that influence both task performance and judgments about performance.
Acknowledgments
The authors thank Neal M. Fitzpatrick, Tia Bochnakova, Julia Slocomb, and Britney Wardecker for their assistance in recruitment and data collection. The authors declare that there are no conflicts of interest associated with this study and manuscript. This study was funded in part by a grant from the New Jersey Commission on Brain Injury Research (grant number 0120090178).
