Introduction
Neuropsychologists have been developing and exploring various objective tools to measure what has been described as test-taking “effort” or “symptom validity” for decades (e.g., Larrabee, 2012; Pankratz, Reference Pankratz1979). The development of both free-standing and “embedded” performance validity tests (PVTs; Larrabee, 2012; also known as symptom validity tests, or SVTs) has been successful, with a large body of research supporting their validity in many clinical groups, and relevant professional societies endorsing their use as a practice standard (Bush et al., Reference Bush, Ruff, Tröster, Barth, Koffler, Pliskin and Silver2005). However, some characteristics of PVTs and their use in significantly impaired individuals, including the problem of potential “false positive” errors (PVT classification inaccurately indicative of poor performance validity) remain controversial and inadequately researched. Indeed, an interesting and productive dialogue addressing the state of the science surrounding PVTs, including the “false positive” issue and directions for future research, was recently presented (Bigler, Reference Bigler2012; Larrabee, 2012). Importantly, however, one unresolved issue in the current PVT literature was not addressed directly in that dialogue: the significant variability found in classification accuracy across current PVTs. That is, individual PVTs can vary substantially in their obtained sensitivity and specificity, with clear implications for potential false positive errors among more impaired patient groups. Such variability in measures purported to measure the same construct remains concerning, and efforts to explicate the reasons for these differences are needed.
As noted by Bigler (Reference Bigler2012), much of the extant PVT research has had a generally inadequate guidance from neurocognitive processing theory. That is, while the cognitive literature has typically played a central role in the initial development of PVTs (e.g., the application of knowledge about “chunking” applied to the Rey 15 Item Test or the use of forced-choice recognition (FCR) methodology), there has been an inconsistency with regard to the degree of reliance on these principles in subsequent permutations of PVT methodology (see the wide diversity in design of the numerous PVTs based on forced-choice recognition detailed below). For example, stimuli used in the variety of FCR-based PVTs have included pictures, numbers and words (Green, 2003; Slick, Hopp, Strauss, & Spellacy, 1996; Tombaugh, Reference Tombaugh1997), but the cognitive theoretical implications for varying stimulus modality or other aspects of test methodology for PVT performance has not been adequately studied. This is important, as neuropsychological processing theory provides a framework to understand variability in performance on individual PVTs seen among significantly cognitively impaired clinical groups.
The current review will highlight evidence for such inter-measure variability across PVTs, with a focus on free-standing forced-choice recognition-based measures, and provide an overview of some of the most relevant neurocognitive research as it applies to the diverging ways this methodology has been applied. The aim of this review, consistent with recommendations arising from the aforementioned dialogue, is to bridge the current neurocognitive literature with that addressing contemporary PVTs commonly used in clinical assessments. In addition, the review integrates an existing neurocognitive methodological approach for the systematic evaluation of the individual components of PVTs, providing a methodological direction for the development and evaluation of PVTs based on the cognitive literature.
Classification Accuracy of the Forced-Choice Recognition Memory Paradigm
Most current commonly used PVTs—especially free-standing PVTs—are designed, at least in part, to measure forced-choice recognition memory ability rather than episodic memory (Baddeley, Eysenck, & Anderson, Reference Baddeley, Eysenck and Anderson2009; Green, Lees-Haley, & Allen, Reference Green, Lees-Haley and Allen2002; Tombaugh, Reference Tombaugh1997; Yonelinas, Reference Yonelinas2002). Specifically, while there have been some recent alternative approaches (e.g., the use of “embedded” measures; Lu, Boone, Cozolino, & Mitchell, Reference Lu, Boone, Cozolino and Mitchell2003), the simultaneous forced-choice (SFC) methodology has remained a central methodological approach in identifying invalid effort. In SFC tasks, a person is asked to identify a previously learned target from two simultaneously presented stimuli. Overall, SFC tasks have displayed good validity in detecting suboptimal effort, as they have a reasonable appearance as face-valid measures of memory, while thought to be objectively extremely easy for most examinees providing adequate effort (Baddeley et al., Reference Baddeley, Eysenck and Anderson2009; Chouniard & Rouleau, Reference Chouinard and Rouleau1997).
A meta-analysis by Sollman and Berry (Reference Sollman and Berry2011) evaluated the sensitivity and specificity differences across five SFC measures, with mean obtained sensitivities of 69% and mean specificities of 90% across 47 studies. Of note, however, there was a great deal of variability across measures. For example, sensitivity for the Word Memory Test (WMT; Green, Allen, & Astner, Reference Green, Allen and Astner1996; Green, 2003) ranged from 49% to 100%, with specificity ranging from 25% to 96%, while the Test of Memory Malingering (TOMM; Tombaugh, Reference Tombaugh1997) sensitivity ranged from 34% to 100% with specificity 61–100%. The authors noted the variability across studies and concluded that variables such as the offer of monies to motivate participants, or coaching participants to avoid detection likely affected test performance. Of interest, however, the review stopped short of addressing the potential factors specific to individual PVTs (i.e., characteristics of the tests themselves) that may affect the ability of the test to correctly identify invalid effort (the sensitivity of the test), while avoiding incorrectly identifying valid effort as invalid (the specificity of the test; see Table 1). This is important, as neuropsychologists have generally agreed that specificity must be high (e.g., at least ≥90) for a PVT to be acceptable, even at the expense of lower sensitivity (Boone, Reference Boone2007).
Determination of classification accuracy statistics on symptom validity tests.
Note: Sensitivity=show the disorder in question (e.g., correctly identify a malingerer as malingering); TP/(TP+FN). Specificity=do not show the disorder in question (e.g., correctly identify the non-malingerers as non-malingering); TN/(TN+FP).
While characteristics of the individuals being administered a PVT (e.g., motivation for secondary gain, knowledge about neuropsychological tests, degree and type of cognitive impairment) are important to consider, the characteristics of the PVT—and how they may interact with such individual factors—are equally important. For example, it has been argued that false positive errors (that is, lower specificity) may, in some cases, reflect a greater need for cognitive capacity in successful completion of at least some PVTs (e.g., Bigler, Reference Bigler2012) among more significantly impaired individuals, and evidence to support this claim comes from several converging lines of research.
From a functional neuroimaging approach, multiple studies (e.g., Allen, Bigler, Larsen, Goodrich-Hunsaker, & Hopkins, 2007; Larsen, Allen, Bigler, Goodrich-Hunsaker, & Hopkins, Reference Larsen, Allen, Bigler, Goodrich-Hunsaker and Hopkins2010) have demonstrated that participants completing the WMT displayed activation of neural regions interpreted as reflecting high levels of cognitive effort. Two studies (Batt, Shores, & Chekaluk, Reference Batt, Shores and Chekaluk2008; Loring et al., Reference Loring, Marino, Drane, Finney and Meador2011) using differing methodological approaches to temporarily reduce cognitive function while PVTs were administered both found lower PVT scores under the active condition. Loring et al. (Reference Loring, Marino, Drane, Finney and Meador2011) found a significant reduction in WMT performance among persons receiving a moderate dose of a benzodiazepine. Notably, WMT performance corrected to passing levels of performance when the drug effect cleared. Similarly, Batt et al. (Reference Batt, Shores and Chekaluk2008) found greater reduction in performance on the WMT versus the TOMM in a traumatic brain injury (TBI) sample under a cognitive distraction task condition.
From a clinical groups perspective, while it is well-documented that some individuals with significant cognitive impairment (e.g., bilateral hippocampal atrophy; Goodrich-Hunsaker & Hopkins, Reference Goodrich-Hunsaker and Hopkins2009) can perform well on published PVTs, there is also a growing body of literature indicating that some moderate to severely impaired clinical groups with no incentive to malinger or supply poor effort produce unacceptably low specificity rates, and this appears to vary across individual PVTs (Gorissen, Sanz, & Schmand, Reference Gorissen, Sanz and Schmand2005; Greve et al., 2008; Howe & Loring, 2009; Merten, Bossink, & Schmand, Reference Merten, Bossink and Schmand2007; Rudman, Oyebode, Jones, & Bentham, Reference Rudman, Oyebode, Jones and Bentham2011; Weinborn, Orr, Woods, Conover, & Feix, Reference Weinborn, Orr, Woods, Conover and Feix2003; but cf. Chafetz, Prentkowski, & Rao, Reference Chafetz, Prentkowski and Rao2011; Schroeder & Marshall, Reference Schroeder and Marshall2010). For example, Gorissen et al. found 72% of their schizophrenia sample scored below cutoffs indicative of invalid performance on the WMT, Greve et al. found approximately 20% of their non-malingering TBI group were misclassified with these criteria, and Howe and Loring found 12.5% of their mild-cognitive impairment (MCI) and 38.5% of their mild-moderate dementia sample scored below traditional cutoffs on the Medical Symptom Validity Test (MSVT). Another study using the Victoria Symptom Validity Test (VSVT; Slick et al., 1996), a PVT that uses two performance levels, “easy” and “hard,” concluded that poor performance on the “hard” items may be due to legitimate cognitive deficits (Keary et al., Reference Keary, Frazier, Belzile, Chapin, Naugle, Najm and Busch2013).
Of note, many of these studies have used the traditional “cutoff” approach for FCR-based PVTs, with the recommendations provided by the initial validation studies. However, there can be an evolution in criteria used on any given PVT to classify invalid performance over time, from the initial validation studies to subsequent revisions after use and replication in independent samples, often including clinical groups with known pathology and no known incentive to produce invalid performance. Such changes have often taken place in an attempt to address potential false positive misclassifications, including use of alternate cut scores in dementia (e.g., Colby, 2001). Other PVTs (e.g., the WMT and MSVT) include multiple types of recollection memory measures in addition to an FCR component, and potential false positive errors may be analyzed through patterns of performance across FCR and memory scores, e.g., a “Dementia Profile” (Howe, Anderson, Kaufman, Sachs, & Loring, Reference Howe, Anderson, Kaufman, Sachs and Loring2007). While these approaches represent an exciting growth in the sophistication of evaluation of symptom validity, and include the important consideration of the general cognitive literature, this has generally occurred after the publication of the PVT in an effort to evaluate those who have scored below cutoff on the traditional SFC component of the measure, and do not address which aspects of any given PVT methodology may make it more (or less) vulnerable to false-positive errors in impaired individuals. Therefore, application of neurocognitive theoretical approaches to understand variability in PVTs is a necessary step in evaluating current PVTs, as well as the development of new ones.
Parameters of the Review
Table 2 shows all PVTs reviewed within the context of the selected neurocognitive literature. In an effort to not compromise test security, details about each PVT were strictly limited to what could be reasonably extricated from previously published articles. Reviewing all PVTs was beyond the scope of a single paper, and the current review was limited to common clinically used PVTs with a SFC paradigm (e.g., Sharland & Gfeller, Reference Sharland and Gfeller2007; Strauss, Sherman, & Spreen, Reference Strauss, Sherman and Spreen2006), and an extant research literature base sufficient to allow evaluation of the specific methodology of each PVT within the cognitive literature. The obtained specificity of each PVT was then reviewed in the context of the relevant cognitive component (e.g., stimulus modality or number of unassessed learning trials). Of note, most studies evaluating clinical samples with no known motivation to malinger do not include independent criteria to classify test-taking effort, and specificity values must be interpreted with caution.
Common performance validity tests and their test characteristics
Note. Test abbreviations: 48-PT=48 Pictures Test (Chouinard & Rouleau, Reference Chouinard and Rouleau1997); TOMM=Test of Memory Malingering (Tombaugh, Reference Tombaugh1997); NVMSVT=Nonverbal Medical Symptom Validity Test (Green, 2008); WMT=Word Memory Test (Green, 2003); ASMT=Amsterdam Short Memory Test (Schagen et al., 1997); CVLT-II = California Verbal Learning Test-II (Delis et al., 2000); CARB=Computerised Assessment of Response Bias (Allen et al., 1997); DMT=Digit Memory Test (Hiscock & Hiscock, 1989); MSVT=Medical Symptom Validity Test (Green, 2004); VSVT=Victoria Symptom Validity Test (Slick et al., 1996); PDRT=Portland Digit Recognition Test (Binder, Reference Binder1993).
# Learn=number of learning trials, Feedback=whether feedback on correctness of response was provided, # Stim=number of stimuli required to be learned, Delay Pd=period of time between the end of stimulus presentation and the recognition trial(s), s=seconds; min=minutes, Spec=specificity, the percentage of those with valid performance inaccurately identified as invalid.
a Delay only between testing of the first half and second half of the total test items.
b Two practice items.
c Averaged across samples within study.
d 3 Blocks of 37 items, early termination on Blocks 1 and/or 2 for passing performance.
e Includes a sample trial of two items, as well as an unassessed test trial.
f PDRT averaged sensitivity for “hard” and “total” items cutoffs.
g VSVT specificity of “questionable” performance scores on “hard” items.
The memory and general neurocognitive literature considered most relevant for the SFC methodology in general, and to the diverse ways that methodology has been applied in each PVT reviewed, will be detailed below. Specifically, careful consideration of the learning and memory literature indicates that there are important differences in test stimuli and administration characteristics (summarized for each measure in Table 2) that may contribute to test performance, and subsequent classification accuracy in cognitively impaired individuals.
Importantly, much of the extant literature comes from cognitively healthy samples. Much less is known about how these factors may apply to neuropsychologically impaired individuals, and as PVTs are typically administered to clinical samples, it is imperative that the underlying mechanisms in any PVT that may impact performance are understood. Finally, this review will only be evaluating test characteristics that may affect level of performance among clinically impaired populations—that is, a PVT’s specificity. Reviews of those tests characteristics that may influence a measure’s sensitivity to invalid symptom presentation (e.g., face validity as a memory measure) are beyond the scope of this study.
Cognition and Performance Validity Tests
The number of potential cognitive constructs with implications for PVT performance is large, and not all are included here. Specifically, components that were deemed to be less likely to contribute to the overall PVT performance were not included (e.g., number of stimuli, delay length, use of performance feedback). For example, the number of items to be learned and recalled is an essential feature of learning tasks and well researched in the cognitive literature. However, as FCR literature has repeatedly shown, the capacity for recognition of items is quite robust. Indeed, some studies show FCR capacity in the thousands of items (Standing, Reference Standing1973), and as no PVT considered for this review uses such a high number of items, this characteristic was not included in this review. However, stimuli number may be important for other factors such as enhancing face-validity (and, therefore, a potential factor in PVT sensitivity to suboptimal effort) and should be further explored in this regard.
Upon review, three methodological components that differ across the PVTs, and with the potential to influence the cognitive demands required to learn and process stimuli were identified; (1) stimulus characteristics (e.g., variations in stimulus type and presentation), (2) learning characteristics (e.g., number of learning trials, feedback procedures, unassessed practice trials), and (3) recognition trial characteristics (e.g., the semantic relatedness of foils). Of note, while this review outlines these PVT characteristics as independent, it is acknowledged that the effects of these factors may also interact with each other.
Stimulus Characteristics: Words, Numbers, and Pictures
The types of stimuli commonly used in PVTs are words, numbers and pictures. Research suggests that each stimulus type may engage different memory processes (Baddeley & Hitch, 1974; Baddeley et al., Reference Baddeley, Eysenck and Anderson2009), and much of the cognitive literature suggests that some stimuli have a learning advantage compared to others.
First identified by Shepard (Reference Shepard1967), there is extensive literature around the “picture superiority effect,” which is demonstrated in explicit recall tasks, supporting that pictures are better retained and more easily retrieved than word-based stimuli (McBride & Dosher, Reference McBride and Anne Dosher2002; Paivio, Reference Paivio1991; Shepard, Reference Shepard1967; Standing, Reference Standing1973; Whitehouse, Maybery, & Durkin, Reference Whitehouse, Maybery and Durkin2006). Nelson (Reference Nelson1979) theorized that pictures hold two advantages over words; pictures are more visually distinct from each other than words, and they access meaning more directly than words. Likewise, Paivio’s (Reference Paivio1991) Dual-Coding Theory (DCT) proposed that the redundancy of the verbal and image code elicited by pictures surpassed that of the singular process of verbal stimuli.
Importantly, several recent studies have examined the picture superiority effect among impaired populations. For example, Ally, McKeever, Waring, and Budson (Reference Ally, McKeever, Waring and Budson2009) found event-related potential (ERPs) patterns associated with familiarity and retrieval in patients with amnestic MCI and healthy controls were similar when presented with pictorial stimuli, however, those components were significantly reduced for word stimuli in the patient group.
Conversely, there is evidence that procedures that facilitate deeper processing of target words can enhance learning and potentially overcome the advantage seen for pictorial stimuli (but see Anderson et al., Reference Anderson, Davidson, Mason, Gao, Binns and Winocur2011). For example, Craik and Tulving (Reference Craik and Tulving1975) manipulated the levels of processing of words through asking the participant questions about the words being shown. Results indicated that recognition performance was facilitated by questions that asked the examinee to process semantic information (e.g., “is this a member of category X?”) rather than structural information (e.g., “is this word in capital letters?”). Additionally, the recognition capacity for words can be influenced by moderators such as the word’s use frequency, whereby uncommon words may be easier to recognize and common words are easier to remember (Glanzer & Bowles, Reference Glanzer and Bowles1976; Gregg, Reference Gregg1976; Hoshino, Reference Hoshino1991).
Dehaene (Reference Dehaene1992) posits that numbers may be represented in what is described as the triple-code model. In this model, numerical stimuli are cognitively processed via a quantity system (nonverbal representation of size and distance between numbers), a verbal system (as with words, numbers may be represented lexically, phonologically, and syntactically), and a visual system (whereby numbers are encoded as strings of digits), and activation of any of the systems is dependent on task demands (Dehaene, Piazza, Pinel, & Cohen, Reference Dehaene, Piazza, Pinel and Cohen2003). That is, if asked to say or recognize a string of numbers, only the visual system may be engaged, whereas if asked to make comparisons between numbers, the quantity system may be engaged. Numbers may also evoke a rapid, internal quantity code (semantic information) (Dehaene, Bossini, & Giraux, Reference Dehaene, Bossini and Giraux1993), as well as have an abstract, visual-spatial representation (Dehaene et al., Reference Dehaene, Piazza, Pinel and Cohen2003; Trick & Pylyshyn, Reference Trick and Pylyshyn1993). Indeed, the mental representation of numbers has been generally accepted as being spatially encoded (Gevers, Reynvoet, & Fais, Reference Gevers, Reynvoet and Fais2003), similarly to that of pictures.
Coupled with the evidence that the cognitive processing of numbers appears to be an innate, biologically determined necessity that is independent of language skill development (Ansari, Reference Ansari2008; Pica, Lemer, Izard, & Dehaene, Reference Pica, Lemer, Izard and Dehaene2004), research from as early as the late 1800s has demonstrated the relative robustness of numerosity across several cognitive domains. For example, in the absence of lexical knowledge, visual processing and identification of numbers is preserved (Déjerine, 1891, as cited in Cohen & Dehaene, Reference Cohen and Dehaene1995; Dehaene et al., Reference Dehaene, Piazza, Pinel and Cohen2003), as is the ability to write and read Arabic numbers in persons with severe agraphia and alexia (Anderson, Damasio, & Tranel, Reference Anderson, Damasio and Tranel1990). Thus, it appears that the recognition of numbers may also require fewer cognitive resources, and may be less sensitive to substantial cognitive deficits.
In summary, the cognitive literature suggests that recognition of pictorial and numerical stimuli may have certain advantages over that of words for both healthy and impaired groups, requiring fewer cognitive resources to facilitate deeper encoding. Subsequently, PVTs with these kinds of stimuli may be relatively less sensitive to cognitive impairment. Consistent with this, a visual inspection of the classification accuracy of PVTs with each stimulus type in Table 2 indicates that the false-positive risk appears to be lower (that is, the specificity is higher) with PVTs using numbers {M=91.99; SD=7.81; 95% confidence interval (CI) [87.83–96.16]} than pictures (M=82.38; SD=22.69, 95% CI [73.32–91.43]) or words (M=75.63; SD=23.89; 95% CI [69.29–83.97]). This may be representative of the degree of cognitive ability required for each of the different PVT stimuli being compared. However, as PVTs differ methodologically in multiple ways, it remains possible that any differences seen may be due to factors other than stimulus type, and other task characteristics must be considered.
Learning Characteristics: Number of Trials and the “Test Effect”
Inherent to the SFC methodology is the presentation of the stimuli (the learning trial or trials). Research of the behavioral and neurological mechanisms involved in the recognition of novel stimuli suggests that while a single viewing of the stimulus may be sufficient (Rutishauser, Mamelak, & Shuman, Reference Rutishauser, Mamelak and Schuman2006; Sevostianov et al., Reference Sevostianov, Horwitz, Nechaev, Williams, Fromm and Braun2002; Shepard, Reference Shepard1967; Standing, Reference Standing1973; Standing, Conezio, & Haber, Reference Standing, Conezio and Haber1970), the recognition process may benefit from multiple learning opportunities. For example, Kausler and Yadrick (Reference Kausler and Yadrick1977) showed participants items in either 1, 2, or 4 learning phases, and found that as the number of exposures to stimuli increased, so too did accuracy for old/new discrimination for those studied items. Perhaps more integral to the facilitation of encoding and subsequent retrieval is what is known as the “test effect”, which asserts that successful retrieval (by testing recall) at various intervals during the memory test improves long-term retention when compared to multiple restudy-only phases (Karpicke & Roediger, Reference Karpicke and Roediger2008; Pastötter, Schicker, Neidernhuber, & Bäuml, Reference Pastötter, Schicker, Niedernhuber and Bäuml2011). Therefore, whereas a single viewing may facilitate recognition on its own, recognition is further facilitated by repeated encoding sessions through restudy and, perhaps to a larger extent, multiple testing. This has implications in the design of PVTs, where measures may or may not allow the opportunity for practice and unassessed “testing” trials.
For example, the Test of Memory Malingering (TOMM) provides a brief sample trial (learning and test phase of two items), followed by a complete unassessed learning and testing phase (Trial 1), whereby all stimuli are presented at once (one at a time), and then tested. Subsequently, only Trial 2 and the later Retention Trial are used for classification purposes. This administration methodology may subsequently improve performance due to the test-effect. Indeed, the TOMM studies indicate a mean specificity of 85.52 (SD=17.18, 95% CI [77.12–93.9]). In contrast, the Word Memory Test (WMT) presents the entire stimuli list twice (that is, with untested restudy only) in succession before testing, both immediately and after a delay. Importantly, there is also an internally generated consistency score (CNS) based on inconsistent performance between two testing trials (e.g., “missing” an item originally, but later recognizing it), with inconsistent performance resulting in a low CNS score, which is important as all three scores are used for classification purposes. That is, the examinee may be penalized for “learning” from feedback and correctly identifying a target later (as performance feedback is given). Of note, mean specificity for PVTs using this type of administration and scoring is 68.88 (SD=26.25; 95% CI [58.89–78.86]) in the studies reviewed.
Whereas there have been recent attempts at evaluating the implications for classification produced by these kinds of scoring and administration differences, e.g., by using only the first trials of each PVT (WMT and TOMM together; Bauer, O’Bryant, Lynch, McCaffrey, & Fisher, Reference Bauer, O’Bryant, Lynch, McCaffrey and Fisher2007), and development of a consistency score for the TOMM (Davis, Wall, & Whitney, Reference Davis, Wall and Whitney2012; Gunner, Miele, Lynch, & McCaffrey, Reference Gunner, Miele, Lynch and McCaffrey2012), to our knowledge no research has yet specifically explored the test-effect, or manipulated the use of unassessed learning trials on PVT performance while holding other cognitively relevant factors constant.
Recognition Trial Characteristics: The Nature of the Target-to-Foil Stimuli
The effect of similarity between the target and the foil in recognition memory tests has been well studied with results supporting this relationship as a “powerful” moderator of recognition (p. 221; Kintsch, Reference Kintsch1970). Many studies report an inverse relationship between recognition accuracy and target-foil similarity, with accuracy declining as target-foil similarity increased (Kintsch, Reference Kintsch1970; Shepard & Podgorny, Reference Shepard and Podgorny1978). Of interest, Tulving (Reference Tulving1981) found that in a forced-choice recognition test of scenery, participants were more accurate when the target and foil were similar than when they were different; however, that accuracy seemed to have been at the expense of confidence in their recognition of the target. Tulving suggested that this divergence from the inverse-relationship theory was the result of the similarly linked photos referring to one memory trace, whereas the dissimilar target and foil were matched to two different traces. Furthermore, a “test-pair similarity effect” was identified as a phenomenon of forced-choice recognition whereby it was demonstrated that recognition was more accurate for word pairs such as “cheetah-leopard” than unrelated words such as “leopard-turnip” (Hintzman, Reference Hintzman2001). However, “distinctiveness theory” posits that these separate memory traces allow for deeper elaboration (Hunt, Reference Hunt2006).
Semantic interference can be thought of as a competitive effect in that the presentation of the target word may activate all of the other words that the examinee may know that were related to the target, and may not be able to inhibit those other words from influencing their response (de Zubicaray, Wilson, McMahon, & Muthiah, Reference de Zubicaray, Wilson, McMahon and Muthiah2001). It has been argued that perhaps this semantic interference has been a factor in PVT performance. For example Batt et al. (Reference Batt, Shores and Chekaluk2008) suggested that the semantically related target-to-foil choices in the WMT may pose difficulties in examinees unable to inhibit incorrect responses. For example, in the WMT learning phase the participant may be presented with a word such as “SHOE,” and in the recall phase the participant is asked to select this word from a semantically related foil of “BOOT.” Competition within the semantic network may result in recollection that the target has “something to do with feet,” and the participant may select “BOOT” erroneously, thus producing an incorrect answer, despite putting forth optimal effort.
Although no current PVT literature examines the impact of the relationship between target and stimuli and impaired samples, a recent study by Bayley, Wixted, Hopkins, and Squire (Reference Bayley, Wixted, Hopkins and Squire2008) found that memory-impaired patients with hippocampal damage were impaired on tests of recognition memory (both “yes/no” and FCR) where the target and foils were very similar, suggesting that individuals with hippocampal damage may have impaired recognition performance on an PVT if the target and foil concepts are highly related.
In light of converging lines of research that indicate that the relatedness of the stimuli can either facilitate or hinder recognition in both cognitively healthy individuals and impaired samples, it is important that this factor is studied comprehensively across PVTs and clinical groups. Consideration of the PVTs in Table 2, the semantic relationship between the target and foils is only different on the TOMM, and some portions of the MSVT, and firm conclusions are difficult to draw based on this inconsistency.
Summary and A Way Forward
Considered within the neurocognitive literature, there are several important factors that are potentially relevant to how healthy and impaired groups may perform on PVTs with differing methodologies, and it is possible that one or more of these factors is influential in producing the variable classification accuracy rates among existing PVTs. It is important to note, however, that these PVTs may vary on several constructs (e.g., type of stimulus and number of learning trials) and as such, the differences seen may be due to multiple factors, or combinations of factors. Understanding exactly which of these constructs may affect PVT performance and, more importantly, how they may affect performance must be better understood to most effectively design new PVTs, as well as interpret the variable classification accuracy seen in existing PVTs.
Evaluation of PVTs in clinical samples involves both practical and methodological challenges. Specifically, recruiting individuals with significant cognitive impairments can be difficult, and from a methodological standpoint, assuming that all individuals in a clinical sample are providing valid effort is never a certainty. The clinical equivalent of the simulation design (e.g., see Rogers, Reference Rogers2008 for a detailed description), which assists in evaluating the ability of a prospective PVT in identifying invalid effort through facilitating an analogue malingering group, is essential and long overdue.
It is well established in the cognitive literature that increasing the cognitive load (typically by way of various distraction tasks) at various points of learning and/or retrieval, is an effective tool for the understanding of how memory operations work (for a review of Cognitive Load Theory, see Kirschner, Ayres, & Chandler, Reference Kirschner, Ayres and Chandler2011). Reducing available attentional resources (increasing cognitive load) by way of a distraction task has been argued to simulate the effects of aging, alcohol intoxication, and certain types of drugs on memory systems (Castel & Craik, Reference Castel and Craik2003; Craik, Reference Craik1982). Importantly, differences in working memory capacity should not arise if a task can be performed with automatic processing (see Tulhoski, Engle, & Baylis, Reference Tuholski, Engle and Baylis2001), as is the case with recognition memory (Baddeley et al., Reference Baddeley, Eysenck and Anderson2009; Yonelinas, 2003). However, the application of cognitive load manipulation has only recently been implemented as a mechanism for understanding the constructs applied in PVTs.
Batt et al. (Reference Batt, Shores and Chekaluk2008) used a simple addition distraction task during the learning phases of both the TOMM and WMT with a group of individuals with severe TBI. As previous research with varying levels of acquired brain injuries revealed that the addition of a distraction task did not significantly impact automatic memory processing (i.e., implicit memory tasks) when compared with healthy distracted controls (Schmitter-Edgecombe & Nissley, Reference Schmitter-Edgecombe and Nissley2000; Watt, Shores, & Kinoshita, Reference Watt, Shores and Kinoshita1999), Batt et al. postulated that any performance difficulties on an PVT would indicate that processing other than automatic would be required to avoid misclassification. When given under usual standardized administration (that is, without the cognitive load manipulation), 44% scored below the usual cutoff the WMT compared with 16% on the TOMM. However, Batt et al. found these fail rates substantially increased under conditions of increased cognitive load: WMT (75%) versus TOMM (33%).
Whereas it could be reasonably argued that a sample of severe TBI participants would have a higher risk of false-positive classification on either PVT (as is stated by both test authors), this study does raise the notion that borrowing from the already well-validated cognitive load literature genre may provide an avenue for the application of a methodology within healthy individuals that allows clinicians and researchers to systematically evaluate constructs in existing PVTs as well in the creation of new PVTs, analogous to the simulated malingering methodology (e.g., test coaching, symptom coaching, etc.). However, an application of this methodology to the PVT literature remains untested in cognitively healthy individuals. That is, the use of a distraction task in a healthy sample may provide further insight as to the cognitive demands required to successfully complete PVTs that vary methodologically in the ways reviewed above.
Initial Validation of a Clinical Analogue Methodology (CAM) for PVTs
To evaluate this approach in a healthy sample, an extension of the Batt et al. study to a cognitively healthy sample, was completed. The distraction task was obtained from Batt et al. and consisted of an audio presentation of single digits being read aloud, and participants in the CAM were instructed to mentally add three to that number and state their answer aloud to the administrator. Healthy young (Age M=24.86; SD=9.36) participants were screened and removed for psychiatric, medical, and litigation confounds. Approval for the procedures was granted by the Human Research Ethics Office at the University of Western Australia. Full effort performance from distracted and nondistracted individuals was measured on the TOMM and the WMT (administered in counterbalanced order). The difference between the CAM and healthy participants in the current study was significant for both WMT (t(52) =4.80; p=.000; Cohen’s d=.46) and TOMM T2 (t(52) =2.32; p=.02; Cohen’s d=.47) scores. When results of the healthy distracted sample in this experiment are compared with Batt et al.’s non-distracted (that is, standard PVT administration) severe TBI sample, clear performance similarities emerge. Results obtained from Batt et al. (personal communication, June, 2010) revealed that mean performance scores on the measures used for classification purposes (the lowest of the WMT IR, DR, and CNS percent correct versus TOMM Trial 2 per cent correct) by their non-distracted severe TBI sample (n=25) were: WMT lowest =82.70 (SD=14.08) and TOMM T2=94.48 (SD=12.53). Mean performance on the same measures in the healthy CAM sample (n=27) in our current study were WMT lowest =88.24 (SD=9.40) and TOMM T2=98.74 (SD=2.61), compared with the nondistracted participants (WMT lowest =97.61; SD=3.77; TOMM T2=99.92, SD=0.38). Single sample t tests indicate nonsignificant, but trend-level differences of moderate effect size (WMT lowest; t(40.9) =−1.65; p=.051; Cohen’s d=.52. TOMM T2; t(25.4) =−1.67; p=.054; Cohen’s d=.66) between our healthy CAM participant group and means reported for Batt’s nondistracted TBI sample. Overall, results suggest the current CAM manipulation adversely affected the performance on the PVTs compared to healthy participants not subjected to the CAM, to a level marginally higher than that of a severely impaired clinical group.
Of note, there are a variety of ways to manipulate the strength of the cognitive load manipulation (Paas, Tuovinen, Tabbers, & Van Gerven, Reference Paas, Tuovinen, Tabbers and Van Gerven2003), and further research would be useful in directly comparing healthy CAM participants to reference clinical groups with known levels and patterns of cognitive impairment, and potentially manipulating the cognitive load manipulation to better simulate these kinds of cognitive impairments. For example, in an unpublished study, Shores and Walker (Reference Shores and Walker2007) used the distraction task as outlined above as well as a “severe” reduced cognitive capacity condition (recall of an audio presentation of five numbers in reverse order of presentation), demonstrating a marked performance discrepancy between their two healthy-distraction groups whereby the “severe” manipulation produced more deleterious effects on the PVT than the “mild” condition.
In summary, these results suggest that: (1) increasing cognitive load reduces FCR performance for healthy individuals, (2) implementation of the CAM shows promise for simulating PVT performance of a clinically impaired sample, (3) the CAM subsequently may be useful in evaluating the level of cognitive ability required by varying the range of relevant components of any given PVT (e.g., stimulus modality, number of learning trials), with subsequent implications for classification accuracy in significantly impaired clinical groups. Therefore, the CAM shows promise as a useful part of the process of initial validating new PVTs, along with existing methodologies (analogue malingering, known groups, etc.). Furthermore, the CAM’s ease of implementation in comparison to clinical known groups (e.g., Greve et al., 2008) or drug administration paradigms (Loring et al., Reference Loring, Marino, Drane, Finney and Meador2011) would facilitate replication by independent research groups. Of note, in the final stages of preparing this manuscript Barhon, Batchelor, Meares, Chekaluk, and Shores (Reference Barhon, Batchelor, Meares, Chekaluk and Shores2014) reported similar success in using a dual task paradigm to simulate cognitive impairment in healthy adults, further supporting the potential utility of this approach.
Summary and Conclusions
While the classification rates of most commonly used PVTs are often treated as being roughly equivalent, these tests have demonstrated variable classification rates across a range of impaired clinical samples. One likely explanation for some of this variability is the cognitive load implications of the divergent methodological choices in how the FCR approach has been applied across each of these PVTs. The neurocognitive literature provides guidance in interpreting some of the patterns seen in the current literature, with PVTs that use numbers or pictorial stimuli, and that allow multiple learning trials with the opportunity to benefit from performance feedback and the “test effect” appearing, in general, to have higher specificity across studies including those with significantly impaired samples. Conversely, with regard to the nature of the target and foil semantic relationship, the cognitive literature indicates that this relationship can either facilitate or inhibit the FCR system, and as such, must be further explored.
However, almost all commonly used PVTs vary on multiple relevant neurocognitive constructs, and without systematically controlling for other relevant characteristics, these conclusions remain tentative. Of note, though, the importance of considering these factors is not a new observation (e.g., Bigler, Reference Bigler2012; Rudman et al., Reference Rudman, Oyebode, Jones and Bentham2011), and PVTs with methodologies that require greater cognitive resources have generally been adapted in the validation process (e.g., use of lower cutoff values for some word-based, compared to picture-based PVTs), or after (e.g., development of the “dementia profile” for PVTs that have the advantage of FCR plus memory measures with a known gradient of difficulty). Importantly, however, the specific contribution of these known neurocognitive factors and their influence on PVT design and interpretation has not been systematically evaluated thus far, particularly for severely cognitively impaired individuals. One reason for the relative lack of such a comprehensive research literature is the difficulty in conducting these kinds of studies with clinical populations.
Cognitive load theory provides a framework with which we can systematically evaluate the above-mentioned differences. Specifically, implementation of the Batt et al. (Reference Batt, Shores and Chekaluk2008) or other CAM (i.e., cognitive load) approach would allow researchers to vary the components of the SFC methodology and evaluate the results in an analogue clinical group, similarly to what is currently applied in the evaluation of PVT sensitivity values with simulated malingering scenarios. As shown by the data presented in this study, there is some initial support for the use of this methodology in PVT research in cognitively healthy individuals. Importantly, the characteristics of the cognitive load task used is also likely to affect how “impaired” the analogue clinical group performs on PVTs. The degree to which the task taxes cognitive resources can be manipulated, and it is possible that further research in this regard can improve the accuracy of the analogue clinical group in simulating more severe impairment. Conversely, development of a simpler distraction task may provide a simulation analogous to a mildly impaired sample.
In addition to evaluating the factors identified in this review, the CAM approach would facilitate the synergy between PVTs and the larger neurocognitive literature. Application of these interactions in developing new free-standing and embedded PVTs is a particularly exciting possibility. For example, use of procedures that facilitate deeper processing of target words (e.g., asking participants questions that engage semantic processing) can enhance learning and potentially overcome the advantage seen for pictorial stimuli (e.g., Craik & Tulving, Reference Craik and Tulving1975; Glanzer & Bowles, Reference Glanzer and Bowles1976; Gregg, Reference Gregg1976; Hoshino, Reference Hoshino1991), while still maintaining the face validity of the PVT as a memory measure.
Additionally, the analysis of patterns of performance typically seen in the learning process may be useful. For example, primacy and recency effects create a U-shaped curve in recall performance known as the serial position effect, which is seen across healthy individuals of all ages (Carlesimo, Sabbadini, Fadda, & Caltagirone, Reference Carlesimo, Sabbadini, Fadda and Caltagirone1997), and has shown some initial promise as a performance validity indicator (Bernard, Reference Bernard1991). Similarly, the “temporal grouping effect” seen, for example, when a six-item auditory list is explicitly grouped by inserting an extended temporal pause between the third and fourth items (e.g., ABC-DEF), resulting in improved recall accuracy compared to ungrouped lists (e.g., ABCDEF) (Maybery, Parmentier, & Jones, Reference Maybery, Parmentier and Jones2002), and is another phenomenon commonly understood in the neurocognitive literature that may be useful as a performance validity indicator.
Acknowledgments
The authors acknowledge Dr. Paul Green for allowing the use of the WMT used in this study, as well as Kellie Batt and Arthur Shores for providing the materials for the distraction task and unpublished manuscript. We also thank Dr. Steven Woods for his very helpful comments on earlier drafts of this study. No conflicts of interest are noted.
