Statement of Research Significance
Research Question(s) or Topic(s): There is currently no agreement about how many performance validity tests should be failed in order to consider neuropsychological test data uninterpretable. This study examined the relationships between zero, one, and multiple performance validity test failures and cognitive test performance in patients with traumatic brain injury. Main Findings: Cognitive test performance of participants who failed one performance validity test was moderately worse than those who failed zero, and more similar to that of those who failed two or more. Study Contributions: This work suggests that failure of even one performance validity test should raise concern about the validity of cognitive test findings in persons with traumatic brain injury. Such failure should be considered in the context of patient background characteristics. In clinical practice, it should be addressed in a way that reduces harm and supports the most adaptive outcomes.
Introduction
Inclusion of at least two performance validity tests (PVTs) is standard of care in clinical neuropsychological assessment to allow conventional interpretation of cognitive test performance (Sweet et al., Reference Sweet, Heilbronner, Morgan, Larrabee, Rohling, Boone, Kirkwood, Schroeder and Suhr2021). However, there is no consensus about how many PVTs should be failed in order to consider clinical data invalid (Lippa, Reference Lippa2018). Some authors (e.g., Schroeder et al., Reference Schroeder, Martin, Heinrichs and Baade2019) have suggested that a criterion of ≥ 2 PVT failures provides the most robust information, whereas others have argued that even a single PVT failure may be concerning, depending on evaluation context and examinee characteristics (Denning, Reference Denning2023). Again, others have argued that there is too much collinearity among PVTs (Leonhard, Reference Leonhard2023), although this view has been challenged on statistical grounds (Rohling et al., Reference Rohling, Larrabee and Davis2025). A recent study suggested a dose-dependent relationship such that additional PVT failures resulted in progressive lowering of neuropsychological test scores (Kukla et al., Reference Kukla, Finley, Kent, Newkirk, Payne, Fegter and Soble2025).
Performance validity has been studied most extensively in samples of persons with traumatic brain injury (TBI; Azeredo et al., Reference Azeredo, Geraldo, Moreira and Barbosa2024). TBI is one of the most common acquired neurological conditions (Karamian et al., Reference Karamian, Lucke-Wold and Seifi2024). It is often associated with cognitive deficits, particularly when there is prolonged loss of consciousness and/or evidence of acute intracranial lesions on neuroimaging (Roebuck-Spencer & Sherer, Reference Roebuck-Spencer, Sherer, Morgan and J.H.Ricker2018; Stucky et al., Reference Stucky, Kirkwood, Donders, Stucky, Kirkwood, M. and Donders2020). However, studies have been inconsistent in how they handled participants with TBI who failed only one PVT. For example, Proto et al. (Reference Proto, Pastorek, Miller, Romesser, Sim and Linck2014) reported that individuals failing only one PVT performed significantly worse on various cognitive tests than those who failed no PVTs. In contrast, in a recent study (Truter, Reference Truter2025), persons who failed one PVT were grouped with those who failed zero. In several other studies, persons who failed only one PVT were considered indeterminate and excluded from the analyses (e.g., Henry, Reference Henry2023; Uiterwijk et al., Reference Uiterwijk, Stargatt and Crowe2022). However, this method may cause spectrum bias by contrasting extreme groups (i.e., those who failed multiple PVTs vs. those who failed none), which could potentially overestimate classification accuracy (Schroeder et al., Reference Schroeder, Martin, Larrabee, Schroeder and Martin2021). For these reasons, the purpose of this exploratory study was to evaluate the correlates of zero, one, and multiple PVT failures on cognitive test performance in persons with TBI. In addition, we sought to include primarily clinical participants with a broad range of injury severity because most prior studies focused on uncomplicated mild TBI and/or were limited to participants who were engaged in financial compensation-seeking.
Methods
This article reports a retrospective analysis of data on patients with TBI. It was approved by the institutional review board at Mary Free Bed Rehabilitation Hospital (#2025.01) and conducted in accordance with the Helsinki declaration.
Participants
We retrieved archival data from a secured database on persons who had undergone an outpatient neuropsychological evaluation over a period of almost eight years, using the following selection criteria: (a) diagnosis of TBI, defined as an acute external force to the head with associated change in consciousness, (b) medically stable, (c) completion of a neuropsychological evaluation within 1–36 months after injury, with inclusion of the Trail Making Test (TMT) and at least two independent PVTs, of which at least one should be a stand-alone instrument. The TMT had routinely been administered during this period unless there were contraindications (e.g., severe uncorrected visual impairment or orthopedic injury to the dominant hand). It is one of the most widely used neuropsychological tests and has strong evidence of sensitivity to severity of TBI (Lange et al., Reference Lange, Iverson, Zakrzewski, Ethel-King and Franzen2005). Inclusion of at least two PVTs was also part of routine clinical care, consistent with consensus guidelines in the field of clinical neuropsychological assessment (Sweet et al., Reference Sweet, Heilbronner, Morgan, Larrabee, Rohling, Boone, Kirkwood, Schroeder and Suhr2021).
We excluded potential participants who had a prior neurological history with intracranial compromise (e.g., brain tumor), psychiatric dysfunction requiring hospitalization (e.g., schizophrenia), pervasive developmental disorder (e.g., autism spectrum), or inability to provide informed consent due to severe cognitive dysfunction (e.g., dementia). We assigned participants to one of three groups, based on results from multiple independent PVTs: zero failures, one failure, and two or more failures. Data collection continued until there were sufficient sample sizes in each of the PVT groups to have statistical power of .80 with alpha set at .05 to detect a difference of about two-thirds of a standard deviation between any of them. Of the complete final sample of 306 participants, 123 (40.20%) had also previously been included in another study concerning the clinical utility of a performance validity indicator for a test of memory and learning (Donders & Vos, Reference Donders and Vos2023).
Table 1 presents the characteristics of the complete final sample and by PVT group. Sex was based on birth assignment. Participants self-reported race. The complete racially minoritized subgroup included 15 persons who identified as Black, 10 as Latin/x, three as Asian, and nine as Biracial or Other. Participants with uncomplicated mild TBI had a duration to follow verbal commands ≤ 30 min (as determined by a qualified medical professional in acute care) and did not have neuroimaging evidence of an acute intracranial lesion (as determined by a board-certified neuroradiologist). Participants with complicated mild–severe TBI all had an acute intracranial lesion on neuroimaging, which could be focal (e.g., hemorrhagic contusion) and/or diffuse (e.g., axonal shearing) in nature. Of these individuals, half (n = 61, 50%) had a duration to follow verbal commands ≥ 1 day (range = 0–36).
Background characteristics of participants with traumatic brain injury
Note: PVT = Performance Validity Test. BIPOC = Black, Indigenous, or people of color. *Statistically significant main effect of groups (p < .05). Groups with non-identical subscripts are statistically significantly different from each other after Bonferroni correction (p < .017).
Procedures
Participants were seen for outpatient neuropsychological evaluations, with informed consent. All tests were administered and scored by psychometrists with Master’s degrees or postdoctoral residents under the supervision of a licensed neuropsychologist or by the neuropsychologists themselves. There was a hiatus in data collection of about 6 weeks in the spring of 2020 due to the COVID-19 pandemic.
Measures
The TMT has two paper-and-pencil components. Part A requires the examinee to connect with a line a series of numbers across a page, and do so in order (i.e., 1–2–3, etc.). Part B requires the examinee to connect with a line a series of numbers and letters across a page, and do so in alternating order (i.e., 1–A–2–B–3–C, etc.). Both components are scored in terms of times to completion, which are then converted into a T score (M = 50, SD = 10) on the basis of norms that are stratified by age and education, with higher T scores reflecting better performance. Although cut-off scores have been suggested for both part A and part B of the TMT as potential PVTs, we did not use those in this study because they are known to have poor classification accuracy in persons with genuine cognitive impairment (Peak et al., Reference Peak, Marceaux, Chicota-Carroll and Soble2024).
We used four independent performance validity measures. Two were freestanding instruments, including the Test of Memory Malingering (TOMM; Tombaugh, Reference Tombaugh1996; n = 215, 70.26%) and the Dot Counting Test (Boone et al., Reference Boone, Lu and Herzberg2002; DCT; n = 103, 33.66%). The other two were embedded measures, including Reliable Digit Span (RDS; Greiffenstein et al., Reference Greiffenstein, Baker and Gola1994; n = 246, 80.39%) and the forced-choice index from the California Verbal Learning Test – Third Edition (CVLT–3; Delis et al., Reference Delis, Kramer, Kaplan and Ober2017; n = 266, 86.93%). In the complete sample, 214 persons (69.94%) had data on three of these measures, 79 (25.82%) had data on two, and 13 (4.25%) had data on four.
We applied the following criteria for the respective independent performance validity measures, using empirically supported cut-offs that were specific to TBI severity when possible. For the TOMM, we followed the recommendations from Martin et al. (Reference Martin, Schroeder, Olsen, Maloy, Boettcher, Ernst and Okut2020), including cut-offs of < 42 on Trial 1 or < 49 correct on either Trial 2 or Retention in participants with uncomplicated mild TBI, and < 45 correct on either Trial 2 or Retention in participants with complicated mild-severe TBI. For the DCT, we used the E-score cut-off of > 19 suggested by Boone et al. (Reference Boone, Lu and Herzberg2002) for participants with complicated mild-severe TBI, and the more stringent cut-off of > 13.8. suggested by McCaul et al. (Reference McCaul, Boone, Ermshar, Cottingham, Victor, Ziegler, Zeller and Wright2018) for persons with uncomplicated mild TBI. For RDS, we used the original cut-off of < 8 suggested by Greiffenstein et al. (Reference Greiffenstein, Baker and Gola1994) in persons with uncomplicated mild TBI, but an adjusted cut-off of < 7 in persons with complicated mild–severe TBI, given concerns about unacceptably high false positives of that original cut-off with greater injury severity (Suhr & Barrash, Reference Suhr, Barrash and Larrabee2007). Finally, for CVLT–3 Forced Choice, we applied the universal cut-off of < 15 suggested by Donders & Vos (Reference Donders and Vos2023) because there were no empirically cross-validated reasons to use a different cut-off for various injury severities on this particular instrument. In the complete sample, failure rates on each of these indicators (based on those who completed them) were as follows: 38/215 for TOMM (17.67%), 26/103 (25.24%) for DCT, 42/246 (17.07%) for RDS, and 20/266 (7.52%) for CVLT–3 Forced Choice. Of the 41 participants who failed multiple PVTs, 35 failed two, and six failed three.
Statistical analyses
We compared the three PVT groups with chi-square tests for discrete variables. For continuous variables, we used analysis of variance when the variables were normally distributed and Kruskall–Willis test when they were not. For post hoc comparisons, we applied a Bonferroni correction to alpha to guard against spurious findings (i.e., minimum required level of .017 [.05/3]).
Results
Inspection of Table 1 reflects that there were no statistically significant differences between the three PVT groups in terms of age, race, level of education, or injury circumstances (p > .10 for all these variables). However, there were statistically significant main group differences in terms of sex (χ2 [2] = 13.57, p < .001, V = .21), proportion of persons with uncomplicated mild injuries (χ2 [2] = 31.54, p < .001, V = .32), time since injury (H [2] = 6.88, p < .03, η2 = .02), and presence of financial compensation-seeking (χ2 [2] = 30.55, p < .001, V = .31). Follow-up contrasts demonstrated that in most cases, these findings were driven by the group who failed 0 PVTs being different from those who failed either 1 or ≥ 2, with the latter two groups not differing meaningfully from each other. The only exception was time since injury, for which only the difference between the groups who failed, respectively, 0 and ≥ 2 PVTs, met the a priori criterion for statistical significance. Compared to the other two groups, those who passed all PVTs had a greater proportion of men, a lower proportion of uncomplicated mild injuries, and a lower proportion of participants who were seeking financial compensation. The time since injury was also relatively shorter for the group who passed all PVTs than for the group who failed ≥ 2.
Figure 1 presents the findings from the TMT for the complete sample and by PVT group. The main effect of groups was statistically significant for both part A (F [2, 303] = 23.47, p < .001, η2 = .13) and part B (F [2, 303] = 26.14, p < .001, η2 = .15). Post hoc contrasts demonstrated that in each case, the group who passed all PVTs performed better than both of the other two groups, who did not differ meaningfully from each other.
Trail Making Test performance (in T scores) of participants with traumatic brain injury. Note: PVT = performance validity test. Error bars reflect the standard error of the mean.
We also evaluated to what degree the findings might be related to how many PVTs were administered in each case. Because this number was not fixed but driven by examiner choice (e.g., giving more PVTs if one of two was failed, or no more if two had already been passed), we used the proportion of PVTs failed for this purpose. This variable correlated moderately with performance on both part A (ρ = –0.37, p < .001) and part B (ρ = –0.39, p < .001) of the TMT.
Finally, as a qualitative analysis, we considered the possibility that at least some of the cases in the complicated mild–severe TBI group could have been false positives. The total PVT failure rate in this group was 9.85% (13/132). Of this subgroup, eight participants (6.06%) failed only one PVT and five (3.79%) failed two. Specific failure rates were 7/118 (5.93%) for CVLT–3 Forced Choice, 6/109 (5.51%) for RDS, 5/88 (5.68%) for TOMM, and 2/44 (4.55%) for DCT. However, when these persons failed more than one PVT, at least one was always (5/5; 100%) a stand-alone instrument. In contrast, when they failed only one PVT, it was almost always (7/8; 87.50%) an embedded one.
Discussion
The purpose of this investigation was to evaluate the relationships between zero, one, and multiple PVT failures on cognitive test performance in persons with TBI with a broad range of injury severity and who were not exclusively engaged in financial compensation-seeking. The results indicated that failure of even one PVT was associated with poorer performance on both parts of the TMT, to a moderate degree. These findings extend the findings of Proto et al. (Reference Proto, Pastorek, Miller, Romesser, Sim and Linck2014), who found a similar effect for part B but not part A of the TMT. The latter may have been related in part to the fact that their study only included persons with mild TBI and also may have lacked sufficient statistical power to reject the null hypothesis due to small sample sizes in some of their subgroups. Our findings are consistent with those of Denning (Reference Denning2023), who found in a mixed clinical sample that those who failed one PVT had cognitive test scores that were worse than those who failed zero, and more similar to those who failed two. However, unlike Kim et al. (Reference Kim, Currao, Brown, Milberg and Fortier2024), we did not find a notable difference in cognitive test performance between those who failed only one PVT compared to those who failed two or more. This may be due to the fact that theirs was a mixed psychiatric sample, not all of whom had TBI. Our findings are also at odds with those of Rohling et al. (Reference Rohling, Larrabee and Davis2025), who reported that assigning those who failed only one PVT to a “valid performance” group did not change diagnostic accuracy very much. Differences in sample characteristics, most notably the fact that they included predominantly cases who had been referred for assessment of psychiatric disorders in a civil forensic context, likely accounted for this difference, considering that several prior studies have reported an association between subjective psychiatric dysfunction and PVT failure that is distinct from the influence of medical variables (Erdodi et al., Reference Erdodi, Seke, Shahein, Tyson, Sagar and Roth2017; Phillips et al., Reference Phillips, Whitman Kent, Ismail, Piszczor, Galindo, Shapiro, Gonzalez, Cerny and Soble2025). At the same time, our finding that failure of a greater proportion of PVTs was associated with worse performance on the TMT was consistent with other recent research that has demonstrated a lowering of neuropsychological test scores with increases in PVT failures (Kukla et al., Reference Kukla, Finley, Kent, Newkirk, Payne, Fegter and Soble2025).
We also found that, compared to those who passed all PVTs, those who failed one or more of them were more likely to (a) be female, (b) have sustained an uncomplicated mild TBI, (c) be involved in financial compensation-seeking and (d) be seen at a longer time frame since injury (with the latter difference reaching the a priori criterion for statistical significance only for those who failed ≥ 2 PVTs). The combination of financial compensation-seeking with persistent subjective complaints in the context of uncomplicated mild TBI is a commonly known threat to validity of neuropsychological test results (Boone et al., Reference Boone, Litvin, Victor and Boone2021; Schroeder et al., Reference Schroeder, Boone, Larrabee and Boone2021). However, sex differences have not been widely reported in PVT research, even though women are known to be more likely than men to report persistent symptoms after uncomplicated mild TBI (Dennis et al., Reference Dennis, Yengo-Kahn, Kirby, Solomon, Cox and Zuckerman2019; Rabinowitz et al., Reference Rabinowitz, Li, McCauley, Wilde, Barnes, Hanten, Mendez, McCarthy and Levin2015). When PVT failure rates have been specifically investigated, the results have not been consistent. For example, Sullivan-Baca et al. (Reference Sullivan-Baca, Naylon, Zartman, Ardolf and Westhafer2020) reported that in a mixed clinical sample, men performed worse on some, but not all, formal performance validity measures than women. In contrast, Lesica et al. (Reference Lesica, Luu and Lawrence2024) did not find a sex difference in PVT failure in an epilepsy sample. Donders and Vos (Reference Donders and Vos2023) did find that women with TBI were more likely than men to fail multiple PVTs, but it should be noted that their study included about 40% of the same participants as the current one. At this time, the meaning and implication of the current finding of a sex difference in the likelihood of PVT failure are simply unclear, and it will require replication in an independent sample.
It is important to appreciate that the failure of a single PVT does not necessarily reflect that all the data are invalid, let alone that the examinee was most likely malingering. Such a single failure is common across a range of clinical conditions, even in the absence of incentives to underperform, for reasons that may not always be clear (McWhirter et al., Reference McWhirter, Ritchie, Stone and Carson2020). In the current study, a single PVT failure occurred in slightly less than 10% of the participants who had acute intracranial lesions on neuroimaging. This was consistent with base rates when cut scores are set to maintain a specificity level of .90, as is customary in the literature (Schroeder et al., Reference Schroeder, Boone, Larrabee and Boone2021). Our findings suggest that especially when such a single failure is on an embedded indicator, it is important to avoid a knee-jerk conclusion that the data must be invalid with patients who have unequivocal cerebral compromise.
The extent to which PVTs are failed is also an important consideration. For example, scores on a forced-choice test that are at or below chance levels are much more compelling evidence of non-credible responding than a single borderline fail (Erdodi, Reference Erdodi2023). In this context, examinee characteristics must also be taken into account. For example, of the four participants in the current investigation who scored at or below chance level on all trials three of the TOMM, all had uncomplicated mild injuries and three of them were pursuing personal injury litigation. There has also been debate about whether PVT failure might represent a “cry for help,” with theoretical arguments proposing (Young, Reference Young2019) as well as opposing (Dandachi-Fitzgerald et al., Reference Dandachi-FitzGerald, Merckelbach and Merten2022) that concept, but no firm empirical data to support it.
Another consideration is what to do in clinical practice when a single PVT is failed. The easiest option is to administer additional PVTs, which might boost diagnostic confidence through strength in numbers if multiple criteria are failed (Soble et al., Reference Soble, Alverson, Phillips, Critchfield, Fullen, O’Rourke, Messerly, Highsmith, Bailey, Webber and Marceaux2020). However, there is also evidence that providing feedback on non-credible results may lead to improvement on subsequent neuropsychological test performance (Roor et al., Reference Roor, Knoop, Dandachi-FitzGerald, Peters, Bleijenberg and Ponds2020). Carone et al. (Reference Carone, Iverson and Bush2010) were one of the first to provide suggestions for providing feedback about invalid neuropsychological test performance. Recently, Schroeder & Martin, Reference Schroeder and Martin2025) described a more comprehensive model for this purpose, which they defined as a “firm-beneficent approach,” to be applied after conclusion of the evaluation. This involves objectively communicating that the data are not valid, without going into speculation or discussion why, but instead focusing on aspects of the examinee’s life that could realistically see benefit if some form of intervention were to occur. Of course, these kinds of strategies should be applied only in clinical cases and not in medicolegal or other forensic ones where the examinee is not the client.
We acknowledge some limitations of this investigation. First of all, our referred convenience sample was almost 90% White, which was more than regional demographics would lead one to expect (i.e., about three-quarters). Whether this disparity is related to differences in healthcare access, biases in referral practices, or something else is unknown. Nevertheless, the findings cannot be generalized to persons who are Black, Indigenous, or of color, and replication in a more diverse sample is desirable. This is especially important because of recent concerns about elevated proportions of PVT failures in Black as compared to White participants in mixed samples (Denning & Horner, Reference Denning and Horner2024; Hromas et al., Reference Hromas, Rolin and Davis2022). Another limitation was the fact that not all participants received the same or the same number of performance validity tests. This relative weakness was likely outweighed by the fact that whenever possible, we used cut-offs that were specific to the clinical disorder under consideration, as well as its severity, consistent with the recommendations by Sweet et al. (Reference Sweet, Heilbronner, Morgan, Larrabee, Rohling, Boone, Kirkwood, Schroeder and Suhr2021). However, future research should examine more closely the influence of various combinations of stand-alone and embedded PVTs used, because there is some research to suggest that selection of different combinations of instruments can result in discrepant findings (Tierney et al., Reference Tierney, Taiwo, Choudhury, Lippa, Troyanskaya, Pastorek, Miller, Romesser and Sim2023). It should also be noted that we used only one cognitive test (TMT) as the dependent variable because it had been almost always administered and had a known body of literature supporting its validity in patients with TBI. However, future research should explore the correlates of various numbers of PVT failures with results on other neuropsychological tests.
With these considerations in mind, we conclude that in the context of clinical neuropsychological evaluations of persons with TBI, even one PVT failure should raise potential concern about the conventional interpretability of the cognitive test findings. In addition, in future research studies, it is advisable to investigate such individuals as a separate group as opposed to excluding them or mixing them with those who fail either zero or multiple PVTs. A goal for such future research is the evaluation of the associations of zero, one, and multiple PVT failures in other common conditions such as attention-deficit/hyperactivity disorder, epilepsy, and chronic fatigue syndrome.
Acknowledgments
The authors thank Tracy Maher and Ashlee Ramos for their assistance with database management.
Funding statement
There was no funding for this research.
Competing interests
The authors report no conflicts of interest.
