Introduction
During brain injury rehabilitation, patients re-learn old skills and learn new skills with the ultimate goal of optimizing social participation and well-being (Wade, Reference Wade2005). Although learning is essential for achieving rehabilitation gains, there is a lack of research-based guidelines on how best to assess learning potential in rehabilitation. A promising assessment method to evaluate learning potential that has been used in patients with acquired brain injury (ABI) and several other populations (e.g., schizophrenia, dementia), is dynamic testing (Calero & Navarro, Reference Calero and Navarro2004; Fernández-Ballesteros et al., Reference Fernández-Ballesteros, Botella, Zamarrón, Molina, Cabras, Schettini and Tárraga2012; Rempfer, Brown, & Hamera, Reference Rempfer, Brown and Hamera2011; Uprichard, Kupship, Pine, & Fletcher, 2009; Watzke, Brieger, Kuss, Schöttke, & Wiedl, Reference Watzke, Brieger, Kuss, Schöttke and Wiedl2008).
Dynamic testing is an umbrella term for procedures that examine the effects of brief training on test performance where pre- to post-test change reflects a patient’s learning potential (Grigorenko & Sternberg, Reference Grigorenko and Sternberg1998). It is proposed that dynamic tests provide unique information about a person’s abilities beyond the information that is provided by conventional tests (Caffrey, Fuchs, & Fuchs, Reference Caffrey, Fuchs and Fuchs2008; Grigorenko, 2009). Conventional tests provide information about cognitive deficits that may hamper or facilitate learning, whereas dynamic tests more specifically evaluate patients’ potential to learn and improve cognitive performance. In patients with cognitive impairments due to ABI or a psychiatric or neurodegenerative disorder, several one-session dynamic tests have been applied that target cognitive abilities. The most frequently used is the dynamic Wisconsin Card Sorting Test (dWCST) (Kurtz, Jeffrey, & Rose, Reference Kurtz, Jeffrey and Rose2010; Rempfer et al., Reference Rempfer, Brown and Hamera2011; Uprichard et al., Reference Uprichard, Kupshik, Pine and Fletcher2009; Vaskinn et al., Reference Vaskinn, Sundet, Friis, Ueland, Simonsen, Birkenaes and Andreassen2009; Watzke, Brieger, & Wiedl, Reference Watzke, Brieger and Wiedl2009). The WCST is a renowned measure of executive functioning that can be used to evaluate a patient’s ability to form, maintain, and shift cognitive sets (Heaton, Reference Heaton1981). In contrast to the conventional WCST, the dWCST includes a brief training phase in which additional feedback and instructions are given about the sorting principle after the initial testing phase. The training is followed by a second test phase (i.e., test-train-test design). The degree of change in test performance in response to the training is proposed to represent a patient’s learning potential or capacity to benefit from training.
The dWCST has already shown some evidence of validity in several longitudinal studies that included patients with ABI (Uprichard et al., Reference Uprichard, Kupshik, Pine and Fletcher2009) and psychiatric disorders (Kurtz & Wexler, Reference Kurtz and Wexler2006; Rempfer et al., Reference Rempfer, Brown and Hamera2011; Sergi, Kern, Mintz, & Green, Reference Sergi, Kern, Mintz and Green2005; Watzke et al., Reference Watzke, Brieger, Kuss, Schöttke and Wiedl2008, Reference Watzke, Brieger and Wiedl2009). These studies found that dWCST learning potential was a significant predictor of rehabilitation outcome and provided unique information beyond the conventional WCST in predicting outcome (Sergi et al., Reference Sergi, Kern, Mintz and Green2005). Although promising, there are some issues that affect interpretation of these findings and hence the potential utility of the dWCST in clinical practice. First, adding a training phase may have consequences for the validity of the post-test. For instance, Wiedl, Schöttke, Green, and Nuechterlein (Reference Wiedl, Schöttke, Green and Nuechterlein2004) administered the dWCST to patients with schizophrenia and reported that the training phase alters the cognitive abilities assessed by the post-test. Not surprisingly, both the pre- and post-test scores were associated with tests of executive functioning. The post-test score was, however, additionally associated with verbal memory tests. This suggest that the dWCST pre- and post-test scores reflect different cognitive constructs. These findings have not yet been confirmed by other researchers.
A second issue is that the role of practice effects on pre- to post-test improvement has not yet been examined. Improvements in test performance due to repeated exposure to the same test materials reflect practice effects, whereas pre- to post-test gains due to the brief training reflects learning potential. Just like the conventional WCST, the dWCST has a single solution and is therefore prone to practice effects (Lezak, Howieson, & Loring, Reference Lezak, Howieson and Loring2004). Performance basically depends on discovery of the sort and shift principle. After the sorting principle is discovered or explained, patients are likely to improve their test performance during a second administration of the test (Lezak et al., Reference Lezak, Howieson and Loring2004).
Further psychometric validation of the dWCST would provide researchers and clinicians with a better understanding of what the dWCST actually measures and how to interpret test results. Such research may also guide any necessary refinement of the dWCST for clinical use. Hence, the purpose of this study was to investigate the validity of the dWCST in patients with ABI by: (1) Examining the feasibility of administration and general psychometric properties of the dWCST in terms of score distributions, administration time, floor and ceiling effects, intercorrelations between different learning potential indices (i.e., post-test score, gain score), and associations between the post-test score and gain score and other neuropsychological tests. (2) Determining whether pre- to post-test improvement reflects learning potential or practice effects. (3) Determining whether the dWCST can discriminate between patients and controls by comparing dWCST performance of patients with ABI and healthy controls matched on age, gender, and education.
Hypotheses
We hypothesized that the distribution of the dWCST learning potential indices would demonstrate an absence of floor or ceiling effects and that the learning potential indices would be highly intercorrelated. We expected that the dWCST scores would be significantly associated with other measures of executive functioning at rehabilitation admission. Also, we hypothesized that patients and controls, would demonstrate a significant change in pre- to post-test scores for the dWCST, whereas patients who received a repeated assessment of the WCST (rWCST) would demonstrate no significant pre- to post-test change. Compared to the rWCST group, patients in the dWCST group would demonstrate similar pre-test scores, but significantly higher gain and post-test scores. Furthermore, we expected that, compared to healthy controls, patients would demonstrate significantly lower pre-test, post-test, and gain scores for the dWCST.
Methods
Participants
Patients were recruited from five rehabilitation centers in The Netherlands. For all patients (i.e., the dWCST and rWCST group), the same in- and exclusion criteria were used. Inclusion criteria were (1) diagnosis of traumatic or non-traumatic ABI based on medical records; (2) ≥18 years of age; (3) sufficient command of the Dutch language based on clinical judgment; and (4) completion of the dWCST at admission to rehabilitation. Exclusion criteria were: (1) severe aphasia based on a Dutch Aphasia Foundation (Stichting Afasie Nederland, SAN; Deelman, Koning-Haanstra, Liebrand, & van de Burg, Reference Deelman, Koning-Haanstra, Liebrand and van de Burg1987) scale score less than 4 or clinical judgment; (2) premorbid psychiatric disorder and/or substance abuse for which hospital admission was necessary; (3) minimally conscious state or post-traumatic amnesia at the time of assessment; (4) degenerative or progressive brain disease; (5) active participation in another study to avoid participation burden; (6) no informed consent; and (7) completion of the conventional WCST during a neuropsychological assessment.
The control group consisted of volunteers who were recruited through personal contacts. Inclusion criteria were: (1) ≥18 years of age; (2) sufficient command of the Dutch language based on clinical judgment; and (3) written informed consent. Exclusion criteria were: (1) history of a neurological event or psychiatric disorder; and (2) cognitive impairments, indicated by a Montreal Cognitive Assessment (MoCA; Nasreddine et al., Reference Nasreddine, Phillips, Bedirian, Charbonneau, Whitehead, Collin and Chertkow2005) score below 24. The medical ethics committee of the University Medical Center Utrecht and the five participating rehabilitation centers approved the study protocol. All data were obtained in compliance with regulations of the participating rehabilitation centers. All participants gave informed consent.
Feasibility of administration and general psychometric properties
Data from patients who were participating in a longitudinal cohort study examining factors influencing outcome of ABI rehabilitation (dWCST group) were examined. These patients were recruited from November 2012 to December 2013 from inpatient clinics of five rehabilitation centers in The Netherlands: Adelante Zorggroep, Hoensbroek; De Hoogstraat Rehabilitation, Utrecht; Reade Rehabilitation Center, Amsterdam; Rijndam Rehabilitation, Rotterdam; and Rehabilitation Center Tolbrug, ‘s-Hertogenbosch.
Learning potential versus practice effects
A subgroup of patients from the dWCST group was selected, namely all patients who were recruited at De Hoogstraat Rehabilitation (dWCST subgroup). We only included patients from De Hoogstraat Rehabilitation to ensure uniformity of administration procedures from staff at the same clinic, and also achieve a large sample size for the main experimental condition. dWCST performance of these patients was compared to another group of patients (rWCST group). Patients in the rWCST group were recruited from July 2013 to December 2013 from the inpatient clinic of De Hoogstraat Rehabilitation. These patients were not eligible to participate in the dWCST group because they were admitted to one of the inpatient clinics in which the recruitment period had already ended.
Patients with ABI Versus Healthy Controls
dWCST performance of the dWCST subgroup was compared to a group of healthy controls (control group) matched on age, gender, and education. The control group was recruited from October 2013 to December 2013 as a reference sample to compare performance of patients and healthy controls.
Measures
Wisconsin Card Sorting Test (WCST)
The participant was given two decks of 64 cards containing figures which differ in color (yellow, red, green, and blue), shape (star, square, circle, and cross), and number (one, two, three, and four). The participant was instructed to place each card below one of four stimulus cards. The participant had to deduce the sorting principle from the experimenter’s feedback (“right”, “wrong”) (Heaton, Reference Heaton1981). The administration time of the conventional WCST is approximately 20 to 30 min.
Cognitive functioning
A nationally recommended core battery of cognitive tests (Nederlands Instituut van Psychologen, 2010) was administered by a psychological assistant. Verbal memory was assessed with the Rey Auditory Verbal Learning Test (RAVLT; Rey, Reference Rey1958). Attention was evaluated with the Trail Making Test parts A and B (TMT; Reitan, Reference Reitan1956). Part A is a measure of psychomotor speed and Part B is a measure of divided attention and executive functioning. Executive functioning was also evaluated with the letter fluency test (LFT; Schmand, Groenink, & van den Dungen, Reference Schmand, Groenink and Van den Dungen2008). Language performance was evaluated with the Boston Naming Test (Kaplan, Goodglass, & Weintraub, Reference Kaplan, Goodglass and Weintraub1983) short form and the Category Fluency Test (CFT; Van der Elst, van Boxtel, Van Breukelen, & Jolles, Reference Van der Elst, Van Boxtel, Van Breukelen and Jolles2006). The Star Cancellation Test (SCT; Wilson, Cockburn, & Halligan, Reference Wilson, Cockburn and Halligan1987) was used to assess patients’ visual perception and to screen for unilateral spatial neglect. In addition, premorbid intelligence was estimated with the Dutch version of the National Adult Reading Test (NART; Nelson, Reference Nelson1982), the ‘Nederlandse Leestest voor Volwassenen’ (NLV; Schmand, Lindeboom, & van Harskamp, Reference Schmand, Lindeboom and Van Harskamp1992). These tests are described in more detail elsewhere (Lezak et al., Reference Lezak, Howieson and Loring2004).
Procedures
For all patients, the treating rehabilitation physician confirmed the eligibility criteria and obtained informed consent. After patients provided written informed consent, demographic and disease characteristics were obtained from their medical records. In the dWCST group (and thus the dWCST subgroup), the dWCST was administered within approximately 2 weeks of enrollment. Within approximately the same week, a cognitive screening was conducted by a psychological assistant as part of routine assessment procedures. In the rWCST group, the rWCST was administered within 1 week after informed consent was obtained. In the healthy control group, a home visit was scheduled. During the home visit, participants were checked for eligibility. If eligible, the dWCST was administered.
Administration procedures of the dWCST and rWCST
The dWCST and rWCST were administered in a quiet room by a trained clinician or trained neuropsychology student. Before commencement, patients and healthy controls were asked to give a detailed description of the four stimulus cards (one red triangle, two green stars, three yellow crosses, four blue circles) to assess potential visual (e.g., achromatopsia) or cognitive (e.g., visual form agnosia) problems that could influence test performance. The test was not completed when such problems were considered to significantly influence performance.
During administration of the dWCST, a pre-test – train – post-test paradigm was used. The dWCST was administered according to the test protocol of Wiedl and Wienöbst (Reference Wiedl and Wienöbst1999) which was translated from German into Dutch. The dWCST pre- and post-test were administered according to standard WCST administration procedures (Heaton, Reference Heaton1981). The first deck of 64 cards was used for the pre-test as well as for the training, and the second deck of 64 cards was used for the post-test. The training phase consisted of three major elements: (1) explanation of the sorting rules (e.g., “There are three possible ways to match the cards: you can match the card by color, by number of the objects, or by shape”); (2) during the card sorting, the patient was told why a response was right or wrong after each card sort (e.g., “This was wrong, we don’t sort for color”); and (3) how many consecutive correct responses the patient needed for the rule to change (e.g., “After ten consecutive correct sorts, the rule will change. You will then sort for color or number”).
During administration of the rWCST, a test-test-test procedure was adopted. The deck of 64 cards was administered three times using standard WCST procedures (Heaton, Reference Heaton1981). The first deck of 64 cards was used for the first two administrations (i.e., pre-test and test without training) and the second deck of 64 cards was used for the third administration (i.e., post-test).
For both the dWCST and rWCST, the total number of correct responses was recorded for the pre-test and post-test. For both test phases, the score ranged from 0 to 64 with higher scores indicating better performance. The time needed to administer the dWCST and rWCST was recorded.
Statistical analysis
dWCST indices
For all patients and controls, the pre-test score and three different learning potential indices were calculated. The computational methods are described below. The learning potential indices were selected based on their use in prior studies to index learning potential (Fiszdon & Johannesen, Reference Fiszdon and Johannesen2010; Weingartz, Wiedl, & Watzke, Reference Weingartz, Wiedl and Watzke2008) and their feasibility for use in clinical settings. For all indices, higher scores indicate better performance.
Pre-test score
Total correct after the pre-test.
Post-test score
Total correct after the training phase.
Gain score
Ratio calculated by dividing actual performance change (i.e., difference score) by potential performance change. The following formula was used: (post-test total correct – pre-test total correct) / (64 – pre-test total correct) in which 64 is represents the maximum total correct that can be achieved on the post-test (Weingartz et al., Reference Weingartz, Wiedl and Watzke2008).
Groups
Previously established cutoff values were used based on the reliable change approach by Wiedl et al. (Reference Wiedl and Wienöbst1999). Patients were categorized as “high achiever” (pre- and post-test ≥43 correct); “strong learner” (pre- to post-test improvement ≥15 points); “poor learner” (pre- to post-test improvement <15 points); or “decliner” (pre- to post-test decline ≥15 points). The Reliable Change Generator (Devilly, Reference Devilly2005) was used to determine whether these cutoff values were reliable (Jacobson & Truax, Reference Jacobson and Truax1991) in our sample of patients with ABI. Based on the rWCST pre-test standard deviation (SD=12.1) and test-retest reliability (rpre-test, post-test=0.81) a reliable change index of 15 was calculated. Thus, reliable change was considered present when the pre- and post-test differed by ≥15 points which is in agreement with the previously used cutoff values.
Feasibility of administration and general psychometric properties
The distribution of numerical learning potential scores was evaluated using Kolmogorov Smirnov, skewness, and kurtosis tests. In case of a non-normal distribution, transformations were performed to improve normality. If these did not improve normality, non-parametric statistics were used. Feasibility of the dWCST was investigated by calculating the time needed to administer the dWCST. Floor and ceiling effects for the dWCST pre-test and numerical learning potential indices were examined. Floor or ceiling effects were considered present if more than 15% of all patients obtained the lowest (post-test score=0; gain score=−64) or highest (post-test score=64; gain score=1) possible score (Terwee et al., Reference Terwee, Bot, De Boer, Van der Windt, Knol, Dekker and De Vet2007). Pearson or Spearman correlations were used to examine intercorrelations between the post-test score and gain score and their association with other neuropsychological tests. Associations between the dWCST pre- and post-test scores and between the pre-test score and other neuropsychological tests were also examined for comparison purposes. Between-group differences in neuropsychological test performance was evaluated using One-way between-groups analyses of variance and independent samples t tests or Kruskal-Wallis and Mann-Whitney U Tests.
Learning potential versus practice effects
First, independent samples t tests or Mann-Whitney U Tests, and Chi square tests were used to examine differences between the dWCST subgroup and the rWCST group regarding age, gender, education, and functional independence. Between-group differences in dWCST and rWCST scores (pre-test score, post-test score, gain score) then were examined using independent samples t tests or Mann-Whitney U Tests. For both groups, paired samples t tests or Wilcoxon Signed-Rank tests were used to determine whether there was a significant change in pre- to post-test performance. Also, between-group differences in the number of poor learners and strong learners were examined using Chi square tests. Decliners and high achievers were not included in this analysis. A decline in performance indicates neither learning potential nor a practice effect, and high achievers do not require a training phase to perform adequately.
Patients with ABI versus healthy controls
First, independent samples t tests or Mann-Whitney U Tests, and χ2 were used to examine whether the dWCST subgroup and the control group were adequately matched for age, gender, and education. Second, differences in dWCST scores (pre-test score, post-test score, and gain score) between the dWCST subgroup and the control group were evaluated using independent samples t tests or Mann-Whitney U Tests. For both groups, paired samples t tests or Wilcoxon Signed-Rank tests were used to determine whether there was a significant change in pre- to post-test performance. Furthermore, between-group differences in the number of patients with low performance (decliners and poor learners) versus adequate performance (strong learners and high achievers) between the dWCST subgroup and the control group were examined using χ2 tests.
For all analyses, Pearson or Spearman correlation coefficients between 0.30 and 0.49 were considered to be moderate and correlations exceeding 0.50 were interpreted as large (Cohen, Reference Cohen1983). Due to the large number of associations examined, alpha was set at 0.01. Data were analyzed using SPSS version 21.0.
Results
Participants
For the dWCST group, 125 patients with ABI were recruited from the five rehabilitation centers (response rate 78.1%). Of these, 21 patients were excluded for the following reasons: the conventional WCST was already administered during the routine neuropsychological assessment (n=3); refusal to complete the dWCST (n=2); withdrew from study (n=4); dWCST was not administered due to early discharge (n=4); dWCST was discontinued by the examiner because the test was perceived as too demanding for the patient (n=7). A total of 104 patients were included in the dWCST group. Of these, 63 patients were recruited at De Hoogstraat Rehabilitation and thus were selected for the dWCST subgroup. For the rWCST group, 32 patients were recruited. Of these, the rWCST was discontinued by three patients during or after the pre-test, and for one patient the rWCST was discontinued by the examiner because the test was perceived as too demanding for the patient, leaving a total of 28 patients. Based on clinical judgment, all 28 patients in the rWCST group and all 104 patients in the dWCST group, showed sufficient visual and cognitive abilities to complete the test. For the control group, 30 healthy controls were recruited. The MoCA scores of the control participants ranged from 24 to 30 (mean, 27.6±1.73), indicating adequate cognitive functioning for all healthy controls.
There were no differences between patients in the dWCST subgroup and the rWCST group regarding age (U=676.0; z=−1.77; p=.08; r=−.19), gender (χ2=1.90; p=.17; phi=−.14), education (χ2=.68; p=.41; phi=−.09), and functional independence (χ2=2.16; p=.34; Cramer’s V=.16). The dWCST subgroup was adequately matched with the control group for gender (χ2=.27; p=.61; phi=.05), age (U=897.0; z=−.40; p=.69; r=−.04), and education (χ2=.63; p=.43; phi=.08). On average, the dWCST was administered significantly later than the rWCST (median, 27 vs. 11 days after admission, respectively; U=435.5; z=−3.66; p<.001; r=.39). Table 1 shows the characteristics for all participants groups.
Characteristics of the participant groups
a High education ≥ higher vocational education.
b n=95.
c n=27.
Score Distributions and Administration Time
All learning potential indices showed a left-skewed, non-normal distribution (skewness=−.98 to −.99). Data transformations did not improve normality. Therefore, non-parametric statistics were used. dWCST administration time varied from 10 to 55 minutes (median=30.0 min). In total, 86.5% of patients completed the dWCST in 40 minutes or less. In Table 2, an overview of the dWCST scores is presented. When classifying patients into groups, 34.9% were high achievers, 42.9% were strong learners, and 22.2% were poor learners. There were no decliners. Figure 1 shows the median dWCST scores for the pre-test, training and post-test for the three learner groups.
Median dynamic Wisconsin Card Sorting Test scores for the patient subgroup divided into poor learners, strong learners, and high achievers.
Descriptives, floor- and ceiling effects, and intercorrelations of dynamic Wisconsin Card Sorting Test scores of the dWCST group
*p≤.01
a Gain score=(post-test total correct – pre-test total correct)/(64 – pre-test total correct); dWCST=dynamic Wisconsin Card Sorting Test; IQR=interquartile range
Floor and Ceiling Effects and Intercorrelations
There were no floor or ceiling effects for the dWCST post-test score and gain score. In total, 11.1% of patients had ≥58 items correct and 1.0% had all 64 post-test items correct. The post-test score showed a large, significant association with the gain score (r=.62; p<.001).
Associations with Neuropsychological Tests
In Table 3, the correlation coefficients are shown for the associations between the dWCST scores and the neuropsychological tests. The pre-test score correlations were presented for comparison. The pre-test score was not significantly associated with any of the cognitive tests. The post-test score and gain score showed significant, moderate associations with long-term memory (rpost-test=37; p<.001; rgain score=.31; p=.001) and recognition memory (rpost-test=.37; p<.001; rgain score=.34; p=.001). Furthermore, the post-test score showed a moderate significant association with short-term memory (r=.32; p=.001) and a small, significant association with language (r=.26; p=.01; r=.26; p=.009). The dWCST pre- and post-test score showed a large, significant association (r=.50; p<.001).
Spearman correlations between the dynamic Wisconsin Card Sorting Test pre-test, post-test score, and gain score, and Neuropsychological Tests of the dWCST group
**p≤0.01; * p≤0.05.
a Some patients could not complete all tests due to for instance reading or visual problems (N range 97–104);
b R-AVLT=Rey Auditory Verbal Learning Test; WCST-64=Wisconsin Card Sorting Test-64 (Kongs, Thompson, Iverson, & Heaton, Reference Kongs, Thompson, Iverson and Heaton2000).
c Higher scores reflect worse performance.
Between-group differences in neuropsychological test performance are presented in Table 4. No significant between-group differences in neuropsychological test performance were observed at a 0.01 level. For two tests, there was a trend toward significance. At a 0.05 level of significance, language scores were significantly lower for poor learners compared to high achievers (U=320.0; z=−2.5; p=.01; r=−0.31), and poor and strong learners obtained lower scores for the measure of attention and executive functioning compared to high achievers (Upoor=256.5; z=−2.4; p=0.02; r=−0.31; Ustrong=437.5; z=−2.0; p=.04; r=−0.24) (Table 4).
Learning Potential Versus Practice Effects
Test scores of the dWCST subgroup were compared to test scores of the rWCST group. The median scores for both patient groups are presented in Table 5. There were no significant between-group differences in pre-test performance (U=882.0; z=.000; p=1.00). Compared to patients who were administered the dWCST, the rWCST group had lower scores for the post-test (U=475.5; z=−3.50; p<.001; r=.37), and gain score (U=416.0; z=−4.01; p<.001; r=−.42). Furthermore, no significant change from pre- to post-test scores was found in the rWCST group (Z=−1.3; p=.182; r=−.17). In the dWCST group, the pre- to post-test improvement was significant (Z=−6.10; p<.001; r=−.54). There were significantly more strong learners in the dWCST group compared to the rWCST group (χ2=17.3; p<.001; phi=−.55; 42.9% vs. 3.6%, respectively). In Figure 2, the median dWCST scores are displayed for the dWCST and rWCST group as well as the control group.
Median score for dynamic Wisconsin Card Sorting Test subgroup, repeated Wisconsin Card Sorting Test group, and the control group (dynamic Wisconsin Card Sorting Test).
Between-group differences in neuropsychological test performance
**p≤.01; *p≤.05.
a Some patients could not complete all tests due to for instance reading or visual problems.
b R-AVLT=Rey Auditory Verbal Learning Test.
c Higher scores reflect worse performance.
Patients with ABI Versus Healthy Controls
Test scores of the dWCST subgroup were compared to the control group. The median dWCST scores for both groups are presented in Table 5. Compared to the dWCST subgroup, the control group had significantly higher scores for the pre-test (U=.599.0; z=−2.85; p=.004; r=−.30), and post-test (U=561.5; z=−3.16; p=.002; r=−.33) (see Figure 2). There were no significant between-group differences for the gain score (U=887.5; z=−.47; p=.636; r=−.05). Just like in the dWCST subgroup, the control group showed significant pre- to post-test improvement (Z=−3.9; p<.001; r=−.50). Furthermore, the difference in the number of poor learners between the dWCST subgroup and the control group was non-significant, but showed a trend toward significance (χ2=5.22; p=.02; phi=.24; 65.1% vs. 40.0%, respectively).
Between-group comparisons of dynamic and repeated Wisconsin Card Sorting Test scores
**p≤0.01, *p≤0.05.
a dWCST=dynamic Wisconsin Card Sorting Test (test-train-test); rWCST=repeated Wisconsin Card Sorting Test (test-test-test).
b Learning potential group (poor learners vs. strong learners)×participant group (dWCST subgroup, rWCST group).
c Learning potential group (poor learners vs. strong learners/high achievers)×participant group (dWCST subgroup, control group).
Discussion
This study investigated the validity of the dWCST for assessing learning potential in patients with ABI. The results showed that adding training to the WCST takes, on average, only approximately 10 min additional administration time. The dWCST showed no floor or ceiling effects, and a large intercorrelation between the two numerical learning potential indices. The pre-test score showed no significant associations with other neuropsychological tests, whereas the numerical learning potential indices were significantly associated with language and/or memory. The dWCST group had a similar pre-test score to that of the rWCST group, but a higher post-test and gain score. This is evidence that the dWCST measures learning potential opposed to practice effects. Compared to healthy controls, patients obtained similar gains but significantly lower pre- and post-test scores. The ratio of poor learners between-groups was not significantly different. This is partial support for the sensitivity of the dWCST to discriminate patients from controls.
Feasibility of Administration and General Psychometric Properties
The dWCST showed good feasibility and adequate psychometric properties. As hypothesized, the learning potential indices on the dWCST were significantly associated with one another which is in line with previous findings (Weingartz et al., Reference Weingartz, Wiedl and Watzke2008). This indicates that the learning potential indices measure a similar construct. The pattern of associations between the post-test score and gain score and memory performance is consistent with a study by Wiedl et al. (Reference Wiedl, Schöttke, Green and Nuechterlein2004), thus reinforcing the view that memory is associated with learning (Boosman, Visser-Meily, Winkens, & Van Heugten, Reference Boosman, Visser-Meily, Winkens and van Heugten2013; Lezak et al., Reference Lezak, Howieson and Loring2004). Effective performance requires patients to remember what they have learned during the brief training. Post-test scores were also significantly associated with language tests, which may be due to the verbal nature of the training. In contrast with our hypothesis, and the findings of Wiedl et al. (Reference Wiedl, Schöttke, Green and Nuechterlein2004), the post-test was not significantly associated with tests of executive functioning. This may be explained by the use of different tests of executive functioning between the studies. Dissociations in performance among executive tests and low between-test correlations have been reported previously (Jurado & Rosselli, Reference Jurado and Rosselli2007). In the current study, the LFT and TMT were used as measures of executive functioning. Similar to the WCST, these tests measure inhibition (Jurado & Rosselli, Reference Jurado and Rosselli2007). However, the WCST measures several additional functions (i.e., rule detection, concept formation, set maintenance) (Jurado & Rosselli, Reference Jurado and Rosselli2007) which may explain the lack of a significant association between the tests. Wiedl et al. (Reference Wiedl, Schöttke, Green and Nuechterlein2004) used the Tower of Hanoi which requires patients to formulate and execute a strategy while complying with a set of rules, and thus may assess abilities more closely related to the dWCST.
The substantial memory component and verbal nature of the training may also explain why some patients with low pre-test performance showed a substantial improvement after training whereas others showed only a marginal shift in performance. In particular, it is possible that patients who have considerable verbal memory and/or language impairments, may benefit less from training on this particular task compared to patients with better language and verbal memory functioning. Although other authors have suggested distinct neuro-cognitive profiles between the three learner groups in psychiatric populations (Bisoglio, Mervis, & Choi, Reference Bisoglio, Mervis and Choi2014; Kurtz et al., Reference Kurtz and Wexler2006; Rempfer, Hamera, Brown, & Bothwell, Reference Rempfer, Hamera, Brown and Bothwell2006; Wiedl, Wienöbst, Schöttke, Green, & Nuechterlein, Reference Wiedl, Wienöbst, Schöttke, Green and Nuechterlein2001;), between-group differences in this study were only found at trend-level. Also, the learning potential indices were not significantly associated with demographic characteristics, level of functional independence, or diagnosis (traumatic vs. non-traumatic ABI). It is possible that a range of other injury-related, cognitive or psychological factors contributed to the observed differences in amount of learning (e.g., injury location, visual memory, motivation) (Boosman et al., Reference Boosman, Visser-Meily, Winkens and van Heugten2013).
Learning Potential Versus Practice Effects
The lack of significant pre- to post-test improvement on the rWCST suggests that training effects were apparent over and above practice effects. Although only one patient in the rWCST group showed significant gains (i.e., strong learner), several other patients did show some improvement in pre- to post-test performance (see Figure 2). These marginal gains may reflect test-specific improvement rather than learning potential. For the conventional WCST, improvements between the first and second deck of cards have been attributed to continued exposure and corrective feedback (Merrick, Donders, & Wiersum, Reference Merrick, Donders and Wiersum2003). Differentiating test-specific gains from learning gains due to the training is a complex process. The lack of improvement on the rWCST may also be partly attributed to the demanding nature of the repeated administration condition in terms of the high number of trials and limited instructions and feedback provided by the experimenter. This may have influenced patients’ motivation and attention during the task. These issues may have contributed to the significant decline in performance observed for one patient in the rWCST group. Such a decline in performance has also been observed between the first and second deck of the conventional WCST (Sherer, Nick, Millis, & Novack, Reference Sherer, Nick, Millis and Novack2003).
Patients with ABI Versus Healthy Controls
In line with the results of a previous study that compared dWCST performance of healthy controls and patients with schizophrenia (Ohrmann et al., Reference Ohrmann, Kugel, Bauer, Siegmund, Kölkebeck, Suslow and Pedersen2008), patients obtained similar gains but lower pre- and post-test scores compared to healthy controls. This provides partial support for the sensitivity of the dWCST to discriminate patients from controls. The finding that patients and controls obtained similar gain scores may be explained by the fact that gain scores can produce disproportionately high or low scores (Weingartz et al., Reference Weingartz, Wiedl and Watzke2008). Of note is that additional analyses revealed no significant differences in dWCST scores of healthy controls and patients classified as high achievers. This, along with the finding that there were no decliners, suggests that the dWCST has greatest utility for differentiating between poor and strong learners.
Learning Potential Indices
The learning potential indices can be used to interpret individual test performance. In this study, three different learning potential indices were used: the post-test score, gain score and group classification. Several studies have discussed the strengths and limitations of these and other indices in more detail (Fiszdon & Johannesen, Reference Fiszdon and Johannesen2010; Waldorf, Wiedl, & Schöttke, Reference Waldorf, Wiedl and Schöttke2009; Weingartz et al., Reference Weingartz, Wiedl and Watzke2008). Most importantly, the learning potential indices need to be interpreted together to provide a clearer picture of a patient’s learning profile, that is, a patient’s initial performance, magnitude of change from training and their post-test performance. These scores can also be used to examine within-group variability when using the categorical approach. Any of the learning potential indices viewed in isolation could be misleading.
Limitations
Several limitations of this study should be noted. First, patients were not randomly allocated to the two assessment procedures (dWCST; rWCST). Although the groups did not differ in their level of functional independence and demographic characteristics, it is conceivable that between-group differences were confounded by other baseline differences between the two groups (e.g., cognition) and differences in the timing of assessment after admission to rehabilitation. The dWCST was administered significantly later than the rWCST. In the rWCST group, there were seven patients who were assessed in an early phase after ABI (<3 weeks post-ABI) which may have negatively influenced their potential for gains. However, additional analyses, including only patients who were assessed at least 3 weeks post-ABI revealed a similar pattern of non-significant (i.e., pre-test score) and significant (i.e., post-test score and gain score) findings.
A second limitation is the mixed etiology of our sample and the lack of injury specific analyses. However, clinical practice is also mixed. Third, although this study provides support for the validity of the dWCST, the clinical value of the dWCST for predicting rehabilitation outcomes was not established. Future studies could, for instance, examine the neuro-cognitive and psychological characteristics and long-term outcomes of poor and strong learners.
Conclusion
This study provides preliminary support for the validity of the dWCST as an instrument for assessing learning potential in patients with ABI. Evidence is provided for the absence of substantial practice effects as well as its ability to discriminate between patients with ABI and healthy controls. Further research is needed to investigate the predictive validity of the dWCST in comparison to other indices that are commonly used for prognostic purposes (e.g., functional independence).
Acknowledgments
This study is funded by the National Initiative Brain and Cognition (NIBC). This quick-result project is embedded in the pillar “The Healthy Brain, Program Cognitive Rehabilitation” [grant number 056-11-014]. The authors thank the participating rehabilitation centers: Adelante Zorggroep, Hoensbroek; De Hoogstraat Rehabilitation, Utrecht; Reade, Rehabilitation Center, Amsterdam; Rijndam Rehabilitation, Rotterdam; Rehabilitation Center Tolbrug, ’s-Hertogenbosch. Also, thanks to Ilse Blaas, Femke Coenen, Nelleke Kip, Elvire Mastboom, Linette Thiessens, Maartje van der Blij, Marel Vermeulen, and Syanah Wynn for their assistance in data collection. There were no competing interests.
