VALIDITY STUDIES

There appears to be a misperception by Schmidt (2003) that the D-KEFS tests have little validity data to support their use. It is the case that the D-KEFS manual contains little new validity data. However, early in the development of the D-KEFS, we elected to publish our validity data in peer-review scientific journals rather than in the test manual. We chose this approach for several reasons. First, the development of the D-KEFS took over 10 years, and we felt that scientific journals offered a more efficient way to disseminate the validity results. Second, validity data published in peer-reviewed journals allow all neuropsychologists the opportunity to evaluate the results of the studies, not just those who have purchased the test manual. Third, and most important, validity data published in peer-review journals have the added safeguards that go with the scientific review process, namely, that one's research methods and conclusions are deemed sound and appropriate by senior researchers who serve on the editorial boards of these journals (such safeguards do not exist for validity data published only in test manuals).

The D-KEFS test manual does provide references of the validity studies that had been conducted at the time of its publication, but these references appear to have been missed or overlooked by Schmidt (2003). In addition, studies of the D-KEFS published in the literature after the publication of the test manual also appear to have been overlooked by Schmidt (2003). Rather, Schmidt (2003) seems to focus on only one section of the test manual where we presented some new pilot data from small samples of patients with Alzheimer's or Huntington's disease, and to incorrectly regard these data as our primary attempt to conduct validity studies of the D-KEFS. However, we presented these pilot data in the manual to illustrate that, even in more diffuse dementia populations, intriguing dissociations can be found in component processes of executive functions as assessed by the D-KEFS.

As an update, we are providing here a list of validity studies that have been conducted with various D-KEFS tests both before and after the publication of the test manual. These studies have demonstrated that D-KEFS tests are sensitive to the assessment of executive-function deficits in numerous clinical populations, including patients with focal frontal-lobe lesions (Baldo et al., 2001; Baldo et al., 2003; Baldo et al., in press; Delis et al., 1992; Dimitrov et al., 1999); focal ventromedial prefrontal damage (Cato et al., in press); frontal-temporal dementia (Hallam et al., in press); mild cognitive impairment (Grant et al., in press); subcortical ischemic disease (Kramer et al., 2002); lateralized right-hemisphere damage (Crouch et al., 1996); Parkinson's disease (Beatty et al., 1990; Bondi et al., 1993; Dimitrov et al., 1999); multiple sclerosis (Beatty et al., 1995); chronic alcoholics (Beatty et al., 1993); normal aging (Wecker et al., 2000); autism (Kleinhans et al., 2003); Asperger's syndrome (Kleinhans et al., 2003); schizophrenia (Beatty et al., 1994); psychopathy (Kalinian, 2003); childhood stroke (Nichols et al., 2003); attention deficit disorder (Donnelly et al., 2001); and fetal alcohol syndrome (Mattson et al., 1999; Schonfeld et al., 2001). In addition, studies have been conducted that compared the D-KEFS Sorting Test and the Wisconsin Card Sorting Test in the assessment of executive-function deficits in clinical populations (Beatty et al., 1990; 1994). These and other D-KEFS publications are listed in the reference section below (note that the D-KEFS Sorting Test was formerly called the California Card Sorting Test).

The study of a test's validity is an ongoing process, and there is always a need for further investigations of all neuropsychological tests but particularly for new tests such as the D-KEFS. We should point out that there has been a lack of validity studies conducted for one of the D-KEFS instruments, the Proverbs Test. More research is especially needed for this test regarding its sensitivity to deficits in verbal abstraction skills.

In the past, a common approach to examining the validity of an assessment instrument has been to conduct factor-analytic studies of the test's variables using the instrument's normative sample or mixed clinical populations. However, we recently published a study that demonstrated empirically serious limitations of this approach in examining the validity of process-oriented neuropsychological tests (Delis et al., 2003). In writing an invited commentary on our factor-analytic study, Larrabee (2003), a leader in the field of factor analysis, agreed with our view and pointed out other limitations in past validity studies that have used factor-analytic techniques. It was for these reasons that we did not conduct factor-analytic studies of the D-KEFS tests or develop across-test composite indices based on factor-analytic results.

RELIABILITY

We conducted numerous reliability analyses of the D-KEFS measures, and these data are published in the D-KEFS technical manual (Delis et al., 2001b). Schmidt (2003) was also critical of these data, stating that only 17% of the reliability values published in the D-KEFS manual (53 of the 316 coefficients) were above a .80 value. However, the reliability coefficients found for the D-KEFS tests are comparable to those published for a number of other commonly used neuropsychological tests, including the Wisconsin Card Sorting Test and Wechsler Memory Scale–III. For example, none of the reliability coefficients of the Wisconsin Card Sorting Test are above a .80 value (Heaton et al., 1993). It may be the case that tests such as the D-KEFS, Wisconsin Card Sorting Test, and memory instruments tap a wider spectrum of complex, effortful cognitive processes compared to more homogeneous, fundamental tasks such as the Vocabulary or Picture Completion subtests, and as a result, the potential for performance variability or measurement error may be greater for more complex tests. However, it is often the complexity of these tasks that make them so sensitive to the detection of even mild brain damage. In addition, tests of executive functions and memory often pose special problems for calculating reliability in traditional ways (e.g., item-interdependence makes it difficult to use split-half reliabilities; see Delis et al., 2000). Although caution must always be exercised in ascribing brain damage to low scores on any test, especially for more complex tasks such as the Wisconsin Card Sorting Tests, D-KEFS tests, and memory instruments, the utility of these instruments for detecting neurocognitive deficits has been demonstrated in numerous studies.