INTRODUCTION
The goal of this review is to highlight key developments and turning points in neuropsychological assessment over the past several decades. We will touch on a breadth of assessment-related topics to provide an overview of the evolution of neuropsychological evaluation, leading to consideration of current practices and areas for future research. One limitation is that the content here is selective and admittedly draws heavily from practices based in the United States; while a majority of published neuropsychological tests and test norms have been developed within the United States and primarily for English-speaking populations, such cultures actually contain a disproportionately small fraction of the world’s population. Still, the accomplishments and progress of neuropsychological assessment to date are remarkable, and we hope to inspire continued growth through this review. By its nature, neuropsychology is a distinctly transdisciplinary service. Neuropsychologists are in the unique position to be able to interpret brain measurement across modalities in a clinically meaningful way (e.g., neuroimaging, proteomics, genetics, technology-based service delivery). Leveraging the systematic and foundational approaches of our field, neuropsychologists are encouraged to continue to expand their repertoire and impact by creatively integrating brain assessment techniques across biological, cognitive science, and technology-based platforms as the field continues to evolve and define itself.
CLINICAL NEUROPSYCHOLOGICAL ASSESSMENT
Although the roles of neuropsychologists have evolved over time, the primary purposes for clinical neuropsychological assessment have remained fairly constant, to: (1) detect neurological dysfunction and guide differential diagnosis, (2) characterize changes in cognitive strengths and weaknesses over time, and (3) guide recommendations regarding everyday life and treatment planning. With the advent of increasingly sensitive and multimodal neurologic biomarker data, neuropsychological assessment shifted from its original role in “finding the lesion” to in-depth characterization of the patterns arising from disruptions in brain-behavior relationships. Although perhaps taken for granted given the body of gold-standard measures available at this stage in the field, advancement of cognitive assessment tools that are (1) sensitive to brain–behavior relationships of interest, (2) developed alongside other brain measurement tools, and (3) accessible and feasible across diverse settings continue to be needed areas of study. Ultimately, the neuropsychological questions we ask are only limited by the tools with which we have to probe the brain.
NEUROPSYCHOLOGICAL ASSESSMENT APPROACHES: A BRIEF MODERN HISTORY
Lurian Approach
Alexander Luria’s seminal work following World War II in Russia represents an important early time-point in neuropsychological assessment. Luria’s innovative ideas that the brain underlies the ability to carry out goal-directed behaviors and is shaped by environmental and cultural contexts (e.g., language) led to his systematic characterization of functional brain systems (Luria, Reference Luria1966). Although largely qualitative, Luria’s primary goal was to describe the cerebral bases that support corresponding functional systems, emphasizing the importance of understanding the multiple components that may comprise even simple neurobehavioral functions (Luria, Reference Luria1966). Luria stimulated appreciation of brain specialization and moved away from the one-size-fits-all diagnosis of brain disease; however, his techniques were highly flexible and nonstandardized, making it difficult to reliably reproduce across examiners and patients.
Recognizing this problem, his student, Anne-Lise Christensen, published a more structured version of the Lurian approach, combining both qualitative and quantitative aspects of his battery (Draper, Reference Draper1976). Ultimately, Charles Golden further standardized and combined the works of both Luria and Christensen in his development of the Luria-Nebraska Neuropsychological Battery (LNNB; Golden, Purisch, & Hammeke, Reference Golden, Purisch and Hammeke1979), scaling down almost 2000 original measurement items to 269 items covering 14 scales (e.g., motor, rhythm, memory, intelligence; Purisch, Reference Purisch2001).
In the United States, other neuropsychological assessment standardizations were also developing. In the 1950s, Arthur Benton was among the first thought leaders to criticize neurology for the lack of validated tools to measure common neurological syndromes (e.g., aphasia, agnosia). Through his pursuit of systematic neurobehavioral assessment, Benton developed several individual measures that are still widely used today (e.g., Benton Visual Rention Test) and shaped the Iowa-Benton school of neuropsychology (Tranel, Reference Tranel2009). Additionally, through these test developments, Benton began raising awareness of the apparent import of demographic factors (e.g., age and education) on test performances.
Flexible Battery
One assessment style that naturally extended from clinicians’ early instincts to describe observed behaviors was the flexible integration of standardized measures. In the pure flexible battery approach, neuropsychologists administer only those measures directly related to the patient’s presenting symptoms (e.g., multiple memory measures to characterize memory symptoms). Consequently, the administered battery inherently changes with a patient’s referral question, while the examiner pursues the aim of being both as efficient and sensitive as possible to the presenting problem (Benton, Reference Benton1994; Lezak, Reference Lezak1976).
Fixed Battery Approach
Concurrently, an entirely distinct assessment approach was developing that used a common, relatively comprehensive set of measures administered to all patients regardless of presenting symptoms, neuromedical history, or clinically apparent syndrome. Although lengthier and requiring greater burden and expense, fixed batteries maintain a constant testing condition, allowing for direct comparison of measures across disparate neurological disorders. Given their highly standard nature, research regarding fixed batteries is more easily facilitated, has garnered more comprehensive normative data, better understanding of their psychometric properties, and a broader empirical base from which to draw interpretations.
Two major leaders in the history of neuropsychological assessment pioneered the fixed battery approach, Ward Halstead and Ralph Reitan, his student (Reitan & Davidson, Reference Reitan and Davidson1974). As a physiological psychologist in the mid-twentieth century, Halstead espoused a philosophy of neuropsychological assessment that was akin to a series of scientific experiments. He believed that there was no science in the individual event, and emphasized the need to develop exact, systematic procedures with sufficient comparison cases to interpret an individual test score (Reitan, Reference Reitan1994). Reitan similarly held strong beliefs that empiricism should be central in neuropsychological assessment (Grant & Heaton, Reference Grant and Heaton2015).
Building off of Halstead’s early work, Halstead and Reitan closely observed individual patients in clinic and at home, helping to shape and refine measures that captured the observed brain–behavior relationships that could be generalized more broadly. Backed by more than two decades of blind interpretation of patient results and studies of groups with diverse neurologic disorders, Reitan ultimately published the Halstead-Reitan Battery (HRB) in 1985. The goal of the HRB was to be a systematic, quantitative means to measure the presence, location, extent, and nature of neurological disease equitably across clinical syndromes (Reitan, Reference Reitan1985). With its replicable procedures and highly quantitative results, the HRB intended to shift the practice of neuropsychology from an “art to a science” (Reitan, Reference Reitan1985) and indeed has been one of the most researched assessment batteries in neuropsychology (Kreutzer, DeLuca, Caplan, Reference Kreutzer, DeLuca and Caplan2011).
As part of the trend toward highly quantitative assessment in the 1970s, some early approaches attempted to develop computerized algorithms for neuropsychological interpretation: the Key Approach by Russell, Neuringer, and Goldstein (Reference Russell, Neuringer and Goldstein1970), Finkelstein’s BRAIN (Reference Finkelstein1977), and Adams’ ability-based algorithm (Adams, Reference Adams1975). An overarching goal of these automated programs was to leverage the predictive power of computers to more accurately localize, categorize, and potentially infer etiology of brain damage such that “a technician or clerk, without knowledge of neurology, neuropsychology, or psychometrics” could use these tools (Deuel, Reference Deuel1971, p. 95). Although forward-thinking, the accuracy of such automated systems was inconsistent. The programs reliably identified presence of brain injury, but inferences regarding localization and etiology were imprecise and led to the conclusion that contextual information (i.e., clinical history) was critical to neuropsychological interpretation (Adams, Kvale, & Keegan, Reference Adams, Kvale and Keegan1984).
Boston Process Approach
Another important turning point in neuropsychological practices was the development of what is now referred to as the “Boston Process Approach” led by Edith Kaplan (Kaplan, Reference Kaplan1988). Although the Boston approach can be used with either flexible or fixed batteries, it is most commonly associated with and perhaps conceptually suitable to more flexible evaluations. The Boston approach places emphasis on how the patient comes to an answer (e.g., types of errors committed) rather than reliance on a single objective score. In this approach, testing the limits of a patient’s cognitive abilities to elicit behaviors that may not traditionally present during standardized testing is emphasized.
Through the influence of Kaplan’s training, Dean Delis advanced the standardization of the Boston approach, along with Kaplan and Joel Kramer, into what have come to be some of the most widely used neuropsychological assessment tools today. The California Verbal Learning Test (CVLT, Delis, Kramer, Kaplan, & Ober, Reference Delis, Kramer, Kaplan and Ober1987; and CVLT-second edition, Delis, Kramer, Kaplan, & Ober, Reference Delis, Kramer, Kaplan and Ober2000) and the Wechsler Adult Intelligence Scale-Revised as a Neuropsychological Instrument (Kaplan, Fein, Morris, Delis, Reference Kaplan, Fein, Morris and Delis1991), followed by the Delis Kaplan Executive Functions System (DKEFS; Delis, Kaplan, Kramer, Reference Delis, Kaplan and Kramer2001) were borne out of a need to provide statistical parameters quantifying cognitive strategies that deviated from expectations (e.g., CVLT primacy effects; DKEFS error analysis).
Flexible Evaluation Approach
To avoid misconceptions, the pure flexible battery differs from the flexible evaluation approach that is used by over three-quarters of contemporary clinical neuropsychologists (Larrabee, Reference Larrabee2008). Current common practice represents aspects of both flexible and fixed approaches, such that a fairly standardized, fixed set of measures is given to most patients with some flexibility to add or subtract measures given the specific referral question (Bigler, Reference Bigler2007).
NORMATIVE STANDARDS
As these fundamental assessment tools developed, there also grew increasing recognition that nonneurological, premorbid factors were significantly impacting test scores. The concept that cultural environment shapes brain development dates back at least to Luria (Luria, Reference Luria1966). Mounting evidence in the 1970-80s then quantified that >40% of the variance in many test scores was accounted for by a single demographic factor (e.g., age, education), while even greater amounts of normal variance could be adjusted for when combing demographic factors together (Heaton, Grant, Matthews, Reference Heaton, Grant and Matthews1991). Yet, development of a systematic way to adjust for such premorbid factors lagged.
While initial normative efforts focused on the effects of age alone (e.g., Wechsler measures), Robert Heaton, one of Reitan’s students, recognized this gap and pioneered the development of normative standards adjusting for multiple background demographic factors. Publication of the “Heaton Norms” for the expanded HRB in 1991 signaled the most comprehensive standards at the time, adjusting for effects of age, sex, and education (Heaton et al., Reference Heaton, Grant and Matthews1991). His subsequent revision in 2004 incorporated the effects of race, integrating the increasing literature demonstrating its influence on test performance (Heaton, Miller, Taylor, & Grant, Reference Heaton, Miller, Taylor and Grant2004).
His advocacy to apply race as a proxy for important socio-cultural factors that impact brain development (e.g., education quality, socioeconomic status, medical care access, testing acculturation) represents a pivotal point in assessment approaches that at times is still received as controversial today (see detailed discussion by Manly, Reference Manly2008). Heaton was not alone in addressing the need: in parallel, the Mayo Clinic, led by Robert Ivnik and Glenn Smith, was carrying out the Mayo Older Americans Normative Studies (MOANS), which aimed to improve the utility of commonly used neuropsychological tools for older adults (Ivnik et al., Reference Ivnik, Malec, Smith, Tangalos, Petersen, Kokmen and Kurland1992a, Reference Ivnik, Malec, Smith, Tangalos, Petersen, Kokmen and Kurland1992b). The MOANS project began publishing normative standards for various measures in 1992, and continued to update these standards through the early 2000s, including adjustments for racial minorities and careful consideration of what is considered neurologically normal in the elderly (e.g., Harris, Ivnik, & Smith, Reference Harris, Ivnik and Smith2002; Lucas et al., Reference Lucas, Ivnik, Willis, Ferman, Smith, Parfitt and Graff-Radford2005; Machulda et al., Reference Machulda, Ivnik, Smith, Ferman, Boeve, Knopman and Tangalos2008; Smith, Wong, Ivnik, & Malec, Reference Smith, Wong, Ivnik and Malec1997).
Other novel normative advancements were the development of co-normed batteries (e.g., WAIS-IV, WMS-IV, CVLT-II) and “robust norms” (Zhu & Tulsky, Reference Zhu and Tulsky2000). Co-normed batteries allow for test score interpretation on a common metric that is adjusted for the same demographic factors from the same normative cohort. Thus, they provide an equitable standard against which performances on all tests can be compared and interpreted. Robust normative standards simply refer to the use of as neurologically normal individuals as possible to provide standards of comparison. This may be particularly relevant in adult standards in which underlying neurodegenerative pathology may accumulate decades before clinical manifestation (Holtzer et al., Reference Holtzer, Goldin, Zimmerman, Katz, Buschke and Lipton2008). Inclusion of preclinical but asymptomatic individuals in a normative cohort may increase representativeness, but it also increases variability and decreases the sensitivity to acquired injury or disease. Nevertheless, there are many practical difficulties that limit feasibility in identifying highly neurologically normal individuals for inclusion in large-scale normative studies (e.g., expense in accessing brain scans and comprehensive family histories).
Despite major growth in normative development in the past two decades, many other interesting and potentially important questions about how to operationalize “normal” brain functioning remain unanswered. For example, do age effects on cognition differ across presumably “normal” groups based upon educational backgrounds, cognitive reserve, access to quality healthcare, nutrition, lifestyle factors, or environmental risks? Do education effects differ on the basis of how resourced the person’s educational system has been, how education was valued in the home, and what other stresses in the environment may have interfered with educational pursuits?
For example, Jennifer Manly’s innovative work parsing out contributing factors that account for racial differences on neuropsychological testing demonstrates that quality of education may be a critical underlying differentiator in these testing disparities (Manly et al., Reference Manly, Jacobs, Touradji, Small and Stern2002). Other questions include, will gender effects on test performance differ based upon the society’s traditional gender roles, gender-based opportunities and expectations? Also, if such effects of background factors change over time, how will this inform the need for updated normative standards, or perhaps even generation-based norms that reflect different effects of age, education, gender, and race/ethnicity? Answers to these questions now, and how they may change across time with shifting cultural values (e.g., gender/minority equality programs), may importantly impact our expectations and interpretation of neuropsychological test performances in the future.
CULTURAL AND INTERNATIONAL CONSIDERATIONS
Relatedly, during the past several decades, there has been a dramatic shift in the demographics of the United States, especially including individuals who speak languages other than English and who were born, grew up, and were educated (in part or totally) in cultures outside of the United States. Converging evidence demonstrates that test performance is significantly influenced by the values, customs, experiences, and cognitive styles that differ from the majority culture on which tests were originally developed and standardized (e.g., Arnold, Montgomery, Castaneda, & Longoria, Reference Arnold, Montgomery, Castaneda and Longoria1994). Although their overlap and individual contributions are less clear, a host of culture-specific factors are undoubtedly associated with test scores across populations even within a single country (e.g., United States): acculturation, language, education quality and literacy, poverty and low socioeconomic status, familiarity with the evaluation process, and communication style (Fuiji, Reference Fuiji2017).
As a rule of thumb, the larger the discrepancy between the individual being assessed and the majority culture in which the measure was developed, the higher the possibility that the test score may not reflect the construct it is posited to measure. Even when a measure is developed in the patient’s native language and for examinees with the same racial/ethnic background, variability within a culture may still significantly influence test scores. For example, a recent publication demonstrated significant effects of frequency of everyday Spanish language use and location of education and birth among healthy U.S. Hispanic adults, even on a battery developed and normed in a national Hispanic sample (Flores et al., Reference Flores, Casaletto, Marquine, Umlauf, Moore, Mungas and Heaton2017).
These issues are further underscored and potentially amplified when neuropsychological measures developed in American culture are applied in non-U.S. contexts. The importance of the basic human ability being assessed may differ across cultures and be dependent on the values and everyday requirements of a given culture. For example, in the majority U.S. culture, speeded information processing is valued and educational experiences frequently reinforce that faster performances indicate better results. However, in Hispanic culture, speed and quality are oftentimes contradictory goals, such that slow and careful processing is thought to lead to the best results (Ardila, Rosselli, Matute, & Guajardo, Reference Ardila, Rosselli, Matute and Guajardo2005).
Some authors have suggested that what are needed in developing countries are tests that are more relevant to the backgrounds of those populations. However, it is unknown whether changes in the tests themselves would make them more valid in other settings, or whether all that is needed are different normative standards. Nonetheless, it may be that some tests simply are not useful across settings whereas others are, and we do not yet know what factors may drive these differences. Similarly, especially in the large and diverse developing world, it is unclear whether normative standards in one country or setting can be generalizable to other settings and, if they can, what common factors account for that generalizability. These are all questions to which neuropsychology researchers still need to seek answers. The need for valid neuropsychological practice and research is worldwide and it is simply not feasible to develop population-specific test norms, or new tests for every population on earth.
Currently, measures developed in the United States and Europe are commonly applied in other cultural contexts, although there are several notable efforts for culture-specific test and normative developments. As one example, Egypt has been at the forefront developing standardized Arabic evaluations, beginning as early as the 1920s (led by El-Kabbani) and extending into commonly used batteries developed in other cultures (e.g., Halstead-Retain and Luria-Nebraska; Al-Joudi, Reference Al-Joudi2015). A recent review by Fasfous, Al-Joudi, Puente, and Perez-Garcia (Reference Fasfous, Al-Joudi, Puente and Perez-Garcia2017) identified that, although 117 individual neuropsychological measures were used in studies examining Arabic populations, only 53 were normed in the culture and evidenced appropriate cultural adaptation and validation per standard guidelines. Notably, almost all of the identified measures were tools initially developed in the United States and Europe: Verbal Fluency, Wechsler Memory and Intelligence batteries, Trail Making Test, Wisconsin Card Sorting Test, Ravens Matrices, and Bayley Scale of Infant Development-second edition were among the most extensively used cognitive measures in contemporary Arabic countries (Fasfous et al., Reference Fasfous, Al-Joudi, Puente and Perez-Garcia2017).
Other efforts to develop normative standards outside of American and European cultures suggest comparability of the construct measured when normative standardizations and underlying issues of cultural validity are appropriately addressed (e.g., Zambia, Hestad et al., Reference Hestad, Menon, Serpell, Kalungwana, Mwaba, Kabuba and Heaton2016, Kabula et al., Reference Kabuba, Menon, Franklin, Heaton and Hestad2017; China, Gupta et al., Reference Gupta, Iudicello, Shi, Letendre, Knight, Li and Atkinson2014, Heaton et al., Reference Heaton, Cysique, Jin, Shi, Yu, Letendre and Marcotte2008, Shi et al., Reference Shi, Kang, Yao, Ma, Li, Liang and Zhang2015; Brazil, de Almeida, et al., Reference de Almeida, Ribeiro, de Pereira, Badiee, Cherner, Smith and Heaton2013; India, Ghate et al., Reference Ghate, Mehendale, Meyer, Umlauf, Deutsch, Kamat and Alexander2015, Kamat et al., Reference Kamat, Ghate, Gollan, Meyer, Vaida, Heaton and Mehendale2012, Reference Kamat, McCutchan, Kumarasamy, Marcotte, Umlauf, Selvamuthu and Bharti2017, Malda, van de Vijver, Srinivasan, Transler, & Sukumar, Reference Malda, van de Vijver, Srinivasan, Transler and Sukumar2010; Czech Republic, Bezdicek et al., Reference Bezdicek, Motak, Axelrod, Preiss, Nikolai, Vyhnalek and Ruzicka2012, Reference Bezdicek, Stepankova, Motak, Axelrod, Woodard, Preiss and Poreh2014; South Africa, Nell, Reference Nell1999; Nell, Myers, Colvin, & Rees, Reference Nell, Myers, Colvin and Rees1994; Cameroon, Ruffieux, et al., 2010; South Korea, Ko, Rosen, Simpson, & Brown, Reference Ko, Rosen, Simpson and Brown2014), although there are some exceptions (e.g., lexical fluency and block design in Cameroon, Ruffieux, et al., 2010).
A common theme in reports of these cross-cultural test adaptations is the nonequivalence of the testing experience from Western to non-Western environments (e.g., Cassitto, Camerino, Hanninen, & Anger, Reference Cassitto, Camerino, Hanninen and Anger1990; Valciukas, Levin, Nicholson, & Selikoff, Reference Valciukas, Levin, Nicholson and Selikoff1986). A host of factors, including repetition of test instructions, in-depth orientation to the testing environment, performance motivation, and profound effects of vocation and socioeconomic status have been identified as potentially contributing to test performance (Weinstein, Fucetola, & Mollica, Reference Weinstein, Fucetola and Mollica2001). As the world becomes increasingly interconnected and the need to assess and treat culturally different individuals grows, our field and its journals have an ethical responsibility to increase their priorities for such cross-cultural research endeavors.
SCIENTIFIC ADVANCEMENTS IN NEUROPSYCHOLOGICAL ASSESSMENT
Techniques for Defining Longitudinal Change
Neuropsychological batteries were originally constructed with the goal of identifying brain dysfunction, but a prominent emerging role of the neuropsychologist is to monitor syndrome progression or recovery via repeated evaluations. This is particularly true in the rehabilitation setting. As such, quantifying what constitutes significant change on a test battery is another relatively recent advance in our empirical assessment approach. Among the first and still widely used methods is the Reliable Change Index (RCI; Jacobson & Truax, Reference Jacobson and Truax1991). The RCI was developed by Jacobson and Truax from the personality literature to establish whether the difference between an initial and follow-up test score exceeds what can be attributed to chance variation versus a clinically significant change. The RCI is based upon the reliability of the individual measure, which can be used to estimate the standard error of the difference between scores on the test in question.
Several years later, Chelune, Naugle, Luders, Selak, and Awad (Reference Chelune, Naugle, Luders, Selak and Awad1993) developed change modeling that additionally adjusted for practice effects, and McSweeney, Naugle, Chelune, and Luders (Reference McSweeny, Naugle, Chelune and Luders1993) proposed a linear multivariate regression approach adjusting for demographic effects and baseline performances to additionally control for regression to the mean. In 1996, similar regression models developed by Sawrie et al. and Hermann et al. included test–retest intervals (Standardized Regression Based norms) to quantify cognitive changes following epilepsy surgeries (Hermann et al., Reference Hermann, Seidenberg, Schoenfeld, Peterson, Leveroni and Wyler1996; Sawrie, Chelune, Naugle, & Luders, Reference Sawrie, Chelune, Naugle and Luders1996). These methods along with more complex multiple regression-based modeling including baseline scores, other potentially contributing factors (e.g., retest interval, demographics, nonlinear effects), and their interactions were then directly compared by Temkin, Heaton, and colleagues in neurologically normal and clinical groups (Heaton et al., Reference Heaton, Temkin, Dikmen, Avitable, Taylor, Marcotte and Grant2001; Temkin, Heaton, Grant, & Dikmen, Reference Temkin, Heaton, Grant and Dikmen1999).
Of interest, the authors found that the simple RCI was the least accurate method while the other approaches appeared comparably accurate. However, regardless of approach, baseline performance on individual tests had the largest impact on change classification, especially in clinical cohorts, and should be accounted for when developing change modeling (see Cysique et al., Reference Cysique, Franklin, Abramson, Ellis, Letendre, Collier and Simpson2011 for further support and importance of “neuropsychological competence” predicting longitudinal cognitive change).
These studies laid the groundwork for the Serial Assessment report from the Wechsler Advanced Clinical Solutions software and the development of other calculators and published regression equations aiding in our quantification of cognitive “impairment” and change scores. For example, Crawford and colleagues leveraged existing datasets to develop enriched regression-based predictions of cognitive functioning for the individual case, which may also be applied longitudinally (Crawford & Garthwaite, Reference Crawford and Garthwaite2007; Crawford, Garthwaite, Denham, & Chelune, Reference Crawford, Garthwaite, Denham and Chelune2012). Such programs that use the multitude of existing datasets to answer ongoing neuropsychological questions are innovative and will help optimize the growth of our field.
Common Metric Assessment Batteries
In an effort to leverage data and combine scientific efforts, development of standardized methods to evaluate cognition comparably across studies has been a major recent goal of the U.S. National Institutes of Health (NIH). Beyond answering innumerable scientific questions with greater power, such tools are particularly well-positioned for multisite epidemiological studies and clinical trial monitoring. With these goals in mind, several novel common metric batteries were developed that are population-specific (Measurement and Treatment Research to Improve Cognition in Schizophrenia, MATRICS; Nuechterlein et al., Reference Nuechterlein, Green, Kern, Baade, Barch, Cohen and Marder2008), construct-specific (Executive Abilities: Measure and Instruments for Neurobehavioral Evaluation and Research, EXAMINER; Kramer et al., Reference Kramer, Mungas, Possin, Rankin, Boxer, Rosen and Widmeyer2014), and address broader, lifespan neurobehavioral assessment questions (NIH Toolbox for Assessment of Neurological and Behavioral Function; Gershon et al., Reference Gershon, Wagster, Hendrie, Fox, Cook and Nowinski2013).
While these batteries represent massive accomplishments in the standardized assessment of neurologic function, and potential turning points in our resulting future understanding of brain function, they still maintain the common pitfalls of our traditional set of tools. Namely, although large infrastructure and broad expertise were dedicated to the theoretical and empirical development of these measures, efforts toward gathering sufficient normative information across populations, languages, and especially minority cultures, is ongoing and difficult, and generalizability of results with each battery remains to be demonstrated.
Novel and Translational Constructs
Although the traditional set of cognitive domains are an important foundation in the neuropsychological toolset (e.g., episodic memory, language, visuospatial skills), several domains drawing on the cognitive psychology and neurosciences literatures are highly complementary and warrant mention. These constructs have significantly furthered our capture of brain–behavior relationships and their assessments have demonstrated strong psychometric properties. Application of such measures depends entirely on the investigator or clinician’s aims (e.g., if aiming to measure interpersonal function, social cognition is highly appropriate), although we encourage readers to consider such constructs for inclusion in their neuropsychological toolkit. We note only several exemplars here and encourage reading across related fields for further inspiration (e.g., Carter & Barch, Reference Carter and Barch2007; Merikle, Smilek, & Eastwoord, Reference Merikle, Smilek and Eastwood2001).
Action Fluency
Although not a novel cognitive construct, action fluency (ability to generate verbs, for example, “things that people do”) was developed as a counterpart to the traditional lexical and semantic fluency tasks. Action fluency taps into a more selective frontal-striatal network compared with the more temporal networks involved in noun generation (Piatt, Fields, Paolo, & Troster, Reference Piatt, Fields, Paolo and Troster1999). Clinical lesion studies demonstrate a double-dissociation such that patients with frontal versus temporal lobe injury demonstrate disproportionate difficulties generating verbs than nouns, and vice versa (Damasio & Tranel, Reference Damasio and Tranel1993). Similarly, more recent work in Parkinson’s and HIV diseases (diseases with known selective involvement of frontal-striatal networks) show disproportionate disparities on verb compared to lexical or semantic generation tasks that is importantly predictive of daily functioning outcomes (Piatt, Fields, Paolo, Koller, & Troster, Reference Piatt, Fields, Paolo, Koller and Troster1999; Woods, Carey, Troster, Grant, & Centre, Reference Woods, Carey, Troster and Grant2005; Woods, Scott, et al., Reference Woods, Scott, Sires, Grant, Heaton and Troster2005; Woods et al., Reference Woods, Morgan, Dawson, Scott and Grant2006).
Prospective Memory
Prospective memory (PM) is the ability to remember an intention in pursuit of a future goal. PM necessitates both executive (e.g., planning, monitoring, set-shifting) and episodic memory (e.g., recalled intention) abilities, drawing on frontal-temporal systems and their related networks (Ellis & Kvavilashvili, Reference Ellis and Kvavilashvili2000). Although initial studies date back to Elizabeth Loftus in Reference Loftus1971, the past decade has seen a surge of renewed interest in the neuropsychological study of PM (Raskin, Reference Raskin2004). Prospective remembering declines across a range of neurologic conditions (e.g., HIV, Carey et al., Reference Carey, Woods, Rippeth, Heaton and Grant2006; schizophrenia, Twamley, Woods, Dawson, Narvaez, & Jeste, Reference Twamley, Woods, Dawson, Narvaez and Jeste2007; Parkinson’s disease, Kliegel, Altgassen, Hering, & Rose, Reference Kliegel, Altgassen, Hering and Rose2011; even normal aging, Einstein & Mcdaniel, Reference Einstein and Mcdaniel1990) and is consistently associated with declines in everyday functioning, even more so than traditional episodic memory measures (Einstein & Mcdaniel, Reference Einstein and Mcdaniel1990; Kliegel et al., Reference Kliegel, Altgassen, Hering and Rose2011; Woods et al., Reference Woods, Iudicello, Dawson, Moran, Carey, Letendre and Grant2007, Reference Woods, Iudicello, Moran, Carey, Dawson and Grant2008, Reference Woods, Dawson, Weber, Gibson, Grant and Atkinson2009; Zogg, Woods, Sauceda, Wiebe, & Simoni, Reference Zogg, Woods, Sauceda, Wiebe and Simoni2012).
Social Cognition
Social cognition was originally identified through the social psychology literature and is a multifaceted construct referring to the ability to perceive, interpret, and generate responses to the intentions and behaviors of others (see McDonald, Reference McDonald2017, for a comprehensive review; McDonald, Flanagan, & Rollins, Reference McDonald, Flanagan and Rollins2011; McDonald, Flanagan, Rollins, & Kinch, Reference McDonald, Flanagan, Rollins and Kinch2003). Examples of social cognition tasks include identifying emotions, theory of mind (ability to understand and attribute others’ mental states), and knowledge of social pragmatics (Green, Olivier, Crawley, Penn, & Silverstein, Reference Green, Olivier, Crawley, Penn and Silverstein2005).
Although its early and most in-depth neuropsychological study is largely in the context of schizophrenia-related adaptive functioning (e.g., NIMH MATRICS), there has been an explosion of studies in the last several years recognizing the primary social cognition impairment across neurologic conditions (e.g., behavioral variant of frontotemporal dementia, Shany-Ur & Rankin, Reference Shany-Ur and Rankin2011; Autism spectrum disorder, Pinkham, Hopfinger, Pelphrey, Piven, & Penn, Reference Pinkham, Hopfinger, Pelphrey, Piven and Penn2008; Alzheimer’s disease, Bediou et al., Reference Bediou, Ryff, Mercier, Milliery, Henaff, D’Amato and Krolak-Salmon2009; substance use disorders, Homer et al., Reference Homer, Solomon, Moeller, Mascia, DeRaleau and Halkitis2008). Importantly, social cognitive measures are more strongly associated with community functioning outcomes than traditionally assessed constructs (Pijnenborg et al., Reference Pijnenborg, Withaar, Evans, van den Bosch, Timmerman and Brouwer2009).
Incorporating social cognition measures into the standard neuropsychological toolbox may, therefore, afford highly relevant information for guiding daily functioning recommendations. Several examples of well-validated measures that neuropsychologists may consider include the NIMH MATRICS Social Cognition subdomain (Green et al., Reference Green, Nuechterlein, Gold, Barch, Cohen, Essock and Marder2004), Implicit Association Test (Greenwald, McGhee, & Schwartz, Reference Greenwald, McGhee and Schwartz1998), Ekman and Friesen Battery of Emotion Processing (Ekman & Friesen, Reference Ekman and Friesen1976), and Benton Facial Recognition Test (Benton, Hamsher, Varney, Spreen, Reference Benton, Hamsher, Varney and Spreen1983).
Everyday Functioning
A common complaint regarding neuropsychological evaluations is their apparent lack of relevance to the real-life problems that the patient may be experiencing. Expansion of the standardized neuropsychological toolset to include assessments of everyday abilities that are also sensitive to neurologic decline will enrich the ecological validity of evaluations. The Behavioral Assessment of the Dysexecutive Syndrome (BADS) was among the first batteries developed with this in mind (Wilson, Alderman, Burgess, Emslie, and Evans, Reference Wilson, Alderman, Burgess, Emslie and Evans1996). For example, one of the BADS subtests, the Six Elements Test, is a widely used measure of cognitive multitasking that elicits planning and strategic thinking in a relatively unstructured context.
Other direct measures of daily living skills have grown in popularity in research settings, although their application in the clinic appears less common. A review by Moore, Palmer, Patterson, and Jeste (Reference Moore, Palmer, Patterson and Jeste2007) identified 31 published performance-based measures of various functional skills, ranging from medication management and cooking to dressing and safety. We also encourage review of other comprehensive texts, such as self-report everyday measures as reviewed in Robyn Tate’s A compendium of Test, Scales, and Questionnaires (Reference Tate2010) and The Neuropsychology of Everyday Functioning edited by Marcotte and Grant (Reference Marcotte and Grant2009), which provides in-depth performance-based measurement approaches of daily functioning across diseases (e.g., driving simulators).
Additionally, given the ubiquity of the Internet, tools assessing the ability to navigate the World Wide Web are highly relevant, easily accessible, and gaining momentum. For example, Woods and colleagues (2016) recently examined the validity of the Simulated Market Task (S-MarT) and Web-based Evaluation of Banking Skills (WEBS) as examiner-controlled Web sites of common household (i.e., shopping and banking) and health-related (i.e., pharmacy refills and healthcare communications) abilities. These tasks differentiated between HIV+ individuals with and without mild cognitive impairment independent of previous Internet experience, and were moderately related to performances on standard neuropsychological measures (Woods et al., Reference Woods, Iudicello, Morgan, Cameron, Doyle and Smith2016; Woods, et al., Reference Woods, Iudicello, Morgan, Verduzco, Smith and Cushmanin press).
Although the reliability, internal consistency, and concurrent validity of these real-world measures are consistently high, surprisingly few studies have directly examined their predictive validities. Given that this is one of their primary goals, future work supporting the ability of objective functional measures to relate to real world status is needed, although admittedly difficult to achieve (e.g., operationalization of real world status is inherently complex). In an effort to not simply become another neuropsychological test, performance-based instrumental activities of daily living measures must also balance ecological validity (e.g., less structured environment for task completion) and ability to reliably administer the task. Empirical translations of neuropsychological assessments that more precisely predict and guide patient recommendations for the real world continue to be an area of ongoing growth for the field.
TRENDS AND FUTURE DIRECTIONS
Technological Advancements: Computerized Neuropsychological Assessment
As technology becomes ever more accessible, the availability of computer- and tablet-based neuropsychological assessments has rapidly risen over the past decade. Recent U.S. NIH assessment initiatives investing in the development of computerized tools (e.g., NIH EXAMINER, Kramer et al., Reference Kramer, Mungas, Possin, Rankin, Boxer, Rosen and Widmeyer2014; NIH Toolbox, Gershon et al., Reference Gershon, Wagster, Hendrie, Fox, Cook and Nowinski2013) mirror and further highlight this trend. Although we are only able to briefly touch upon computerized assessment techniques here, we refer readers to the 2012 American Academy of Clinical Neuropsychology and National Academy of Neuropsychology position paper discussing relevant issues ranging from ethics and privacy to psychometrics, device marketing, and automated reporting services (Bauer et al., Reference Bauer, Iverson, Cernich, Binder, Ruff and Naugle2012).
Of historical relevance, the MicroCog Assessment of Cognitive Functioning in 1993 represents one of the first computerized cognitive assessment batteries commercially available via Pearson (see review, Elwood, Reference Elwood2001). Relatedly, the Automated Neuropsychological Assessment MetricsR (ANAM) is unique in that it was initially developed by the U.S. Department of Defense and became commercially available through Vista Sciences at the University of Oklahoma (Reeves, Winter, Bleiberg, & Kane, Reference Reeves, Winter, Bleiberg and Kane2007). There are now decades of clinical and laboratory data generated on the ANAMR, including >300 peer-reviewed articles, detailing its development and use (for comprehensive review, see the McCaffrey (Reference McCaffrey2007) special issue of Archives of Clinical Neuropsychology).
While several other relatively older computerized batteries (e.g., Cambridge Neuropsychological Test Automated Battery, Sahakian & Owen, Reference Sahakian and Owen1992; Cutler et al., Reference Cutler, Shrotriya, Sramek, Veroff, Seifert, Reich and Hironaka1993; Lenehan, Summers, Saunders, Summers, & Vickers, Reference Lenehan, Summers, Saunders, Summers and Vickers2016; and CogState, Fratti, Bowden, & Cook, Reference Fratti, Bowden and Cook2016) have developed a large user-base and substantial validation data, novel assessment platforms continue to be developed. Online, tablet, and smart phone-based test batteries, wearable devices, as well as virtual reality-based cognitive assessment represent only some of the variety of emerging modalities (Brouliette et al., Reference Brouliette, Foil, Fontenot, Correro, Allen, Martin and Keller2013; Parsey & Schmitter-Edgecombe, Reference Parsey and Schmitter-Edgecombe2013; Parsons, Reference Parsons2015). Using these platforms, one innovative application is frequent, at-home serial cognitive assessments over multiple days or weeks. Such sequential ambulatory testing approaches have demonstrated strong reliability, construct validity with gold-standard measures, and enhanced ecologic validity (Sliwinski et al., Reference Sliwinski, Mogle, Hyun, Munoz, Smyth and Lipton2016). Combining both traditional static, comprehensive testing with ambulatory, brief tests may represent a complementary brain measurement approach that is more adept at detecting clinical changes at earlier stages of disease or injury, akin to other medical fields (e.g., cardiology one-time stress test vs. real-time Holter monitor).
Computers can do many things well and automatically, supporting their integration into neuropsychological practices (e.g., follow standardized procedures, time responses, and quickly and accurately score results). These assessment techniques have the potential to provide highly standardized and ecologically valid windows into real-time cognitive and behavioral changes, and could greatly improve accessibility (and costs) for underserved people who may have difficulty traveling to laboratory or healthcare settings. Yet, integration of such technologies is complex, not simply plug-and-play, and there are still major hurdles that must be addressed to ensure competent and valid application of online assessment (e.g., Internet connectivity, or examinee difficulties with comprehension of instructions, inconsistent effort, or distractions). A major consideration is that technology-based assessment tools inherently rely on continually changing technologies; while integration of the latest technology is appealing, any alterations to a validated measure will change its psychometric properties potentially rendering the new version as invalid or at the very least, highly difficult to interpret without a costly process of re-norming. If technology changes are unavoidable after test norming, careful piloting and documentation of any effects on test results are highly recommended and should occur before adopting or deploying the new technology. Technology will undoubtedly play a major role in the future of neuropsychological assessment by enhancing outreach, and potentially standardization and facilitation of diagnosis and early treatment. The development and implementation of such initiatives necessitates involvement of the neuropsychologist with our highly trained psychometric skillsets at the forefront.
Integration into Primary Care and Brain Health Assessment Initiatives
Although our understanding of the (oftentimes modifiable) factors associated with brain health has grown immensely in the past two decades, the systematic dissemination of this information to the public (including other medical fields) has lagged significantly behind. Emerging initiatives are underway to implement screening tools and visualize neurobehavioral data in real-world practices outside of the neuropsychologist’s office. For example, University of California, San Francisco and Quest DiagnosticsTM have partnered to develop the Dementia Care Pathway, a suite of technology-based tools to support navigation, brain health assessment (tablet-based cognitive screener, TabCAT; Possin et al., manuscript submitted for publication), automated analysis, and multimedia educational materials to nonspecialist providers (Rankin et al., manuscript submitted for publication).
Primary care is at the crux of health prevention for other body systems and is ideally positioned for inclusion of the brain. With the development of sensitive, standardized, easy-to-administer and portable screening devices, objective monitoring of brain status will be possible. Continued efforts to translate the important advances our field into real-world diagnostics and treatments are major areas for anticipated growth.
Neuropsychological Assessment Infrastructure
Another significant, under-addressed need in neuropsychology is normative infrastructure. Despite our progress, too often, we do not know how unusual or abnormal a test score or pattern of scores is for a specific group or individual. Of course, all tests must be interpreted in relation to “normal expectations” for the examinee, yet many tests do not have available norms that are appropriate for many types of examinees. Large-scale normative studies would of course help, but these are expensive and funding agencies typically are not supportive of grant funding for such work.
When a governmental funding agency (e.g., U.S. NIH) does fund norming of a nonproprietary set of cognitive tests (e.g., NIH Toolbox), this typically is a one-time effort and the norms may become out of date due to changes in a national census composition. By contrast, proprietary tests may get revised fairly often due to financial motives, and whatever may have been learned about associations between those tests and others (with their different norms) may be rendered tenuous or invalid. This issue is particularly evident in technology-based assessment development, as noted earlier. This of course is a moving target, but our field and its journals should prioritize work in this area, even if it does not have the cachet of a new, experimental approach to assessing a particular disease or brain system. At the very least, we recommend that new studies that use healthy control groups consider and report whether the groups’ results are in the expected range on available norms.
CONCLUSIONS
Neuropsychology has deservedly prided itself as being more empirically based than other domains of professional psychology. But there is still much to learn. Many types of neuropsychological test interpretations are still performed by clinical judgment unaided by empirical guidelines, and many of the questions without empirical guidelines are actually empirical questions that have just not been asked and answered, to date. The evolution from patient observation to highly standardized measures largely relying on only a pencil and paper to obtain a reliable and objective window into brain functioning is resourceful and remarkable, particularly given the relative nascence of our field. As neuropsychology continues to grow, assessment advancements will center on increased ability to detect brain changes at the earliest possible points, likely in conjunction with multimodal markers of neurological function (e.g., neuroimaging and biofluid markers), and dissemination of our understanding of brain–behavior relationships to promote public brain health more broadly.
Acknowledgments
This project was supported by a Larry L. Hillblom Fellowship Award (2017-A-004-FEL) and the HIV Neurobehavioral Research Center (NIMH P30MH062512). The authors have no conflicts of interest to disclose.
