Neuropsychological Assessment: History, Purposes and Advancements
The field of neuropsychological assessment and the proliferation of tests designed to be sensitive to the effects of brain injury began in the 1940s and 1950s (Shum, O’Gorman, Myors, & Creed, Reference Shum, O’Gorman, Myors and Creed2013). Psychologists, approached by other health professionals, developed simple, albeit sensitive, ‘tests of organicity’ to assist in the diagnosis of damage to the brain (Lezak, Howieson, Bigler, & Tranel, Reference Lezak, Howieson, Bigler and Tranel2012; Shum et al., Reference Shum, O’Gorman, Myors and Creed2013). Neuropsychological assessment continued to flourish into the 1970s and 1980s, despite the concurrent development of computerised neuroimaging techniques (e.g., computed tomography (CT), positron emission tomography (PET) and magnetic resonance imaging (MRI)), which advanced the detection and diagnosis of structural brain damage (Shum et al., Reference Shum, O’Gorman, Myors and Creed2013). Since then, the field has been shaped by the work of pioneers such as Dr Muriel Lezak (e.g., Lezak, Reference Lezak1976, Reference Lezak1983, Reference Lezak1995; Lezak et al., Reference Lezak, Howieson, Bigler and Tranel2012; Lezak, Howieson, & Loring, Reference Lezak, Howieson and Loring2004), and in Australia much of contemporary clinical neuropsychology and neuropsychological assessment can be attributed to the work of Dr Kevin Walsh, the founding father of Australian neuropsychology (e.g., Darby & Walsh, Reference Darby and Walsh2005; Walsh, Reference Walsh1978, Reference Walsh1985, Reference Walsh1987, Reference Walsh1991, Reference Walsh1994; Walsh & Darby, Reference Walsh and Darby1999).
Neuropsychological assessment involves ‘the application of neuropsychological tests and other data-collection techniques to answer referral questions or solve problems for individuals with known or suspected brain injury’ (Shum et al., Reference Shum, O’Gorman, Myors and Creed2013, p. 252). As such, it is ‘concerned with identifying the cognitive, emotional, and behavioural consequences of brain dysfunction’ (Gurd, Kischka, & Marshall, Reference Gurd, Kischka and Marshall2010, p. 15). Neuropsychological assessment is conducted for several purposes, including: diagnosis, assessment of neuropsychological strengths and weaknesses, prognosis, patient care and planning, treatment planning and evaluation, monitoring recovery and rehabilitation, and research (Lezak et al., Reference Lezak, Howieson and Loring2004; Shum et al., Reference Shum, O’Gorman, Myors and Creed2013). Neuropsychological assessment differs from psychological assessment in that neuropsychological tests are designed to be sensitive to brain function (Shum et al., Reference Shum, O’Gorman, Myors and Creed2013). Psychological tests are objective procedures for sampling and quantifying human behaviour to make inferences about a particular psychological construct, using standardised stimuli and methods of administration and scoring (Shum et al., Reference Shum, O’Gorman, Myors and Creed2013). Despite differences in purpose and content, neuropsychological and psychological assessment both rely on the same theories, assumptions and techniques. As such, neuropsychological tests retain all the basic characteristics and advantages associated with a psychological test, and are still required to fulfil standard psychometric properties (viz., norms, reliability, validity) before their application can be considered useful (Shum et al., Reference Shum, O’Gorman, Myors and Creed2013).
Neuropsychological assessment has been advanced by concurrent developments in neuroimaging, computerised assessment and ecological validity. Neuropsychological assessment was traditionally carried out to assist in the diagnosis of brain pathology, and to determine the areas of the brain that may have been damaged following injury or illness. However, the advent of neuroimaging techniques resulted in increased accuracy of locating and classifying brain lesions, as well as examining brain function. This advance in technology has contributed to a change of roles for neuropsychological assessment and an increasing shift in the focus of assessment from diagnosis to the assessment of residual strengths and weaknesses following injury/illness and its accompanying deficits, early evidence of disease onset, as well as providing feedback to clients and guiding rehabilitation efforts – aspects of neurological impairment that brain imaging cannot identify or assess (Lezak et al., Reference Lezak, Howieson and Loring2004; Shum et al., Reference Shum, O’Gorman, Myors and Creed2013). The development of computerised assessment has also advanced the field, with tests capable of being presented in multiple formats, though the role of these in clinical practice remains adjunctive rather than central (Lezak et al., Reference Lezak, Howieson and Loring2004). Finally, the ecological validity of tests, or the degree to which test performance corresponds to real-world performance of the measured construct, is becoming an increasingly important issue, especially within neuropsychology (Chaytor & Schmitter-Edgecombe, Reference Chaytor and Schmitter-Edgecombe2003). Two approaches have been identified in addressing the ecological validity of neuropsychological assessment instruments. These include improving the degree to which the cognitive demands of a test theoretically and topographically resemble the cognitive investment required by a task performed in one's everyday environment (verisimilitude), as well as the degree to which a test predicts everyday functioning (veridicality; Chaytor & Schmitter-Edgecombe, Reference Chaytor and Schmitter-Edgecombe2003).
Development of Neuropsychological Tests: Personal Examples
Memory failure is one of the most commonly reported cognitive complaints in people with brain impairment. Memory impairments are persistent and debilitating problems which compromise safety and psychological well-being, and have significant long-term effects for patients, as well as their carers and families (das Nair & Lincoln, Reference das Nair and Lincoln2012). Memory failures in everyday life also impact negatively on one's independence, as well as social and vocational outcomes. Impairments in memory can be difficult to treat. However, the first step in memory rehabilitation is the accurate identification of deficits.
Visual Memory: Shum Visual Learning Test
Impairment in visual memory is common in many neurological conditions and is a predictor of functional and vocational outcomes in individuals with brain injury (Shum, Harris, & O’Gorman, Reference Shum, Harris and O’Gorman2000). However, the development of visual memory and learning tests has lagged behind that of verbal tests. Furthermore, not many visual memory and learning tests were developed using appropriate visual stimuli that are unfamiliar, complex and non-verbalisable. The Shum Visual Learning Test (SVLT; Eadie & Shum, Reference Eadie and Shum1995; Shum, O’Gorman, & Eadie, Reference Shum, O’Gorman and Eadie1999) is a computerised visual memory test comprised of Chinese characters. The test includes three learning trials, an interference trial, a recognition trial and a delayed recognition trial, and automatically produces four results indices: Overall Learning Score (Trial 1 + Trial 2 + Trial 3); Learning (Trial 3 ÷ Trial 1); Retention After Interference (Trial 5 ÷ Trial 3) and Delayed Retention (Trial 6 ÷ Trial 3).
The validity and reliability of the SVLT were assessed by Eadie and Shum (Reference Eadie and Shum1995), who compared Chinese characters with geometric figures as tools for assessing visual memory. In the first study, the 40 Chinese characters used in the SVLT were found to be significantly less verbalisable than 40 geometric figures from the Biber Figure Learning Test. In the second study, memory for Chinese characters, Biber figures and for items on the Rey Auditory Verbal Learning Test, were compared in a sample of 14 participants with left cerebrovascular damage, 15 participants with right cerebrovascular damage and 29 matched controls. The left cerebrovascular damage group showed impairment on the Rey Auditory Verbal Learning Test but not on the Chinese characters, while the right cerebrovascular damage group showed impairment on the Chinese characters but not on the Rey Auditory Verbal Learning Test. This double dissociation was not evident when comparisons involved the Biber Figure Learning Test, indicating the SVLT to be a more valid measure of visual memory. Normative data are also important for fostering the clinical application of a test as they allow neuropsychologists to compare patient performances with those of appropriate age-matched controls. Shum, O’Gorman and Eadie (Reference Shum, O’Gorman and Eadie1999) collected normative data for the SVLT from 146 people aged between 17 and 83 years, and found that age, but not gender and education, affected SVLT test performance. The study provided normative data for seven age groups, and provided evidence of the SVLT's reliability, with significant test–retest reliabilities (rs ranged from .63 to .82).
The development of the SVLT has implications for both clinical practice and research in neuropsychology. Access to such a test in clinical practice will improve the detection and diagnosis of visual memory and learning problems (in non-Chinese speaking patients) with neurological problems. With regards to research, having access to such a test is important for establishing the neuropsychological profile of patient groups with different neuropathological conditions. Examples of the use of the SVLT in research and clinical settings include the study of the effects of severe traumatic brain injury (TBI) on visual memory (Shum, Harris, & O’Gorman, Reference Shum, Harris and O’Gorman2000), as well as investigation of the genes involved in visual memory (Avgan et al., Reference Avgan, Sutherland, Spriggens, Rodriguez-Acevedo, Haupt, Shum and Griffiths2014; Yogeetha et al., Reference Yogeetha, Haupt, McKenzie, Sutherland, Okolicsyani, Lea and Griffiths2013).
Remote Memory: Australian Remote Memory Battery
Remote memory is an aspect of retrograde memory concerned with memory for events and people in the distant past, and comprises both autobiographical memory and public memory (Kopelman, Reference Kopelman2002). Accurate assessment of autobiographical and public memories of the remote past is important for clinical diagnosis and management in health-care systems (Kopelman, Reference Kopelman2002; Lezak et al., Reference Lezak, Howieson and Loring2004). The Australian Remote Memory Battery (ARMB) was developed for the assessment of remote memory within the Australian context, specifically public memory for the distant past for long-term residents of Australia (Shum & O’Gorman, Reference Shum and O’Gorman2001; O’Gorman & Shum, Reference O’Gorman and Shum2012). The test includes a famous faces subtest (54 photographs of famous faces, e.g., Betty Cuthbert, Eddie Charlton, Cheryl Kernot, Martin Bryant) and a public events subtest (54 public-event questions, e.g., Who was Cliff Young? Which English colony was handed back to China in the 1990s? What happened on the 14th of February 1966?), spanning the period of the 1930s to 1990s, and assesses the ability to recall and recognise famous people and events remote from the present (Shum & O’Gorman, Reference Shum and O’Gorman2001). In terms of its psychometric properties, the ARMB has demonstrated the expected memory effect for valid tests of remote memory, with better recall and recognition shown for more recent than remote famous events and faces (Shum & O’Gorman, Reference Shum and O’Gorman2001). The ARMB has also demonstrated the expected age effect for valid tests of remote memory, in that the memory effect interacts with participants’ age, with older participants demonstrating better memory for the remote past, compared to middle-aged and younger participants (Shum & O’Gorman, Reference Shum and O’Gorman2001).
The intended uses of the ARMB were to assist neuropsychologists working with Australian patients in the diagnosis, assessment and management of patients with Alzheimer's disease, Korsakoff's syndrome or temporal lobe damage, as well as to contribute to and advance the study of retrograde amnesia through the testing of hypotheses derived from models of memory (Shum & O’Gorman, Reference Shum and O’Gorman2001). As such, the ARMB has been used in clinical and research settings, including the assessment of remote memory in individuals with Alzheimer's disease and Korsakoff's syndrome (Shum & O’Gorman, Reference Shum and O’Gorman2001), the assessment of retrograde memory following temporal lobe lesion (Lah & Miller, Reference Lah and Miller2008) and temporal lobectomy (Lah, Grayson, Lee, & Miller, Reference Lah, Grayson, Lee and Miller2004; Lah, Lee, Grayson, & Miller, Reference Lah, Lee, Grayson and Miller2008), as well as the assessment of retrograde memory in stroke patients with unilateral (frontal or temporal) lesions (Batchelor, Thompson, & Miller, Reference Batchelor, Thompson and Miller2008). The ARMB has also been found to be sensitive to other disease processes, including herpes simplex (Geffen, Isles, Preece, & Geffen, Reference Geffen, Isles, Preece and Geffen2008) and temporal lobe epilepsy (Lah, Lee, Grayson, & Miller, Reference Lah, Lee, Grayson and Miller2006). The ARMB is now almost 15 years old and tests of this nature are susceptible to ageing and obsolescence as the store of public knowledge changes (Lah & Miller, Reference Lah and Miller2008). The work conducted by our research team to date encourages further revision of the test and the development of more comprehensive norms so that its usefulness for practitioners and researchers is maintained into the future (O’Gorman & Shum, Reference O’Gorman and Shum2012).
Prospective Memory
Successfully remembering to perform an intended action in the future is a critical cognitive skill underlying our ability to complete many real-world activities in our daily lives (Ellis & Kvavilashvili, Reference Ellis and Kvavilashvili2000). We are constantly making mental notes of tasks we need to complete at a particular time in the future (e.g., to attend an appointment, to pick the children up from school, to take medication or to pay a bill). This type of memory is known as prospective memory (PM), which refers to the ability to remember to carry out an intended action at an appropriate point in the future (McDaniel & Einstein, Reference McDaniel and Einstein2007). Other characteristics of PM include: the intention cannot be carried out at the time of encoding, the delay between intention and action must be filled with an ongoing task, there is no explicit prompt or reminder by others, and the ongoing task needs to be interrupted in order to carry out the intended task. PM plays an important role in independent functioning, and is crucial for the fulfilment of instrumental activities in daily life, health needs and social relations (Kliegel, McDaniel, & Einstein, Reference Kliegel, McDaniel and Einstein2008). However, despite its relevance to everyday functioning, PM is a comparatively new memory construct.
There are three types of PM. The first type occurs when the intended action is linked to a specific time (e.g., at 7 a.m. in the morning) or after a period of elapsed time (e.g., after 3 hours), and is known as time-based PM. Event-based is the second type of PM and refers to when the intended action is linked to a specific cue (e.g., posting a letter when one walks past a mailbox). Alternatively, the intended action may be linked to the end of a specific activity (e.g., taking medication after a meal), in which case it is termed activity-based PM. Further, Ellis (Reference Ellis, Brandimonte, Einstein and McDaniel1996) has described five phases of PM: (1) formation and encoding of intention and action; (2) retention interval; (3) performance interval; (4) initiation and execution of intended action; and (5) evaluation of outcome. These phases are used in research to identify which component of the PM process people experience deficits in, whether this is different across populations and how interventions can be developed to assist with these impairments.
PM impairment, or the inability to remember to fulfil delayed intentions, is a very common problem that has been documented in many clinical populations, including TBI, stroke, Parkinson's disease, multiple sclerosis, HIV infection, herpes simplex encephalitis, affective disorders and attention deficit hyperactivity disorder (ADHD) (Kliegel, Jäger, Altgassen, & Shum, Reference Kliegel, Jäger, Altgassen, Shum, Kliegel, McDaniel and Einstein2008; Shum, Levin, & Chan, Reference Shum, Levin and Chan2011). PM failures have been shown to have a greater impact on the lives of carers than retrospective memory failures (Smith, Della Sala, Logie, & Maylor, Reference Smith, Della Sala, Logie and Maylor2000), and have also been shown to cause more stress and concern than retrospective memory failures (Ward, Shum, Dick, McKinlay, & Baker-Tweeney, Reference Ward, Shum, Dick, McKinlay and Baker-Tweeney2004). However, difficulty arises in developing tests of PM that are representative of real-world examples and, hence, the ecological validity of many PM tasks to date is inadequate.
Comprehensive assessment of prospective memory
The Comprehensive Assessment of Prospective Memory (CAPM; Shum & Fleming, Reference Shum and Fleming2014) is a self-report questionnaire of PM ability. The CAPM comprises three scales, which collect comprehensive data on the process of prospective remembering, capturing the: frequency of PM failures in activities of daily living (Part A); perceived importance of PM failures in activities of daily living (Part B); and reasons for remembering or forgetting to carry out intended actions (i.e., why people think they remember or forget; Part C). Reliability and normative data for the CAPM have been reported by Chau, Lee, Fleming, Roche and Shum (Reference Chau, Lee, Fleming, Roche and Shum2007). In this study, 95 people aged between 15 and 60 years, living in the community, and with no history of brain injury, were recruited. For the norms, a significant difference was found between the 15- to 30-year-old and the 31- to 60-year-old age groups, and the CAPM demonstrated good levels of internal consistency (Cronbach's α ranged from .74 to .94) and test–retest reliability (rs ranged from .74 to .77). Criterion validity and concurrent validity have also been demonstrated for the CAPM in a sample of 45 adults with TBI and their relatives (Fleming et al., Reference Fleming, Kennedy, Fisher, Gill, Gullo and Shum2009). Participant self-reports and relative reports on the CAPM significantly correlated with level of psychosocial integration, supporting criterion validity, while relative reports also correlated significantly with participants’ scores on the Cambridge Prospective Memory Test (CAM-PROMT) and the Memory Intentions Screening Test (MIST), supporting concurrent validity.
The CAPM was ‘designed to assist in the clinical evaluation of prospective memory function among individuals with acquired brain injury, and to assist in the evaluation of rehabilitation programs designed to serve these people’ (Shum & Fleming, Reference Shum and Fleming2014, p. 4). The CAPM can therefore be used in both clinical and research settings to, for example, assess or screen for memory impairments, or plan treatment, due to its easy administration and ability to provide information from both the client's perspective and the perspective of a significant other. The CAPM has been used in research and clinical settings as an outcome measure in intervention studies (Fleming, Shum, Strong, & Lightbody, Reference Fleming, Shum, Strong and Lightbody2005; Shum, Fleming, Gill, Gullo, & Strong, Reference Shum, Fleming, Gill, Gullo and Strong2011), in research documenting the frequency and impact of memory lapses (Chau et al., Reference Chau, Lee, Fleming, Roche and Shum2007), as well as in research examining self-awareness of memory deficits (Roche, Fleming, & Shum, Reference Roche, Fleming and Shum2002), perceived importance of and perceived reasons for PM failures in adults with TBI (Huang et al., Reference Huang, Fleming, Pomery, O’Gorman, Chan and Shum2014; Roche, Moody, Szabo, Fleming, & Shum, Reference Roche, Moody, Szabo, Fleming and Shum2007), and PM function in healthy Chinese people (Chan, Qing, Wu, & Shum, Reference Chan, Qing, Wu and Shum2010). Our research team has since developed the Brief Assessment of Prospective Memory, also for use with TBI samples (Man, Fleming, Hohaus, & Shum, Reference Man, Fleming, Hohaus and Shum2011).
Virtual reality shopping task
The Virtual Reality Shopping Task (VRST) is a computerised measure of time- and event-based PM (Canty et al., Reference Canty, Fleming, Patterson, Green, Man and Shum2014). It consists of an ongoing task where the participant manipulates their avatar within a virtual shopping centre to achieve certain tasks, while they are also required to complete time-based (e.g., to send a text message at a certain time) and event-based (e.g., press the ‘T’ key when you hear the word ‘sale’) PM tasks. As such, the VRST offers an ecologically valid medium through which to assess PM impairment.
The intended use of the VRST was to assist neuropsychologists to determine whether patients have a normal capacity to remember a simple set of intentions and, if not, help to identify their functional limitations. The VRST focuses on behavioural responses rather than subjective data, thus providing a bridge between conventional neuropsychological tests, behavioural observation and the logistical parameters of in-patient settings. Furthermore, the VRST is more likely to be readily accepted by test-takers and their families because of its inherent novelty and face validity (Canty et al., Reference Canty, Fleming, Patterson, Green, Man and Shum2014).
The VRST has been used in research to assess the PM performance of individuals with severe TBI (Canty et al., Reference Canty, Fleming, Patterson, Green, Man and Shum2014), with this study also demonstrating the sensitivity, convergent validity and ecological validity of the VRST. Performance of the TBI group was significantly worse than that of controls on event-based PM, as measured by a lexical decision PM task (LDPMT), and on time- and event-based PM as measured by the VRST, providing support for the sensitivity of the measure. For the TBI group, event- and total PM performance on the VRST significantly correlated with performance on measures of mental flexibility and verbal fluency, and, additionally, total PM performance correlated with verbal memory, together demonstrating convergent validity. Furthermore, significant correlations were observed between PM performance, as measured by the VRST, and others’ ratings of patients’ occupational activities and independent living skills, evidencing veridicality. The VRST was also rated as significantly more reflective of an everyday activity, interesting, and was afforded a higher recommendation than the LDPMT, evidencing verisimilitude.
Personal Experience and Lessons Learnt
In developing these tests, the author has accumulated some personal experiences and learned a number of lessons. First, before starting test development, it is important to be clear and sure that there is a need to develop another neuropsychological test. This is because doing so requires time, commitment and resources. Second, the test to be developed needs to be firmly based on a strong and recent theoretical framework. Failing to do so may mean the development of an outdated test that neuropsychologists are not interested in using. Third, there is usually a gap between the theoretical and operational definitions of the neuropsychological construct the test intends to assess. While it is easy for one to say what one wants to measure, it is another thing to put that into practice. Much work is needed to clearly define a neuropsychological construct operationally and to specify the characteristics of that construct. Fourth, the set of test items included in a published test do not come about easily. A test developer usually comes up with a much larger set of test items and undertakes a series of item analyses to select and validate a smaller but good set of items. Fifth, test validation is not a short, one-off process. Rather, it takes time for evidence supporting a test's validity to accumulate. This again reinforces the idea that development of a new test is time and resource intensive. Sixth, for a test to be adopted in clinical practice, suitable and sizeable norms are needed. However, the collection of normative data from an appropriate and well-represented group takes time, effort and resources. In Australia, it is the author's experience that funding bodies do not usually provide grants for developing test norms. Seventh, the utility of a test does not last forever. A test can become obsolete because of reasons such as technological or theoretical developments, or norms being outdated and inappropriate. Eighth, the development of a new test does not guarantee its adoption by other researchers and clinicians. A test developer should not be complacent just because good psychometric evidence has been found. Rather, much work is needed to promote the test by publishing it in an appropriate journal, presenting its utility at conferences and workshops, or by finding a good publisher.
Conclusion
This paper has briefly reviewed the history and purposes of neuropsychological assessment, as well as advancements in this area, and has discussed the development of neuropsychological tests using examples developed by the author and his colleagues. The paper has highlighted the importance of psychometric properties in the development of neuropsychological tests (viz., normative data, validity and reliability), as well as more recent efforts to improve the ecological validity of tests.
