NNN NEUROPSYCHOLOGICAL TEST SELECTION

Data are obtained from the most frequently administered neuropsychological tests and are representative of national assessment trends (Rabin, Paolillo, & Barr, Reference Rabin, Paolillo and Barr2016). The clinical sites (Emory University, Medical College of Wisconsin, University of Florida, and UCLA) were selected since they represent geographic diversity, are nationally recognized programs, involve multiple board-certified clinical faculty, and have established track records of collaborative clinical research. Several NNN investigators have also been involved in formal test development initiatives, learning the difficult lesson that funding, infrastructure, and dedicated/protected research time are essential elements for project success. These institutions provide neuropsychology training at the practicum, internship, doctoral, and postdoctoral levels and are well-positioned to influence practices and expectations of emerging neuropsychologists. Clinical sites are expected to enroll more than 10,000 participants spanning diverse neuropsychiatric and neurologic diseases/syndromes and to compile item-level data on 500+ clinical cases for nearly 50 common neuropsychological measures during its 5-year NIMH funding period.

The NNN established a formal collaborative relationship with Pearson Clinical Assessment, the publisher of many of the most widely used NP tests (Rabin et al., Reference Rabin, Paolillo and Barr2016). The NNN leadership plan includes Pearson in all project discussions although Pearson does not have voting rights in the governing board, which consists of leaders of the four clinical sites and an independent external advisor (Robert Heaton). The Pearson Q-i platform captures item-level responses on Pearson tests including measures of general cognition (Wechsler Adult Intelligence Scale-Fourth Edition, WAIS-IV), memory (Wechsler Memory Scale-Fourth Edition; California Verbal Learning Test-Third Edition), executive function (Delis-Kaplan Executive Function System), and brief neuropsychological assessment screening (Repeatable Battery for the Assessment of Neuropsychological Status). The Q-i platform employs iPads for both stimulus presentation and examiner recording of responses.

Because non-Pearson tests (e.g., Boston Naming Test) lacked a point-of-testing data entry platform, the UCLA Semel Institute Biostatistics Core (SI-Stat) developed web-based data capture applications for these measures (SAILOR - System for Acquisition of Item Level and Observational Responses). Agreements to program data-entry applications have been executed with test publishers permitting appropriate per-use royalty payment for clinical test usage. Pearson Clinical Assessment is providing Q-i use for the NNN initiative as part of the research collaboration with support from NIMH. Neuropsychological tests included in NNN are listed in Table 1.

Table 1.

Measures being obtained as part of the National Neuropsychology Network

* Selected RBANS subtests administered after TeleNP implementation due to COVID-19 pandemic.

STRUCTURED HISTORY PROTOCOL FOR NEUROPSYCHOLOGY AND OTHER COMMON DATA ELEMENTS

The Structured History Protocol for Neuropsychology (SHiP-NP) is a standardized history protocol developed to harmonize clinical data collection across NNN sites including demographics, medication use, and medical/psychiatric history to facilitate data analysis. Demographic data elements are based on the PhenX Toolkit project (Hamilton et al., Reference Hamilton, Strader, Pratt, Maiese, Hendershot, Kwok and Haines2011), which developed consensus measures for “Phenotypes and eXposures” that include age, race, ethnicity, sex, gender, marital status, educational attainment, annual family income, and child-reported parental educational attainment. Medical history, family history, and concomitant medications are recorded based on the NINDS Common Data Elements (CDE) conventions (Grinnon et al., Reference Grinnon, Miller, Marler, Lu, Stout, Odenkirchen and Kunitz2012), and medications associated with increased risks for adverse cognitive effects (e.g., narcotics, benzodiazepines, anticholinergics, sedatives/hypnotics, and selected anti-seizure medications) are highlighted. The SHiP-NP includes a standardized assessment of potential influences on neuropsychological performance including developmental history, academic performance, legal and military history, mental health treatment, and social health determinants (e.g., financial strain). Appropriate follow-up questions are available for specific medical conditions (e.g., epilepsy, traumatic brain injury, stroke, cancer) to characterize symptom presentation, duration, and treatment history. In response to the COVID-19 pandemic, novel coronavirus exposure and associated distress queries were added that were obtained from the Montreal Behavioral Medicine Centre (https://mbmc-cmcm.ca/covid19/; Lavoie & Bacon, Reference Lavoie and Bacon2020). Additional clinical CDEs are linked to the SHiP-NP (see section below: Clinical Interview).

The SHiP-NP is designed for online completion prior to the clinical appointment via a secure website hosted by SI-Stat, although a paper and pencil SHiP-NP version can be used by patients without internet access. The SHiP-NP relies on branching logic and forced response options to gather relevant information while minimizing overall assessment burden. SHiP-NP data are collected and stored securely on a secure UCLA Si-Stat server without Protected Health Information (PHI), which is available only to patients and their clinicians. The SHiP-NP generates patient data sheets and narrative history text files that can be modified for each clinical site and which are easily integrated into clinical reports. For patients unwilling or unable to complete the SHiP-NP, NNN sites collect a minimal demographic dataset including age, gender, education, and handedness, with secondary questions documenting marital status, employment, and languages spoken other than English. The first 135 patients completing the SHiP-NP spent approximately 22 min completing the form (Mdn = 22.4 min; range = 4.4–56.0 min).

MILESTONES ACHIEVED DURING THE INITIAL PROJECT PERIOD

The initial project months (March 2019–August 2019) were dedicated to logistical problem-solving associated with collaboration across sites using different models of clinical service delivery and addressed training and quality control, data capture, and options for obtaining participants’ consent. Although NNN uses the SMART IRB National IRB reliance system, it was still necessary to obtain agreement from site IRBs prior to beginning subject enrollment. The Q-i platform allows individual item data capture for Pearson tests; however, it was necessary during project start-up to develop individual item data capture for non-Q-i measures and to establish appropriate mechanisms for data transfer to the NDA.

INSTITUTIONAL REVIEW BOARD

UCLA serves as the IRB of Record, with all participating sites relying on the SMART IRB agreement (NIH’s National Center for Advancing Translational Sciences (NCATS) Streamlined, Multi-site, Accelerated Resource for Trials (SMART) IRB Reliance Platform). Future NNN sites will be able to sign on to SMART IRB and join this agreement, and if ineligible for SMART IRB, an alternative appropriate IRB agreement will be negotiated. This process coordinates, collects, and verifies information including: (a) local context; (b) site variations in areas such as recruiting, informed consent, HIPAA, populations; (c) conflict of interest disclosure and management; (d) completion of ancillary reviews; (e) training and qualifications of study team; (f) continuing review or closure information; and (g) reportable events such as protocol deviations or adverse reactions. In response to the 2020 COVID-19 outbreak and the emergence of telehealth, NNN was first granted IRB approval to obtain verbal informed consent for study participation. Given the negligible risks to patients of participation in the NNN, that clinical practice remains unchanged, and that no PHI is included, we applied and received approval for a waiver of informed consent from the UCLA IRB. With consent waiver, participants are assigned a random number (i.e., pseudo-GUID) to identify subjects as distinct study participants but which avoids use of PHI for data sharing with Pearson Clinical Assessment, UCLA, and the NDA.

RECRUITMENT

All adult English-speaking patients referred for neuropsychological evaluation are considered potential NNN participants. We initially aimed to enroll approximately 50 cases weekly across our network to reach 10,000 cases total in the database. Soon after starting enrollment on 31 July, 2019, we reached this weekly enrollment target. We experienced a sudden drop in recruitment in March 2020 at the onset of the COVID-19 pandemic, but our clinics have subsequently developed new models of practice and we have again returned to target recruitment levels, expecting to further increase recruitment (see Supplemental Figures 1a and 1b). Participant characteristics are provided in Supplemental Table 1.

TRAINING AND QUALITY CONTROL

Each site has a quality control officer, and each person performing NNN assessments is certified at both the test administration and Q-i assessment interface levels. The NNN has a library of training materials, with support from Pearson Clinical Assessment for Q-i training tools, and the NNN maintains a database documenting training outcomes for all personnel involved in data collection.

DATA ACQUISITION, STORAGE, AND TRANSFER

NNN protocols transmit only data without PHI to Pearson Clinical Assessment, and data received back from Pearson or SI-Stat are based upon assigned study ID. Thus, each site is responsible for creating linking tables to connect NNN study number to other identifiers in their own clinical records. Q-i data transmitted to Pearson Clinical Assessments are tagged based upon site-specific logins with deidentified patient data, which are also how data are transferred and stored in the Pearson database and subsequently shared with the NDA. Pearson executes their usual workflow to score the obtained information, creating data files for the sending site as is their usual reporting standard on Q-i. In parallel, Pearson sends complete data, including all trial timing and individual response selection variables, to the NDA. The test results and normative information generated by Pearson are provided back to the clinic sites in an Excel spreadsheet which facilitates preparation of the formal clinical report, a process that typically takes less than 30 min after data transmission to Pearson. UCLA Si-Stat is responsible for data aggregation with non-Q-i measures and for transfer to the NDA. We continue to resolve issues related to the highly specific data elements that are enabled by the Q-i outputs, which include not only item-by-item scores and timings but also individual examiner annotations as image files (e.g., locations of each block in individual block designs). The complete data dictionary will be accessible on our website and from the NDA.

CLINICAL INTERVIEW

The NNN has adopted conventions for data collection for contemporary diagnostic and clinical status information while allowing individual clinicians to conduct interviews following their current practices. While diagnostic assessments will vary, specification of both pre-assessment and post-assessment diagnoses follows ICD-10-CM and is harmonized with the Diagnostic and Statistical Manual of Mental Disorders, 5^th Edition (DSM-5) codes for psychiatric disorders. The same online platform developed for the SHiP-NP is used to acquire other CDEs including multiple dimensional self-report rating scales. These CDEs were recommended by an NIMH workgroup (Barch et al., Reference Barch, Gotlib, Bilder, Pine, Smoller, Brown and Farber2016) and include the DSM-5 Level 1 Cross-Cutting Symptom measure, a 23-item self-report that assesses 13 domains (depression, irritability, anxiety, mania, somatic symptoms, suicidality, psychotic symptoms, insomnia, memory problems, compulsions, derealization/depersonalization, personality functioning, and substance use disorders) (Available at: https://www.psychiatry.org/psychiatrists/practice/dsm/educational-resources/assessment-measures). Participants who screen positive on the Level 1 assessment also receive the relevant DSM-5 Level 2 measures, which include the Patient-Reported Outcomes Measurement Information System (PROMIS) short-forms for Depression, Anxiety, Anger, and Sleep Problems and self-report ratings of Mania, Substance Use, Repetitive Thoughts and Behaviors, and Somatic Symptoms. Finally, there is a Clinician-Rated Dimensions of Psychosis Symptom Severity because this domain was determined to be unreliable by self-report. Many patients with neurologic referral diagnoses are not expected to screen positive on the Level 1 assessment, although this screening will permit detailed characterization of psychiatric comorbidities in chronic neurologic diseases such as epilepsy or multiple sclerosis or degenerative conditions such as Alzheimer’s disease or Parkinson’s disease.

DIMENSIONAL RATINGS OF EVERYDAY FUNCTIONING AND DISABILITY

Disability ratings are provided by the 36-item self- and informant-reported World Health Organization Disability Assessment Schedule 2.0 (WHO-DAS 2.0; available at: https://www.who.int/classifications/icf/whodasdownloads/en/), which are also incorporated as part of the SHiP-NP/CDE online platform. The WHODAS 2.0 follows the theoretical framework of the International Classification of Functioning, Disability and Health (ICF), permitting it to be used worldwide across all health conditions. Additional ratings of current functioning and quality of life for non-overlapping domains are gathered using validated short forms of the Quality of Life in Neurological Disorder (Neuro-QoL) battery (Cella et al., Reference Cella, Lai, Nowinski, Victorson, Peterman, Miller and Moy2012), comprising a total of 65 additional items for Emotional and Behavioral Dyscontrol; Fatigue; Lower Extremity Function – Mobility; Positive Affect and Well-Being; Satisfaction with Social Roles and Activities; Sleep Disturbance; Stigma; and Upper Extremity Function – Fine Motor, Activities of Daily Living (ADL). These are all programmed in the SHiP-NP/CDE web application, with data collected either prior to each participant’s visit via their own web-enabled devices or on site using one of the sites’ iPads or other data entry devices.

DATA ANALYSES

Initial NNN analyses will apply item response theory (IRT) to define more precisely the constructs measured by each test and how these constructs can be assessed more efficiently. Most IRT models are expected to be unidimensional, although multidimensional IRT (mIRT) models will be explored as appropriate. Following dimensionality determination, efficiency optimization will be explored by specifying fixed short forms, computerized adaptive tests (CATs), or multidimensional adaptive tests. IRT application has led to efficiency gains of 50–95% relative to conventional method for many measures (Choi, Reise, Pilkonis, Hays, & Cella, Reference Choi, Reise, Pilkonis, Hays and Cella2010; Gibbons et al., Reference Gibbons, Weiss, Kupfer, Frank, Fagiolini, Grochocinski and Immekus2008; Moore et al., Reference Moore, Scott, Reise, Port, Jackson, Ruparel and Gur2015). IRT can also characterize the same construct using different item sets, which is particularly useful in longitudinal assessments to ensure that inclusion of a newly derived measure is back-compatible with earlier test versions that may have been used in previous testing, and to enable repeated measurement without using previously exposed items.

As a proof of principle demonstration that modern psychometric methods applied to existing measures can improve test efficiency, we applied IRT to WAIS-IV Matrix Reasoning (MR) data in a group of 549 NNN participants (Reise et al., Reference Reise, Widaman, Bauer, Drane, Loring, Umfleet and Bilder2021). The mean MR raw score was 15.4 (SD = 5.6) and the mean MR scaled score was 9.9 (SD = 3.2), similar to the standardization sample. The first five MR items were completed without error by 97% of the subjects, adding little to the measurement of the MR latent trait. The most difficult MR items also contributed little information at ability levels greater than 1.5 SD above the test mean. Finally, there was considerable overlap in item information across the remaining MR items suggesting that a short form or adaptive version of MR could provide results with precision comparable to the full MR subtest. Using the standard administration start and stop rules, more than half of our sample (282/549 or 51%) were administered 23/26 MR items to obtain their total score. In contrast, a simulated CAT with only 10 MR items was almost perfectly correlated with the ability level estimated from the entire 26-item test (r = .99). This represents more than a doubling of assessment efficiency.

These analyses can be further extended using the nominal or graded response models. In brief, the traditional analyses of the MR responses use dichotomization, considering only whether each item was completed correctly or incorrectly. But because each response is chosen from a multiple-choice array that includes five response options, each distractor can be considered independently as an indicator of ability on either the primary latent trait or on other factors that might be identified following data analysis. In the nominal response model, each response is considered independently, while in the graded response model, the response alternatives can be ranked on an ordinal scale from those that are “closest” to those that are most “distant” from the correct choice. For example, selection of a “close” response option that matches on some but not all item dimensions might be more frequent among those with higher ability, while selection of a “distant” option that does not follow the expected dimensions might be more frequent among those with lower ability or who may be following an unusual response strategy. This approach to response analysis may have special value in performance validity testing in which selection of particularly “distant” choices might reveal a marked deviation from the best estimates of that individual’s true abilities suggesting possible intentional response deficit exaggeration. Indeed, this method examining the relationship of individual responses to estimates of true ability (i.e., person-fit statistics) should allow the design of empirically derived embedded performance validity tests for many measures.

As part of quality control, data analyses will exam measurement invariance in an IRT framework to determine whether putatively identical tests behave comparably across sites, diagnostic boundaries, and groups defined by demographic characteristics. These analyses will establish to what extent data can be combined for subsequent analyses and to what extent unique scoring and interpretation may be indicated for different groups defined by age, sex, education, racial/ethnic, linguistic, or cultural backgrounds. Novel methods for variable harmonization developed for the Whole Genome Sequencing in Psychiatric Disorders consortium (U01 MH105578) will be employed to combine data that were acquired using different instruments in different samples (with no overlap of items across samples and no patient receiving both instruments) (Mansolf et al., Reference Mansolf, Vreeker, Reise, Freimer, Glahn and Gur2020). This method analyzes the goodness-of-fit of correlations among variables to determine if two different variables are sufficiently similar to include in pooled analyses. Items are selected that best match items in another test based on the strengths of their correlations with all the other items in their respective test. When the loss function asymptotes, this process stops yielding the items that are considered equivalent across datasets.

Data analyses will include both confirmatory and exploratory factor analytic approaches. The degree to which alternate models may be more appropriate to our clinical samples, or specified using different tests, will be rigorously explored and analyses will determine to what extent the additional tests measures add to or modify that structure. Alternate models using confirmatory factor analysis will establish how various models differ with respect to efficiency (i.e., assessment time) by determining the degree to which goodness of fit deteriorates with fewer variables or by substituting short-form test scores for long-form results. This permits the creation of short-form tests, new short-form batteries, and CATs to increase efficiency to measure the same constructs or arrive at the same diagnostic conclusions. We will also characterize short-form use across different clinical diagnostic referrals since, for example, a short-form assessment of confrontation naming may be appropriate in patients with referral diagnoses of multiple sclerosis but not for epilepsy patients referred for surgical evaluation. Thus, NNN will provide the evidence base that enables multidimensional adaptive testing to maximize diagnostic information while minimizing burden to patients and facilitating new large-scale collaborative research projects (e.g., as in genomics and population behavioral health).

CLINICAL VALIDATION OF NEUROPSYCHOLOGICAL PROCEDURES

By employing adaptive assessment techniques, assessment protocols may eventually be personalized according to referral diagnoses and patient performance patterns in which both item and test selection maximize the predictive power by sequentially selecting the most informative next item within a test and the most informative next test within a battery. When specific test findings are obtained, prior probability estimates are updated, and the next most informative measure is selected. This process continues until desired levels of precision are obtained with respect to the outcomes of interest. With adaptive testing, assessment protocols are not administered in the same pre-determined sequence, but rather change to the specific clinical context and adjust dynamically according to task performance.

The combination of tests, administration order, and all available sources of information are included to establish the best combination for accurate prediction, whether it will be post-assessment diagnosis or risk of adverse treatment outcome. These analyses will generally use multinomial logistic regression (MLR) models if there are more than two categories, or logistic regression for cases with only two classes.

SUMMARY

The NNN will provide a major resource to advance understanding of neurocognitive function through refined neurocognitive assessment. NNN is designed to establish a strong foundation capable of engaging neuropsychologists across a variety of practices. By contributing neuropsychological findings to the NDA, we aim to stimulate development of the next generation of evidence-based assessments. The NNN will provide foundational data to establish incremental validity of current, reconfigured, and ultimately novel neuropsychological measures for characterizing our emerging understanding of functional systems in the brain and real-world outcomes. We hope the ultimate result will be marked improvements in our ability to assess brain functions, leading to both improved understanding and treatment of neuropsychological impairment and mental illness in the USA. By enhancing the efficiency of assessment and directly addressing the problems of measurement invariance that have long plagued neuropsychological assessment, we anticipate that NNN will promote the development of methods that improve access globally and reduce current inequities in neuropsychological service delivery.

Our vision is to create a platform that has a foundation forged from “classic” NP tests with widespread use in the NP community, and to build upon this foundation a new generation of tasks that may incorporate a range of current technologies to explore modern theories of brain structure–function relationships. By supplementing current test batteries with additional procedures to examine concurrent and predictive validity of newer measures, we believe the field can benefit from both back-compatibility with established standards and future-directed extensions that will be more efficient and possess greater utility than the current methods. The new methods may include the application of virtual and augmented reality, “the internet of things,” videography, wearables, and eye tracking, among other current and anticipated innovations – all of which may enhance ecological validity relative to current methods. New assessment methods combined with multimodal neuroimaging or electrophysiological techniques (e.g., cortico-cortical evoked potentials in the context of stereoelectroencephalography) and advanced computational processing (e.g., artificial intelligence) may further hold the promise of moving assessments beyond neuropsychological “domains” (e.g., motor, visuo-perceptual, and language processes such as word retrieval) to a deeper understanding of complex neural circuits (e.g., distributed neural processing, connectivity metrics) (Fox & Friston, Reference Fox and Friston2012; Gonzalez-Castillo & Bandettini, Reference Gonzalez-Castillo and Bandettini2015). These developments will ultimately be needed if we hope to understand major unanswered questions such as those that center on the complexities of memory (e.g., how novel information is integrated into a sense of autobiographical self and integrated semantic knowledge over time and space), consciousness and its distortions (e.g., déjà vu, jamais vu, reduplicative paramnesia), socio-emotional functions, and default mode processing.

To facilitate engagement of the larger neuropsychology community, neuropsychologists are encouraged to register at the NNN website https://www.sistat.ucla.edu/NNNWeb/index.html, by providing their name, email address, phone number, and organization with which interested person is affiliated. The NNN will provide registered members of the neuropsychology community with progress updates, invitations to provide feedback on new initiatives, and access to data, assessment tools, and algorithms developed by the NNN. We also hope soon to be able to extend registration of clinical patients with the NNN so that the network can grow beyond the current proof-of-concept to include multiple sites that can contribute data and benefit from sharing novel normative and clinical validity data that are rapidly being aggregated by the network nationwide. The mature NNN will help assure that clinical neuropsychology evolves to include the most accessible, efficient, and valid methods for the assessment of brain–behavior relations.