INTRODUCTION
Cognitive impairment is a feature of many diseases and injuries affecting the central nervous system. Rapid identification of impairment is often critical for diagnostic formulation as well as to inform treatment and case management. Brief cognitive screening may provide an efficient method of detection and triage to guide referral for more comprehensive investigations, such as neuropsychological assessment, as part of a stepped-care approach (Block, Johnson-Greene, Pliskin, & Boake, Reference Block, Johnson-Greene, Pliskin and Boake2017; Lanca, Reference Lanca2018). Effective cognitive screening may also be used to document and monitor difficulties when comprehensive assessments are not available or possible, such as in rural settings or when there are competing medical demands (Bryce & Allott, Reference Allott, van-der-EL, Bryce, Hamilton, Adams, Burgat and Rickwood2019; Sheehan, Reference Sheehan2012). The use of cognitive screening is already well-established in primary, tertiary, and community settings in the assessment of health conditions in adults, including mild cognitive impairment, dementia, acquired brain injury or disease, and psychiatric illness (Roebuck-Spencer et al., Reference Roebuck-Spencer, Glen, Puente, Denney, Ruff, Hostetter and Bianchini2017). Critically, cognitive screening has to have demonstrated high diagnostic sensitivity to identify those in need of further assessment (Summers, Bondi, & Bowden, Reference Summers, Bondi and Bowden2019). Screening sensitivity is enhanced when cognitive impairment is corroborated by observable signs or symptoms, or when family or other caregivers express concern over decline (Laver et al., Reference Laver, Cumming, Dyer, Agar, Anstey, Beattie and Yates2016; Pottie et al., Reference Pottie, Rahal, Jaramillo, Birtwhistle, Thombs, Singh and Tonelli2016). Classic cognitive screening tools used in older adults with suspected neurological illness include the Mini-Mental State Examination (MMSE; Folstein, Folstein, & McHugh, Reference Folstein, Folstein and McHugh1975) and Montreal Cognitive Assessment (MoCA; Nasreddine et al., Reference Nasreddine, Phillips, Bedirian, Charbonneau, Whitehead, Collin and Chertkow2005). The use of these measures in adults has been reviewed elsewhere (Arevalo-Rodriguez et al., Reference Arevalo-Rodriguez, Smailagic, Roquéi Figuls, Ciapponi, Sanchez-Perez, Giannakou and Cullum2015; Cullen, O’Neill, Evans, Coen, & Lawlor, Reference Cullen, O’Neill, Evans, Coen and Lawlor2007; Davis et al., Reference Davis, Creavin, Yip, Noel-Storr, Brayne and Cullum2015; Holsinger, Deveau, Boustani, & Williams Jr, Reference Holsinger, Deveau, Boustani and Williams2007; Tsoi, Chan, Hirai, Wong, & Kwok, Reference Tsoi, Chan, Hirai, Wong and Kwok2015). Notably, there is variability in the risk of bias in reviewed studies, indicating the potential for inflated estimates of diagnostic validity (e.g., Tsoi et al., Reference Tsoi, Chan, Hirai, Wong and Kwok2015).
In contrast to adults, cognitive screening in youth, aged 12–25, is not routinely conducted in clinical care (Bryce & Allott, Reference Allott, van-der-EL, Bryce, Hamilton, Adams, Burgat and Rickwood2019). Evidence-based recommendations regarding suitable cognitive screening measures are lacking in this age group (Bryce & Allott, Reference Allott, van-der-EL, Bryce, Hamilton, Adams, Burgat and Rickwood2019; Roebuck-Spencer et al., Reference Roebuck-Spencer, Glen, Puente, Denney, Ruff, Hostetter and Bianchini2017; Summers, Bondi, & Bowden, Reference Summers, Bondi and Bowden2019). Cognitive screening tools commonly used in adults may not be sensitive to the nature and extent of cognitive impairment, or trajectories of cognitive development, in people aged 12–25 (e.g., MoCA and MMSE).
The age range of 12–25 is an important age bracket that requires special consideration in relation to cognitive screening for at least four clear reasons. First, this age range is consistent with definitions of young people presented by the World Health Organization and United Nations (www.who.it; www.un.org), which encapsulates both adolescence (i.e., 10–19 years old) and youth (i.e., 15–24 years old). The ages of 12–25 therefore capture most individuals from the start of secondary school into young adulthood. Secondly, this age range is a particularly vulnerable time for cognitive health. Of critical importance for cognitive screening, young people continue a trajectory of dynamic structural and functional cortical and subcortical maturation between the ages of 12–25 (Bava et al., Reference Bava, Thayer, Jacobus, Ward, Jernigan and Tapert2010; Gogtay et al., Reference Gogtay, Giedd, Lusk, Hayashi, Greenstein, Vaituzis and Thompson2004; Foulkes & Blakemore, Reference Foulkes and Blakemore2018), with neural development traversing traditional definitions of pediatric (i.e., <18 years old) and adult samples (i.e., ≥18 years old). This period of growth coincides with expansion of cognitive functions throughout this age range, emphasizing the importance of up-to-date normative data for individuals at each stage of development (Arain et al., Reference Arain, Haque, Johal, Mathur, Nel, Rais and Sharma2013; Larsen & Luna, Reference Larsen and Luna2018; Weintraub et al., Reference Weintraub, Dikmen, Heaton, Tulsky, Zelazo, Bauer and Gershon2013). The rate of improvement of some cognitive skills may begin to plateau at ages 12–15 (e.g., reading); however, most fluid and crystallized domains continue to develop throughout early adulthood (Weintraub et al., Reference Weintraub, Dikmen, Heaton, Tulsky, Zelazo, Bauer and Gershon2013).
Thirdly, between the ages of 12 and 25, youth are at increased risk of developing mental illness and substance abuse or sustaining brain injury associated with elevated risk-taking behaviors (Allott et al., Reference Allott, Proffitt, McGorry, Pantelis, Wood, Cumner and Brewer2013; Arain et al., Reference Arain, Haque, Johal, Mathur, Nel, Rais and Sharma2013). Common health conditions affecting youth may stem from subthreshold or threshold mental illnesses (e.g., mood, substance use, and psychosis-spectrum disorders; Gore et al., Reference Gore, Bloem, Patton, Ferguson, Joseph, Coffey and Mathers2011). Acquired or congenital neurological or medical conditions [e.g., epilepsy and traumatic brain injury (TBI)] and undetected neurodevelopmental disorders [e.g., intellectual disability and attention-deficit/hyperactivity disorder (ADHD)] are also common in this age group (Gore et al., Reference Gore, Bloem, Patton, Ferguson, Joseph, Coffey and Mathers2011; Kassebaum et al., Reference Kassebaum, Kyu, Zoeckler, Olsen, Thomas, Pinho and Vos2017; Proffitt, Brewer, Parrish, McGorry & Allott, Reference Proffitt, Brewer, Parrish, McGorry and Allott2018; Vos et al., Reference Vos, Barber, Bell, Bertozzi-Villa, Biryukov, Bolliger and Murray2015). In addition, functional demands are likely to increase as a young person transitions into adulthood (i.e., independent living). These demands may exceed a young person’s functional capacity, revealing milder cognitive difficulties that have gone undetected earlier in life (Allott et al., Reference Allott, Proffitt, McGorry, Pantelis, Wood, Cumner and Brewer2013). Accordingly, there has been some shift from traditional pediatric (<18 years) and adult (≥18 years) health services, to youth-focused healthcare services that encapsulate the dynamic period of development and period of increased risk for the onset of brain-related conditions during this phase of life. This provides a fourth reason for focusing on this age range in relation to cognitive screening and has been particularly evident in mental health services, with intake criteria catering for the adolescent to young adult age range. For example, headspace, Australia’s federally funded primary mental health service (https://headspace.org.au/), Youth One Stop Shop (YOSS) in New Zealand (http://www.yoss.org.nz/), Foundry in Canada (https://foundrybc.ca/), and jigsaw in the Republic of Ireland (https://www.jigsaw.ie), are all accessible for people aged 12–25. Maisons des Adolescents in France as well as The Junction in Scotland are accessible to people aged 11–21 requiring mental health support (www.monenfant.frl; https://the-junction.org/). Further, EASA, an early psychosis service in the USA is designed for people aged 15–25 (http://www.easacommunity.org). Many of these services are discussed in Hetrick et al., (Reference Hetrick, Cox, Fisher, Bhar, Rice, Davey and Parker2015).
Patterns of cognitive impairment are highly heterogeneous in disorders affecting youth, particularly in the context of personal illness and environmental factors (e.g., premorbid learning style, age of onset, or social milieu; Doyle et al., Reference Doyle, Vuijk, Doty, McGrath, Willoughby, O’Donnell and Braaten2018; Kavanaugh, Cancilliere, & Spirito, Reference Kavanaugh, Cancilliere and Spirito2019). Nevertheless, cognitive deficits across various health conditions are generally found to predict real-world functioning including educational and occupational, interpersonal, independent living, and psychological domains (Arnett et al., Reference Arnett, Peterson, Kirkwood, Taylor, Stancin, Brown and Wade2013; Kerr & Fayed, Reference Kerr and Fayed2017; Lee et al., Reference Lee, Hermens, Naismith, Kaur, Guastella, Glozier and Hickie2018). Cognitive difficulties in youth may also increase caregiver burden and interfere with the effectiveness of medical and psychological interventions (Gonzalez-Heydrich et al., Reference Gonzalez-Heydrich, Hsin, Gumlak, Kimball, Rober, Azeem and Biederman2014; Hetrick et al., Reference Hetrick, Cox, Fisher, Bhar, Rice, Davey and Parker2015; Patel, Wong, Cuevas, & Van Horn, Reference Patel, Wong, Cuevas and Van Horn2013). Thus, similar to older adults, cognitive screening of youth with suspected cognitive deficits may provide an efficient means for early identification and intervention. Screening may also mitigate potential functional impairment related to undiagnosed cognitive impairment or decline (e.g., diminishing academic achievement and poor vocational participation). Early detection of cognitive impairment during this phase of life can set in motion personalized interventions and services that optimize a young person’s developmental pathway into adulthood. The absence of evidence-based recommendations regarding appropriate cognitive screening tools in people aged 12–25 is therefore an unmet clinical need (Bryce & Allott, Reference Allott, van-der-EL, Bryce, Hamilton, Adams, Burgat and Rickwood2019).
The American Psychological Association (APA) Practice Organization Working Group on Screening and Assessment (2014; APA-WGSA) provided general guidelines for distinguishing cognitive screening from comprehensive neuropsychological assessment in mental and behavioral health conditions (www.apaservices.org/practice/reimbursement/billing/assessment-screening). These guidelines were produced to address the greater uptake of cognitive screening in general healthcare settings and provide an initial foundation to discuss cognitive screening in youth in the absence of age-specific recommendations (Block et al., Reference Block, Johnson-Greene, Pliskin and Boake2017; Roebuck-Spencer et al., Reference Roebuck-Spencer, Glen, Puente, Denney, Ruff, Hostetter and Bianchini2017). With respect to these guidelines, cognitive screening tools in youth should be (1) brief (e.g., <30 min), (2) adequately normed to account for a wide range in cognitive development, (3) sensitive to the diverse range of cognitive difficulties experienced in youth with various health conditions, and (4) administered by health practitioners of various professions (Bryce & Allott, Reference Allott, van-der-EL, Bryce, Hamilton, Adams, Burgat and Rickwood2019).
To be useful, cognitive screening tools should have high classification accuracy relative to a gold-standard such as a neuropsychological assessment (Bunnage, Reference Bunnage and Bowden2017). That is, for youth with suspected cognitive issues, a demonstrated ability to correctly detect true cases of cognitive impairment (i.e., high sensitivity) and rule out false cases of impairment (i.e., high specificity; Bunnage, Reference Bunnage and Bowden2017). There is a natural trade-off between sensitivity and specificity for any given cut-off score on a cognitive test; that is, specificity decreases as sensitivity increases and vice versa. Screening tools in youth that balance sensitivity and specificity may be particularly valuable when comprehensive assessment resources are limited (Bauer et al., Reference Bauer, Iverson, Cernich, Binder, Ruff and Naugle2012; Cullen et al., Reference Cullen, O’Neill, Evans, Coen and Lawlor2007). For example, elevated rates of psychological distress in vulnerable youth may result from screening measures generating elevated false positive test results, while missed treatment opportunities may occur if false negative results are high (Roebuck-Spencer et al., Reference Roebuck-Spencer, Glen, Puente, Denney, Ruff, Hostetter and Bianchini2017). Cognitive screening tools should also provide adequate construct coverage, with the tool measuring a representative range of cognitive abilities in youth (Bunnage, Reference Bunnage and Bowden2017).
Objective
The aims of this systematic review were to (1) describe the types of cognitive screening measures used in people aged 12–25, (2) identify the settings in which these tools are utilized, (3) review the diagnostic accuracy of these screening measures in relation to detecting cognitive impairment according to an accepted objective reference standard, and (4) evaluate the methodology of these studies for risk of bias. It was envisaged that this review would inform recommendations for cognitive screening in youth.
METHOD
Search Strategy
The reporting standards of this review were guided by the PRISMA statement (Moher, Liberati, Tetzlaff, Altman, & The PRISMA Group, Reference Moher, Liberati, Tetzlaff and Altman2009). Searches of systematic review registries (e.g., Cochrane Database of Systematic Reviews and PROSPERO) did not reveal any similar reviews underway as of November 2018. The protocol for this review was prospectively registered with PROSPERO on August 27, 2019 (ID: CRD42019133918; www.crd.york.ac.uk/prospero). Article searches were conducted using the online databases OvidMEDLINE, PsychINFO, Scopus, and ERIC. Title, abstract, and keywords were examined in each database using combinations of the following terms: adolescen*, teen*, young adult*, young people, youth, cognit*, neurocognit*, MMSE*, MoCA, and screen*. The MMSE* and MoCA terms were included as (1) these are two of the most commonly used cognitive screening tools across health populations and (2) articles containing these terms were commonly retrieved during preliminary searches. Final database searching was conducted on December 3, 2018 (see Supplementary Material for final combinations for each database). No search limits were applied in any database. Eligibility was assessed manually according to specified inclusion criteria. Reference lists of all included studies and selected reviews on cognitive screening and computerized assessment were also screened for relevant articles (e.g., Arrieux, Cole, & Ahrens, Reference Arrieux, Cole and Ahrens2017; Cernich, Reeves, Sun, & Bleiberg, Reference Cernich, Reeves, Sun and Bleiberg2007).
Inclusion Criteria
Studies were included for review if they met the following criteria (1) original peer-reviewed research article published, or in press, in English, (2) participants were youth, aged between 12 and 25 years inclusive, as indicated by sample mean, (3) the study reported on a tool designed to screen for cognitive functioning, or the tool was used for this purpose, either as an isolated measure or embedded within a broader screening process, and (4) studies examined, or provided data that supported inferences regarding, the criterion validity of the specified screening tool(s) relative to an accepted standardized criterion of cognitive functioning. Guided by the 2014 APA-WGSA statement, the studies were also included if the screening tool being investigated (1) was brief to administer (i.e., ≤30 min), (2) collected objective test data, (3) required minimal, non-specialist training to utilize, (4) was not integrated with historical or functional information, and (5) was not used for diagnostic purposes (e.g., concussion status). Definitions of cognitive impairment were not used as an inclusion criterion due to potential differences between studies (e.g., 1 standard deviation below the general population on two domains, 2 standard deviations below the general population on one domain, etc.). Further, no search terms related to impairment domains were specified due to the broad range of difficulties anticipated within this age group across various health conditions (i.e., intelligence, memory, etc.). No restriction on the type of analysis was applied [e.g., correlations, regression, sensitivity or specificity analysis, receiver operating characteristic (ROC) curve analysis, etc.] to capture a range of study types.
So as to increase the representativeness of the studies included in the review, additional qualifying criteria were applied in view of the number of studies with imprecise reporting of participant age and variable assessment tools used as cognitive screeners. Specifically, where the mean age was not reported, studies were still considered for inclusion if greater than 50% of the sample were aged 12–25 years inclusive. Further, when only age bands were presented (e.g., 18–26), studies were still considered for inclusion if the midpoint of the age band was less than 26. Studies that did not specifically report any objective indication of participant age were not considered for inclusion, even if age might be assumed (e.g., college samples). Studies using collections of established neuropsychological tasks as screening tools were considered for inclusion if (1) at least two tasks measuring different domain-specific Cattell-Horn-Cattell constructs were used (for an overview, see Jewsbury, Bowden, & Duff, Reference Jewsbury, Bowden and Duff2017; Jewsbury, Bowden, & Strauss, Reference Jewsbury, Bowden and Strauss2016; McGrew, Reference McGrew2009), (2) minimal training was required, and (3) clear guidelines for interpretation were provided. Finally, given the exploratory nature of the review, studies were still considered for inclusion if they examined measures that were not originally designed for cognitive screening or clinical purposes, or required some psychometric training. Screening measures based on self or informant report were not included.
Screening, Extraction, and Risk of Bias
Two authors, SDB and KA, independently screened titles and abstracts for articles meeting inclusion criteria for full-text eligibility (Phase 1). Full-text articles were retrieved when eligibility was uncertain. The same two authors independently reviewed each full-text article retrieved in Phase 1 using the same review criteria (Phase 2). Screening discussions were held after piloting the review criteria in Phases 1 and 2 to mitigate potential for bias early in the screening process. When disagreements regarding article suitability in Phase 1 and 2 could not be resolved between the two screening authors; a third author, SCB, was consulted and disagreement were resolved by consensus. Additional studies cited in the reference lists of included studies, or in reference lists of studies excluded according to the design criterion only (criterion 4 above), were considered for inclusion if they met the above inclusion criteria. This additional process was implemented during screening to increase the likelihood that studies relevant to the current review were detected.
Dual data extraction from articles meeting criteria was conducted by authors SDB and KA. Variables of interest included year of publication, country, study design, recruitment method, sample size, participant demographics (e.g., age, sex, and education), index test, reference standard, and any relevant reported psychometric data [e.g., sensitivity and specificity, area under the curve (AUC), likelihood ratios, correlation coefficients, etc.]. Included studies were also assessed for risk of bias and applicability by the same two authors using the revised Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool according to scoring guidelines presented in Whiting et al. (Reference Whiting, Rutjes, Westwood, Mallett, Deeks, Reitsma and Bossuyt2011). Data extraction and risk of bias and applicability ratings both followed the same dual, independent examination and consensus procedures as described for article screening.
Risk of Bias Assessment Using QUADAS-2
The QUADAS-2 assessment tool was designed to assist the evaluation of study quality in diagnostic accuracy studies (www.quadas.org; Whiting et al., Reference Whiting, Rutjes, Westwood, Mallett, Deeks, Reitsma and Bossuyt2011). Scale development was guided by STARD (Standards for Reporting Diagnostic Accuracy Studies) criteria, which were developed to improve the completeness and transparency of reporting classification accuracy studies (Bossuyt et al., Reference Bossuyt, Reitsma, Bruns, Gatsonis, Glasziou, Irwig and Lijmer2003, 2015; Cohen et al., Reference Cohen, Korevaar, Altman, Bruns, Gatsonis, Hooft and Bossuyt2016). The QUADAS-2 has been recommended in reviews of diagnostic accuracy by the Cochrane Collaboration (www.cochrane.org) and the National Institute for Health and Clinical Excellence (NICE; www.nice.org.uk). This scale is used in four structured phases (1) summarize the review question, (2) tailor the tool to the review, (3) construct a flow diagram for the primary study, and (4) assess risk of bias and concerns regarding applicability. Risk of bias is assessed in four domains (1) participant selection, (2) index tests, (3) reference standard, and (4) flow and timing (i.e., flow of participants through the study and timing of administration of index and reference tests). Each domain contains signaling questions that highlight elements of study design that may introduce bias. Signaling questions deemed applicable to this review are rated as “Yes”, “No”, or “Unclear”. Risk of bias may be judged as “Low” if all relevant signaling questions are rated “Yes”, yet may exist if these questions are rated as “No” or “Unclear”. Concerns regarding applicability may be present if related questions are rated as “High” or “Unclear”.
Statistical Interpretation
The sensitivity of a cognitive screening tool reflects its ability to correctly detect individuals with cognitive impairment, while specificity is its ability to correctly exclude those without cognitive impairment (Bunnage, Reference Bunnage and Bowden2017; Lalkhen & McCluskey, Reference Lalkhen and McCluskey2008; Parikh, Mathai, Parikh, Sekhar, & Thomas, Reference Parikh, Mathai, Parikh, Sekhar and Thomas2008). The rate of true positives (sensitivity) can be plotted against the false positive rate (1-specificity) for given cut-off scores (or decision threshold) of a screening measure to generate a ROC curve. The AUC can be used as an interpretive guide to overall classification accuracy (Mandrekar, Reference Mandrekar2010).
Sensitivity and specificity are also used to calculate predictive power and likelihood ratios. Positive predictive power indicates the probability of an individual with a positive test result (i.e., score suggestive of cognitive impairment) actually being impaired, whereas negative predictive power indicates the probability that an individual with a negative test result (i.e., score not suggestive of cognitive impairment) is not impaired. Predictive values are dependent on the prevalence of impairment in the target population (Bunnage, Reference Bunnage and Bowden2017). Positive likelihood ratios (i.e., sensitivity/100-specifity) are another way of interpreting the diagnostic implications of a score or score range and make use of both sensitivity and specificity information. A positive likelihood ratio greater than 10 is used as strong evidence to “rule in” impairment when the test result is positive. In contrast, negative likelihood ratios (i.e., 100-sensitivty/specificity) indicate how much less likely it is that the person has cognitive impairment at a given score or score range. A negative likelihood ratio below .1 is used as strong evidence to “rule out” impairment when the test result is negative (Bunnage, Reference Bunnage and Bowden2017). For studies examining correlations between cognitive screening measures and reference standards, correlation coefficients reaching at least .1, .3, and .5 can be described as small, medium, and large, respectively (Cohen, Reference Cohen1988).
RESULTS
Search Results
Searches of electronic databases and reviews of additional sources retrieved 1122 records, with 619 studies remaining after removing duplicates. After title and abstract screening (Phase 1), 291 articles were subject to full-text review. Of the full-text articles reviewed, 271 were excluded (Phase 2), primarily for not meeting the age criterion (61%). Twenty articles met review criteria and were included in the qualitative synthesis (see Figure 1).
PRISMA flow diagram.
Study Characteristics
The characteristics of the 20 included studies and an overview of the screening measures retrieved are presented in Tables 1 and 2, respectively. Most studies were conducted in North America (85%, n = 17). The average sample size was 116 (SD = 77; range: 28–279) participants. The median of the average ages of participants across each of the included studies was 17 years old. Fourteen different cognitive screening measures were described in the reviewed studies. These measures varied in assessment length (5–30 min), format (pen-and-paper or computerized) and cognitive domains evaluated, as well as initially intended populations (e.g., military- or sport-related concussion or TBI), neuropsychiatry, neurology, geriatrics, oncology.
Study characteristics of articles included for review
Note: Age, percentage male and education reported from primary analysis only; ADHD = Attention-Deficit/Hyperactivity Disorder; NR = Not reported; IQR = Interquartile range; SD = Standard Deviation. ‡ Total sample below 12 years of age, yet TBI sample (66% of sample) mean age = 12.2 (4.1) and authors report child and adolescent data (mean age = NR).§ Median. † Sample size of population group of interest (not comparison group, if applicable). ¶ Demographic information presented in this table was obtained from the corresponding author and pertains to youth aged 20–24. Normative data in this paper was calculated based on the total sample [i.e., N = 268, Mean age = 52.3 (SD = 21.0; range: 20–85)].
Overview of cognitive screening measures used in youth
Cognitive screening tools were used across multiple target populations including healthy people (k = 7, 35%), concussion or mild TBI (k = 3, 15%), neurological conditions (including epilepsy, TBI, brain tumor; k = 3, 15%), other oncology (k = 2, 10%), autoimmune conditions including multiple sclerosis and systemic lupus erythematosus (k = 2, 10%), congenital heart disease (k = 1, 5%), mental illness (k = 1, 5%), and ADHD (k = 1, 5%). The most commonly used screening tools in the included studies were the computerized Immediate Post-Concussion Assessment and Testing (ImPACT; k = 5) and Automated Neuropsychological Assessment Metrics (ANAM; k = 4). These tools were used primarily in healthy people (k = 6) and those who had sustained concussion (k = 3).
Screening Measure Utility
Diagnostic accuracy
Each included study compared performance on at least one cognitive screening measure (the index test) and one standardized measure or battery of cognitive function (i.e., often a global composite; the reference standard). The index tests and reference standards are displayed in Table 3. Approximately one-third (6/20) of studies presented some classification accuracy data as shown in Table 4 (Boulet-Craig et al., Reference Boulet-Craig, Robaey, Laniel, Bertout, Drouin, Krajinovic and Lippé2018; Hoppe, Fliessbach, Schlegel, Elger, & Helmstaedter, Reference Hoppe, Fliessbach, Schlegel, Elger and Helmstaedter2009; Krull et al., Reference Krull, Okcu, Potter, Jain, Dreyer, Kamdar and Brouwers2008; Pike, Poulsen, & Woo, Reference Pike, Poulsen and Woo2017; Portaccio et al., Reference Portaccio, Goretti, Lori, Zipoli, Centorrino, Ghezzi and Gallo2009; Vega-Fernandez et al., Reference Vega-Fernandez, Vanderburgh White, Zelko, Ruth, Levy, Muscal and Brunner2015). In general, evaluation of screening measure utility was limited by the small number of studies retrieved alongside the varied screening measures, wide range of specific populations examined, or minimal diagnostic accuracy data reported. Only two studies provided AUC data (Pike et al., Reference Pike, Poulsen and Woo2017; Vega-Fernandez et al., Reference Vega-Fernandez, Vanderburgh White, Zelko, Ruth, Levy, Muscal and Brunner2015). Based on these data, the MoCA demonstrated “good” classification accuracy (AUC = .84) for detecting cognitive impairment when used in youth with congenital heart disease (Pike et al., Reference Pike, Poulsen and Woo2017). In contrast, the pediatric ANAM (pedANAM) demonstrated “good” to “poor/very poor” accuracy (AUC = .60–.89) in youth with systemic erythematosus lupus depending on the composite scores used (Vega-Fernandez et al., Reference Vega-Fernandez, Vanderburgh White, Zelko, Ruth, Levy, Muscal and Brunner2015). Only one study, which examined the Neurocog FX in youth with epilepsy, reported positive and negative likelihood ratios (Hoppe et al., Reference Hoppe, Fliessbach, Schlegel, Elger and Helmstaedter2009). Of all the Neurocog FX tasks, only one (Figural Memory) provided a positive likelihood ratio above 10 when compared to the reference standard (i.e., scores lower than 2 standard deviations below mean). However, while specificity was high (i.e., range: .57–.99), the sensitivity of this measure for the purposes of cognitive screening was lower (i.e., range: .14–.76).
Overview of screening measures and reference standards for included studies
Note: 2CH = 2-Choice Reaction Time; ANAM = Automated Neuropsychological Assessment Metrics; BNBC = Brief Neuropsychological Battery for Children; BRB = Rao’s Brief Repeatable Battery; BVMT-R = Brief Visuospatial Memory Test-Revised; CDD = Code Substitution Delay; CDS = Code Substitution; CMS = Children’s Memory Scale; CNS-VS = Central Nervous System – Vital Signs; COWAT = Controlled Oral Word Association Test; RAVLT = Rey Auditory Verbal Learning Test; CPT = Continuous Performance Test; CRI = HeadMinder’s Concussion Resolution Index; CVLT = California Verbal Learning Test; CVLT-II = California Verbal Learning Test-Second Edition; CVLT-C = CVLT = California Verbal Learning Test for Children; D-KEFS = Delis-Kaplan Executive Function System; DCS = Diagnosticum für Cerebralschädigung – revidiert; HVLT-R = Hopkins Verbal Learning Test-Revised; ImPACT = Immediate Post-Concussion Assessment and Cognitive Testing; LANSE-C/A = Lebby-Asbell Neurocognitive Screening Examination-Child/Adolescents; LGR = Logical Reasoning; MCST = Modified Card Sorting Test; MGT = Matching Grid Test; MoCA = Montreal Cognitive Assessment; MSP = Matching to Sample; MTH = Math Processing; NeuroCog FX = Neurocognitive Effects; NIHTB-CB = National Institute of Health Toolbox-Cognition Battery; NR = Not reported; PASAT = Paced Auditory Serial Addition Test; PedAMAM = Pediatric Automated Neuropsychological Assessment Metrics; PPVT-4 = Picture Peabody Vocabulary Test-Fourth Edition; PRT = Procedural Reaction Time; RBANS = Repeated Battery for the Assessment of Neuropsychological Status; SAC = Standardized Assessment of Concussion; SDMT = Symbol Digit Modalities Test; SPT = Spatial Processing; SRT = Simple Reaction Time; STN = Sternberg Memory; TMT = Trial Making Test; WAIS-R = Wechsler Adult Intelligence Scale-Revised; WAIS-III = Wechsler Adult Intelligence Scale-Third Edition; WAIS-IV = Wechsler Adult Intelligence Scale-Fourth Edition; WASI = Wechsler Abbreviated Scale of Intelligence; WCST = Wisconsin Card Sorting Test; WeeFIM = Functional Independence Measure; WISC-R = Wechsler Intelligence Scale for Children-Revised; WISC-III = Wechsler Intelligence Scale for Children-Third Edition; WISC-IV = Wechsler Intelligence Scale for Children-Fourth Edition; WJ-III = Woodcock–Johnson Tests of Cognitive Ability; WMS-III = Wechsler Memory Scale-Third Edition; WRAML2 = Wide Range Assessment of Memory and Learning Second Edition; WRAT(III/IV) = Wide Range Achievement Test (Third/Fourth Edition).
Psychometric data retrieved from primary classification accuracy studies
Note: AUC = Area Under the Curve; BNBC = Brief Neuropsychological Battery for Children; CI = Confidence interval; CPS = Cognitive Performance Score; GMI = General Memory Index; IQ = Intellectual Quotient; LR− = Negative Likelihood Ratio; LR+ = Positive Likelihood Ratio; MoCA = Montreal Cognitive Assessment; NeuroCog FX = Neurocognitive Effects; NPC = Negative Predictive Value; NR = Not Reported; PPV = Positive Predictive Value; SD = Standard Deviation.
Correlational data
The remaining two-thirds of studies (k = 14) were not designed as primary classification accuracy studies, as they presented correlation data between index measures and reference standards only as part of broader investigations of cognitive screening tool utility. These data were reported in various ways, as presented in Table 5. Most of these studies (k = 10, 71%) used computerized measures (i.e., ImPACT, ANAM, NIH Toolbox, CNS-VS, CogState, and CRI) and recruited healthy people (k = 8) or those who had sustained concussion (k = 2). As shown in Table 5, the strongest correlations were between LANSE-C/A subtests (excluding the visuospatial domain) and traditional neuropsychological reference standards, with 11/21 significant large correlations (i.e., rs = .6–.9), followed by RBANS total scores and WAIS/WISC Intelligence Quotients (rs = .61–.57). Most correlations between studies comparing computerized index tests and neuropsychological reference standards with available correlation coefficient data in youth specifically (k = 7) were small-to-moderate (i.e., rs = .20–.49; see Table 5). Of these studies, only 11.9% (i.e., 19/160) of significant correlations between specific variations of computerized screening indices and individual neuropsychological tests or composites were large (i.e., r ≥ .50). Most of these large correlations (i.e., 13/19, 68%) were between processing speed and reaction time indices and related reference tasks [i.e., trail making test (TMT) and digit symbol substitution test (DSST)].
Summary of studies examining correlations between cognitive screening tools and comprehensive cognitive assessment
Note: Cognitive screening subtests have been italicized. Correlations in the expected direction (e.g., TMT) are written as positive. 2CH = 2-Choice Reaction Time; ANAM = Automated Neuropsychological Assessment Metrics; BNBC = Brief Neuropsychological Battery for Children; CDD = Code Substitution Delay; CDS = Code Substitution; CMS = Children’s Memory Scale; CNS-VS = Central Nervous System – Vital Signs; COWAT = Controlled Oral Word Association Test; CPT = Continuous Performance Test; CRI = HeadMinder’s Concussion Resolution Index; CVLT-C = CVLT = California Verbal Learning Test for Children; HVLT-R = Hopkins Verbal Learning Test-Revised; ImPACT = Immediate Post-Concussion Assessment and Cognitive Testing; IQ = Intellectual Quotient; LANSE-C/A = Lebby-Asbell Neurocognitive Screening Examination-Child/Adolescents; LGR = Logical Reasoning; MSP = Matching to Sample; MTH = Math Processing; NeuroCog FX = Neurocognitive Effects; NIHTB-CB = National Institute of Health Toolbox-Cognition Battery; NR = Not reported; PedAMAM = Pediatric Automated Neuropsychological Assessment Metrics; PASAT = Paced Auditory Serial Addition Test; RBANS = Repeated Battery for the Assessment of Neuropsychological Status; SAC = Standardized Assessment of Concussion; SDMT = Symbol Digit Modalities Test; SPT = Spatial Processing; SRT = Simple Reaction Time; STN = Sternberg Memory; TMT = Trial Making Test; WAIS-R = Wechsler Adult Intelligence Scale-Revised; WeeFIM = Functional Independence Measure; WISC-R = Wechsler Intelligence Scale for Children-Revised; WISC-III = Wechsler Intelligence Scale for Children-Third Edition; WISC-IV = Wechsler Intelligence Scale for Children-Fourth Edition; WJ-III = Woodcock–Johnson Tests of Cognitive Ability.
Quality
Ratings of risk of bias and quality of study design are presented in Table 6. The majority of studies were deemed of high or unclear risk of bias in terms of patient selection (QUADAS-2 Domain 1; 70%) due to convenience or unclear sampling methods. Further, most studies were judged to be high or unclear risk of bias for the index test (Domain 2; 85%) and reference standard (Domain 3; 90%). Commonly, the extent to which cognitive screening and reference standard neuropsychological assessment were completed and scored independently (i.e., blind rating) was not documented. In addition, roughly half of the studies were rated as high or unclear risk of bias for participant flow and timing (Domain 4; 50%), with participants either not all receiving the reference standard, not all receiving the same reference standard or not all being included in the analysis. Study status as a primary classification accuracy report did not consistently result in lowered risk of bias ratings across the four QUADAS-2 domains, highlighting poor adherence to STARD criteria across most of the included studies.
QUADAS-2 ratings for studies included in the review
Note: HR = High Risk; LR = Low Risk; QUADAS-2 = Quality Assessment of Diagnostic Accuracy Studies – Version 2; UR = Unclear Risk.
DISCUSSION
This review had four primary aims. These were to (1) describe the cognitive screening measures used for people aged 12–25, (2) describe the settings in which the screening measures were used, (3) review the accuracy of these index measures in detecting cognitive impairment relative to an objective reference standard (e.g., neuropsychological assessment), and (4) evaluate the risk of bias of the included studies. To our knowledge, this is the first systematic review of cognitive screening in youth aged between 12 and 25 years.
Overview of Screening Measures Used in Youth
This review highlighted, for the first time systematically, that there exists a diverse range of cognitive screening tools that have been utilized in people aged 12–25, across many clinical and non-clinical populations. These groups included healthy individuals, people with concussion or mild TBI, neurological conditions, oncological diseases, autoimmune disorders, congenital heart disease, mental illness, and neurodevelopmental conditions. Screening approaches differed widely according to assessment length (i.e., 5–30 min), format (i.e., pen-and-paper or computerized), and the range of cognitive domains evaluated. While the types of screening tools were variable, the most commonly used screening tools retrieved in this review were computerized, particularly the ImPACT and ANAM. These measures were used in 45% of the included studies and were mostly administered in youth who were healthy or those who had sustained concussion.
Classification Accuracy
This review found that only six of the 20 studies retrieved (30%) were designed as classification accuracy studies and provided explicit classification accuracy data. Of these studies, few reported sufficient psychometric data to enable a full appraisal of the evaluated screening tools. Some screening tools included in this review, such as the DIVERGT (oncology), MoCA (congenital heart disease), BNBC (multiple sclerosis), and pedANAM (systemic lupus erythematosus) yielded encouraging sensitivity and specificity ranges (i.e., greater than .80 and .60, respectively) in specific groups of people aged 12–25 (Boulet-Craig et al., Reference Boulet-Craig, Robaey, Laniel, Bertout, Drouin, Krajinovic and Lippé2018; Krull et al., Reference Krull, Okcu, Potter, Jain, Dreyer, Kamdar and Brouwers2008; Pike et al., Reference Pike, Poulsen and Woo2017; Portaccio et al., Reference Portaccio, Goretti, Lori, Zipoli, Centorrino, Ghezzi and Gallo2009; Vega-Fernandez et al., Reference Vega-Fernandez, Vanderburgh White, Zelko, Ruth, Levy, Muscal and Brunner2015). The MoCA may have pragmatic utility in youth in primary or tertiary care when compared to other measures given its brevity (i.e., less than 10–12 min to administer), wide availability, ease of administration (i.e., pen-and-paper), and quick interpretation (i.e., single cut-off score). Nevertheless, Pike et al. (Reference Pike, Poulsen and Woo2017) used memory impairment as the reference standard rather than global cognitive functioning in their evaluation of the MoCA in congenital heart disease. Therefore, the extent to which this screening tool may accurately classify general cognitive impairment in youth is unclear. Only one tool, NeuroCogFX (epilepsy) presented a range of psychometric parameters (i.e., sensitivity, specificity, predictive values, and likelihood ratios); however, the balance of sensitivity (i.e., .12-.76) and specificity (i.e., .57-.99) fluctuated significantly in range for each individual screening task, which may question its clinical usefulness (Hoppe et al., Reference Hoppe, Fliessbach, Schlegel, Elger and Helmstaedter2009). Taken together, while some screening measures demonstrate some promise, there is generally very limited data on classification accuracy of cognitive screening in people aged 12–25. Based on the retrieved articles, there are no instances where test accuracy has been replicated in the target clinical populations. Screening tools could therefore not be compared within or between different youth populations. Independent replication of cognitive screening tests is essential to establishing the validity of cognitive screening tools in youth, or any age-group (Straus, Richardson, Glasziou, & Haynes, Reference Straus, Richardson, Glasziou and Haynes2010).
The majority of studies used correlational data to support the utility of cognitive screening in youth, which provides only indirect evidence of validity. Correlations were primarily reported from studies evaluating computerized tools. Most of the significant correlations between index tests and reference standards were small-to-medium; however, with less than 12% of correlation coefficients in studies with available data deemed to be large (i.e., >.50). The strongest relationships appeared to be between computerized processing speed and reaction time indices and speed-based reference standards (e.g., TMT and DSST). This pattern of results may suggest that some specific computerized screening tasks included in this review may be better suited to measuring lower-level abilities including attention and processing speed rather than a broader range complex cognitive domains. Nevertheless, the assessment of whether computerized assessment tasks most closely overlap with established neuropsychological measures of processing speed requires further research.
Risk of Bias
Most studies were deemed to be at high or unclear risk of bias using the QUADAS-2. This major and novel finding in this review highlights the significant methodological limitations of the published studies, to date. The most problematic domains were related to patient selection, use of index tests and use of reference standards. More specifically, ratings of unclear or higher risk of bias were primarily due to convenience or unclear sampling methods as well as uncertainty regarding whether index tests or reference standards were completed and scored independently. A lack of blinding is well known to inflate correct classification rates (Blacker et al., Reference Blacker, Albert, Bassett, Rodney, Harrell and Folstein1994; Schmidt & Factor, Reference Schmidt and Factor2013; Straus et al., Reference Straus, Richardson, Glasziou and Haynes2010). Sources of bias related to sampling and assessment blinding were present in studies that were and were not designed as primary classification accuracy studies. More stringent adherence to STARD criteria in future studies would therefore appear to be essential to reduce the risk of bias when evaluating cognitive screening in youth (Bossuyt et al., Reference Bossuyt, Reitsma, Bruns, Gatsonis, Glasziou, Irwig and Lijmer2003; Reference Bossuyt, Reitsma, Bruns, Gatsonis, Glasziou, Irwig and Cohen2015; Cohen et al., Reference Cohen, Korevaar, Altman, Bruns, Gatsonis, Hooft and Bossuyt2016). The potential for publication bias must also be considered. While unable to be formally assessed in this type of review, it is possible that authors may have been less likely to publish negative screening papers.
Implications
Based on our search strategy and inclusion criteria, studies examining cognitive screening in people aged 12–25 were limited in number and quality. Given the small number of studies, diverse range of cognitive domains and population groups assessed, and an elevated risk of bias among included studies, not even one screening measure can be currently recommended for use in people aged 12–25. Future studies must focus on evaluating cognitive screening tools in youth in accordance with STARD criteria (Bossuyt et al., Reference Bossuyt, Reitsma, Bruns, Gatsonis, Glasziou, Irwig and Lijmer2003, Reference Bossuyt, Reitsma, Bruns, Gatsonis, Glasziou, Irwig and Cohen2015; Cohen et al., Reference Cohen, Korevaar, Altman, Bruns, Gatsonis, Hooft and Bossuyt2016), including brief assessment measures with extensive normative data (e.g., NIH Toolkit).
The lack of consensus regarding cognitive screening measures in youth may contribute to the limited accessibility of neuropsychological assessment in primary care settings (Delagneau et al., Reference Delagneau, Bowden, Bryce, van-der-El, Hamilton, Adams and Allott2020). A validated tool that could assist clinical teams with the process of identifying cognitive impairment and help guide referrals for comprehensive assessment may support more timely access to neuropsychological evaluation (Allott et al., Reference Allott, van-der-EL, Bryce, Hamilton, Adams, Burgat and Rickwood2019; Delagneau et al., Reference Delagneau, Bowden, Bryce, van-der-El, Hamilton, Adams and Allott2020). This would have particular utility in busy clinical settings if various healthcare clinicians, including those without specialist training in cognitive assessment, could administer the tool and interpret it easily [e.g., general practitioners (GPs), mental health clinicians, nurses, social workers, etc.].
With consideration to the review findings, there is a significant need to establish validated cognitive screening measures for people aged 12–25. This may involve examining whether measures used in older adults are appropriate or need to be adapted, or whether new screening tools need to be developed based on the unique trajectories of cognitive development in this younger age group (Bryce & Allott, Reference Allott, van-der-EL, Bryce, Hamilton, Adams, Burgat and Rickwood2019). Given the large number of screening measures already developed, the first step may be to explore the utility of existing tools used in adults as a “downward extension” (e.g., MoCA), with consideration to methodological limitations identified in past studies. It is plasuable, however, that measures developed for adults may only be appropriate for downward extension without significant adaption to people aged 16 and older, when the rate of development of some cognitve skills may start to flatten (Weintraub et al., Reference Weintraub, Dikmen, Heaton, Tulsky, Zelazo, Bauer and Gershon2013). Similar evaluations are needed to determine whether any screening tool is effective at detecting severe or milder impairment in youth, and whether new theoretically-driven test development relevant to younger adolescents may be required. Particular screening measures may be more appropriate for specific groups of youth (e.g., mental health, TBI, etc.). The needs of individual settings will also likely influence what cognitive screening tools and parameters are acceptable, such as what rates of false positives are acceptable if weighting sensitivity over specificity.
Limitations
Reviewing available data on cognitive screening in youth for the first time, with consideration of contemporary definitions, neurobiological development between ages 12 and 25 and service delivery, is a major strength of this review. However, the findings must be considered in light of some limitations. First, the search term “screen*” was used as one of key criterion given the focus of this review. Consequently, studies that did not use this term may have been missed in the electronic search. Reference lists of included articles and reviews that focused on cognitive screening in youth were searched in an attempt to mitigate this possibility, yet inconsistent use of terminology is a limitation in this field (Roebuck-Spencer et al., Reference Roebuck-Spencer, Glen, Puente, Denney, Ruff, Hostetter and Bianchini2017). We also acknowledge that, given our age range of interest, we did not include the search terms pediatric/pediatric, child*, or similar. Inclusion of these search terms would have made our otherwise focused review untenable; however, their absence may have resulted in some missed articles. Second, some flexibility was incorporated into age-related inclusion criterion given the wide range of reporting standards. That is, studies that did not report the mean age of the sample could still be considered for inclusion if either (1) more than 50% of the sample were aged 12–25 or (2) the midpoint of an age bracket containing relevant data fell within 12–25 (e.g., subset of participants aged 18–26). Consequently, some of the included studies inevitably included some participants over 25, which may have influenced the generalizability of the presented diagnostic accuracy data. Thirdly, this systematic review was guided by screening definitions proposed by Roebuck-Spencer et al. (Reference Roebuck-Spencer, Glen, Puente, Denney, Ruff, Hostetter and Bianchini2017) presented by the National Academy of Neuropsychologists. We acknowledge, however, that aspects of these criteria (e.g., up to 30-min screening duration) may not be appropriate in vulnerable youth and may require further adaptation (Rossell, Reference Rossell2019).
CONCLUSIONS
Cognitive screening in people, aged 12–25, at risk of cognitive problems represents an important method of triage guiding comprehensive assessment. Screening also may provide a technique to document and monitor cognitive concerns. Nevertheless, despite widespread use in adults, few studies have examined the utility of cognitive screening measures in youth, defined in this review as 12–25 years. Studies that have evaluated the usefulness of cognitive screening in youth are limited and generally have not presented strong psychometric validity or are at high risk of bias with limited adherence to the STARD criteria. Studies investigating whether current screening tools used in adults, such as the MoCA, or another measure, are suitable for downward extension for subsets of youth (e.g., 16 or older), or need to be adapted across the 12–25 age bracket, would appear to be essential. Alternatively, novel measures that are uniquely sensitive to trajectories of cognitive development in groups of youth, such as those aged 12–14 or 15–17, may need to be developed. Critically, future studies must address the methodical limitations of past research.
ACKNOWLEDGMENTS
We thank the authors from the reviewed studies for their contribution to the field.
FINANCIAL SUPPORT
KA is supported by a Career Development Fellowship from the National Health and Medical Research Council (NHMRC; APP1141207).
CONFLICTS OF INTEREST
The authors have nothing to disclose.
