Introduction
One in six children between the ages of 2 and 8 years has a diagnosed mental, behavioral, or developmental disorder (Cree et al., Reference Cree, Bitsko, Robinson, Holbrook, Danielson, Smith and Peacock2018) and 73.9% of young adults with a psychiatric disorder received their diagnosis before the age of 18 (Kim-Cohen et al., Reference Kim-Cohen, Caspi, Moffitt, Harrington, Milne and Poulton2003). Despite the high prevalence of mental health problems and their early onset, the field of developmental psychopathology is still plagued by the need for a science of “when to worry” (Wakschlag et al., Reference Wakschlag, Roberts, Flynn, Smith, Krogh-Jespersen, Kaat and Davis2019). Although dimensional approaches to studying psychopathology are beneficial for characterizing clinical phenomenology (Sroufe, Reference Sroufe1990), there is a gap in identifying thresholds that denote which individuals to target for prevention or treatment and when. On the one hand, most children who exhibit problem behavior in early development do not go on to develop a mental health disorder. On the other, many children with severe, chronic psychopathology exhibit vulnerability and/or symptoms in early childhood (e.g., conduct problems) (Shaw & Taraban, Reference Sameroff, Seifer and McDonough2004). For example, elevated preschool irritability (a transdiagnostic developmental indicator of life span psychopathology risk) (Wakschlag et al., Reference Wakschlag, Estabrook, Petitclerc, Henry, Burns, Perlman and Briggs-Gowan2018) increases risk of preadolescent internalizing and externalizing psychopathology more than sevenfold (Wiggins et al., Reference Wiggins, Ureña Rosario, Krogh-Jespersen, MacNeill, Walkup, Smith and Wakschlag2021). However, routine care often fails to detect children who show such precursors of psychopathology and even fewer are identified for treatment (Sheldrick, Merchant, & Perrin, Reference Scott and Schulz2011).
The National Institute of Mental Health's Research Domain Criteria (RDoC) is a heuristic framework designed to characterize psychopathology by integrating domains of functioning that cut across multiple levels of analysis for identifying mental health problems. This goal is well aligned with the rationale of longitudinal developmental research put forth by Baltes and Nesselroade (Reference Baltes and Nesselroade1979), in which within-person change in a construct co-occurs with changes in other constructs. Thus, the RDoC domains may be a useful framework for how developmental psychopathologists can begin to determine which constructs can be used for predicting disorder. However, the lack of developmentally based tools specifically designed for decision making (e.g., when to refer) has been one key limitation of the impact of the RDoC on developmental disorders in real-world settings. Reliably differentiating which vulnerable children and youth have an increased likelihood of developing a disorder from those who will have natural course remission or develop adaptively due to environmental supports is critical for designing programs for prevention/intervention. Well-validated, developmentally based, and generalizable clinical supports are therefore essential for detecting probabilistic risk of psychopathology, particularly at critical or sensitive periods of development during which brain and behavior are most malleable.
The development of a risk assessment tool for future pragmatic clinical and research use may be an optimal method for early prediction of mental disorders. Clinical risk calculators have the potential to generate both categorical risk classification as well as a personalized estimate of one's probability of developing a mental health disorder from a set of risk factors. Estimating an individual's risk for developing a disease is a hallmark of personalized medicine. Risk calculators for predicting physical health outcomes can support a clinician's reasoning for the intensity of the prevention/intervention strategy needed based on a personalized risk estimate for an individual patient (Cannon et al., Reference Cannon, Yu, Addington, Bearden, Cadenhead, Cornblatt and Kattan2016). The Framingham risk calculator is one of the most widely used risk calculators for prevention of cardiovascular disease, which has optimized precision medicine. It has transformed the standard of care in cardiovascular disease by leveraging the lowest burden (e.g., most available, lowest cost, most parsimonious) risk indices for risk prediction (Pencina & D'Agostino, Reference Patel, Leathem, Currin and Karlsgodt2012). Although risk calculators have become a popular risk estimation tool in physical health domains, they have been under-utilized in mental health (Bernardini et al., Reference Bernardini, Attademo, Cleary, Luther, Shim, Quartesan and Compton2017; Wakschlag et al., Reference Wakschlag, Luby, Pool, MacNeill, Adam, Barch and Allen2021). The mental health field could greatly benefit from this approach, given the robust support for the efficacy of prevention programs during early and/or vulnerable periods of development (Bernardini et al., Reference Bernardini, Attademo, Cleary, Luther, Shim, Quartesan and Compton2017; Dawson, Ashman, & Carver, Reference Dawson, Ashman and Carver2000; Luby et al., Reference Luby2019).
Risk calculators must be both sensitive (i.e., identifying individuals with psychopathology as positive) and specific (i.e., identifying individuals without psychopathology as negative). A key goal of the mental health risk calculator is to identify the probabilistic risk that an individual will have impairing psychopathology or convert to a disorder from a high-risk state. We focus on probabilistic risk specifically, rather than absolute certainty, given the typical variation in developmental populations (Wakschlag et al., Reference Wakschlag, Roberts, Flynn, Smith, Krogh-Jespersen, Kaat and Davis2019). Probabilistic risk prediction models estimate the probability that individuals will develop a disorder at a future time point by using a set of pre-defined risk indices. These probabilistic models are critical to providing estimates of future risk among patients for whom the future occurrence of the outcome is unknown at the time of clinical decision making. Each individual is assigned (a) an estimated probability of developing a disorder over a length of time based on these set of risk indices and (b) a categorical risk classification to be used for clinical decision making (Pencina & D'Agostino, Reference Patel, Leathem, Currin and Karlsgodt2012).
Neurodevelopmental vulnerability indicators (i.e., brain and behavior that underlie human capacity to engage in, manage, and adapt to everyday experiences that drive daily health and functioning) are transdiagnostic and therefore may make key targets for capturing the heterogeneous emergence of developmental psychopathology (Rogers et al., Reference Rocha, Fisher, Caye, Silva, Arseneault, Barros and Kieling2017; Wakschlag et al., Reference Wakschlag, Estabrook, Petitclerc, Henry, Burns, Perlman and Briggs-Gowan2018). However, patients – particularly young children – are characteristically difficult to assess, and measures of brain structure and function have not been proven clinically useful in mental disorders to date, in part due to nonspecificity for individual patients and disorders. Here, probabilistic risk calculators serve as an invaluable bridge between predictors identified in more basic psychopathology studies and a practice of identifying the fewest indicators needed for effective decision making in the context of real-world implementation. Furthermore, such methods could identify those subjects in need of more intensive and costly neuropsychiatric testing and determine their value added for diagnostic prediction. By detecting the constructs and units of analysis requisite for identifying the subset of individuals that will go on to develop clinical disorders, we may one day be able to prevent disorder onset. In turn, we can decrease the immense public health burden of mental health treatment and its social corollaries (McDaid, Park, & Wahlbeck, Reference Martel, Pan, Hoffmann, Gadelha, do Rosário, Mari and Salum2019).
By embedding development in risk calculators and leveraging the methodological rigor of the RDoC framework, we can cultivate a population-based strategy for actualizing mental health. We have six primary goals in this conceptual paper. The first goal is to define risk calculator parameters and introduce two risk calculator exemplars in the vulnerable phases of the clinical sequence. Second, we discuss an RDoC-informed approach for selection of neurodevelopmental markers and environmental indicators across multiple units of analysis for risk prediction, accounting for measurement intensity. Specifically, we propose that the RDoC constructs and units of analysis can help guide the selection and measurement of risk indicators. Third, we provide strategies for embedding developmental principles in risk calculator designs. Fourth, we discuss considerations for choosing mental health outcomes. Fifth, we offer strategies for generalizing risk prediction, which includes leveraging integrative data strategies. Finally, we discuss how neurodevelopmentally oriented risk calculators can be used to accelerate clinical translation to mental health disorder prevention.
Risk Calculator Parameters and Exemplars in Vulnerable Phases of the Clinical Sequence
A risk model performs well if individuals with the outcome have a higher predicted risk than those who do not. Three key measures of risk model performance are the concordance (c) statistic, discrimination slope, and model calibration. The c statistic for a binary outcome is the area under the receiver operating characteristic (ROC) curve (AUC) and is the most common statistic for discriminating performance of risk calculators (D'Agostino, Griffith, Schmid, & Terrin, Reference D'Agostino, Griffith, Schmid and Terrin1997). The AUC provides a value from 0 to 1 that represents the ability of the parameter or risk score to distinguish between having the disorder and not having the disorder (or any binary outcome of interest). An AUC between .8 and 1 is considered good discrimination performance, AUC = .7–.8 is moderate, AUC = .6–.7 is fair, and AUC = .5 represents no better than chance. The discrimination slope is an index of improvement in model sensitivity and specificity (Pencina, D'Agostino, D'Agostino, & Vasan, Reference Pencina and D'Agostino2008). Calibration is a measure of how closely predicted probabilities align with real experience (D'Agostino et al., Reference D'Agostino, Griffith, Schmid and Terrin1997), such that perfect calibration occurs when predicted and observed risk are equal.
In the field of cardiovascular prevention, clinical guidelines recommend routine cardiovascular risk assessment. The prevention guidelines then include not only absolute levels of clinical risk factors but also a patient's global risk using the pooled cohort equation (PCE) to guide treatment initiation and intensity. The PCE has consistently demonstrated good discrimination with a c statistic >0.75 across a broad range of populations. Importantly, the PCE includes basic demographic and clinical risk factor levels that are easily and reliably obtained as part of routine clinical care. In general, more sophisticated and expensive biomarkers or genetic risk scores have added little to the discrimination of the PCE model.
In the mental health domain, risk calculators may be most clinically useful when applied in vulnerable phases of neurodevelopment, during which we see the onset of key behaviors and characteristics as well as heightened interindividual variability. These phases often coincide with the “grow out of it” or “watch and wait” mentalities (Luby, Reference Luby, Allen, Estabrook, Pine, Rogers, Krogh-Jespersen and Wakschlag2012; Wakschlag et al., Reference Vargas and Mittal2015) that hinder intervention and canalize behavioral tendencies leading to impairment. The Mental Health, Earlier, roadmap proposed by Wakschlag et al. (Reference Wakschlag, Roberts, Flynn, Smith, Krogh-Jespersen, Kaat and Davis2019) aims to reduce the research-to-practice gap by calling for interdisciplinary developmental science to work toward mental health prevention. Specifically, Wakschlag et al. (Reference Wakschlag, Roberts, Flynn, Smith, Krogh-Jespersen, Kaat and Davis2019) propose two pillars: earlier and healthier. The earlier pillar presents a shift toward transdiagnostic and probabilistic diagnosis processes to distinguish between normative (and potentially transient) behavior and maladaptive development. The healthier pillar is grounded on prevention rather than a reactive treatment approach. By determining an individual's probabilistic risk for psychopathology across multiple levels of analysis, we can better prevent the emergence of disorder. Throughout this paper, we discuss the application of the risk calculator approach to two different neurodevelopmental pathways and phenotypes: (a) psychotic syndromes in youth and (b) common internalizing/externalizing problems in early childhood. We find these to be a useful contrast illustratively because these syndromes have distinct phenomenology and etiologies. Further, the former present a rare and severe disorder that usually onsets in late adolescence/early adulthood and the latter represent the most common forms of psychopathology that typically onset in early childhood. Although not exhaustive, Table 1 provides an overview of key studies presenting risk algorithms predicting a range of psychopathologies including psychosis, bipolar disorder (for a review, see Silva Ribeiro et al., Reference Silva Ribeiro, Pereira, Salagre, Coroa, Santos Oliveira, Santos and Vieta2020), depression, attention-deficit/hyperactivity disorder (ADHD), and to a lesser extent, internalizing/externalizing problems.
Examples of commonly used and otherwise interesting risk calculators in psychiatric and psychological research
Note: ADHD = attention-deficit/hyperactivity disorder; ALSPAC = Avon Longitudinal Study of Parents and Children; AUC = area under the receiver operating characteristic curve; BP = bipolar disorder; BP-NOS = bipolar disorder not otherwise specified; BPRS-E = Brief Psychiatric Rating Scale – Expanded; CHR = clinical high risk; COBY = Course and Outcome of Bipolar Youth; COPS = Criteria of Prodromal States; DSM-IV = Diagnostic and Statistical Manual of Mental Disorders, fourth edition; MHESC = Mental Health, Earlier Synthetic Cohort; EEG = electroencephalography; FePsy = Basel Früherkennung von Psychosen; ICD = International Statistical Classification of Diseases and Related Health Problems; LASSO = Least Absolute Shrinkage and Selection Operator; MDD = major depressive disorder; MRI = magnetic resonance imaging; NAPLS = North American Prodrome Longitudinal Study; PTSD = post-traumatic stress disorder; SHARP = Shanghai At-Risk for Psychosis; SIPS = Structured Interview for Prodromal Syndromes; SLAM = South London and the Maudsley NHS Foundation Trust; SOPS = Scale of Psychosis-risk Symptoms; SZ = schizophrenia
Predicting psychosis
The majority of the research on mental health risk calculators, albeit small, is on the conversion to psychosis from prodromal stages (Cannon et al., Reference Cannon, Cadenhead, Cornblatt, Woods, Addington, Walker and Heinssen2008, Reference Cannon, Yu, Addington, Bearden, Cadenhead, Cornblatt and Kattan2016; Carrión et al., Reference Carrión, Cornblatt, Burton, Tso, Auther, Adelsheim and McFarlane2016; Ciarleglio et al., Reference Ciarleglio, Brucato, Masucci, Altschuler, Colibazzi, Corcoran and Girgis2019; Fusar-Poli, Davies et al., Reference Fusar-Poli, Davies, Rutigliano, Stahl, Bonoldi and McGuire2019; Fusar-Poli et al., Reference Fusar-Poli, Rutigliano, Stahl, Davies, Bonoldi, Reilly and McGuire2017; Fusar-Poli, Werbeloff et al., Reference Fusar-Poli, Werbeloff, Rutigliano, Oliver, Davies, Stahl and Osborn2019; Irving et al., Reference Irving, Patel, Oliver, Colling, Pritchard, Broadbent and Fusar-Poli2021; Lee et al., Reference Lee, Hwang, Kim, Park, Lho, Moon and Kwon2020; Oliver et al., Reference Oliver, Spada, Colling, Broadbent, Baldwin, Patel and Fusar-Poli2020, Reference Ofonedu, Belcher, Budhathoki and Gross2021; Osborne & Mittal, Reference Oliver, Wong, Bøg, Jönsson, Kinon, Wehnert and Fusar-Poli2019; Studerus et al., Reference Sroufe2020; Worthington et al., Reference Wiggins, Ureña Rosario, Krogh-Jespersen, MacNeill, Walkup, Smith and Wakschlag2021; Zhang et al., Reference Worthington, Walker, Addington, Bearden, Cadenhead, Cornblatt and Cannon2018, Reference Yates, Egeland and Sroufe2019). Individuals at clinical high risk (CHR) for psychosis typically exhibit attenuated positive (e.g., hallucinations) and negative (e.g., anhedonia) symptoms and decreased socio-occupational functioning (McGlashan, Walsh, & Woods, Reference McDaid, Park and Wahlbeck2010). Most patients show milder forms of these symptoms (e.g., brief hallucinations or mild anhedonia) before the onset of fully psychotic symptoms (Fusar-Poli et al., Reference Fusar-Poli, Bonoldi, Yung, Borgwardt, Kempton, Valmaggia and McGuire2012). Understanding the risk factors that distinguish those who develop the disorder within the broader group of individuals with CHR syndrome may facilitate the prevention of the transition from CHR to psychosis and improve the course of illness in people who do convert (Fusar-Poli et al., Reference Fusar-Poli, Borgwardt, Bechdolf, Addington, Riecher-Rössler, Schultze-Lutter and Yung2013). There is high variability in symptom progression in CHR individuals, and those that do not convert are still likely to develop a host of maladaptive outcomes. Therefore, CHR individuals are a population that is well suited to the individualized probabilistic prediction of a risk calculator approach (Osborne & Mittal, Reference Oliver, Wong, Bøg, Jönsson, Kinon, Wehnert and Fusar-Poli2019). Thus far, several risk indicators for psychosis prediction in the CHR population have been identified, including individual history of mental health problems, earlier onset of nonpsychotic mental disorders, and various demographic factors (Fusar-Poli, Davies et al., Reference Fusar-Poli, Davies, Rutigliano, Stahl, Bonoldi and McGuire2019; Fusar-Poli et al., Reference Fusar-Poli, Rutigliano, Stahl, Davies, Bonoldi, Reilly and McGuire2017).
The North American Prodrome Longitudinal Study (NAPLS) and the Shanghai At-Risk for Psychosis (SHARP) program have developed two notable risk calculators (NAPLS-2 and SIPS-RC, respectively) that provide probabilistic risk scores for conversion to a psychotic disorder for individuals at CHR.
The NAPLS-2 calculator provided risk scores for psychosis across 2 years (Cannon et al., Reference Cannon, Yu, Addington, Bearden, Cadenhead, Cornblatt and Kattan2016). The NAPLS-2 calculations are based on a multivariate proportional hazards regression model that includes: (a) age, (b) symbol-coding score (Keefe et al., Reference Keefe, Goldberg, Harvey, Gold, Poe and Coughenour2004), (c) the Hopkins Verbal Learning Test – Revised, (d) negative life event sum scores, (e) a drop in global functioning over the prior year, (f) severity of unusual thought content and suspiciousness, (g) family history of psychosis, and (h) total number of lifetime experienced traumas. Of note, the NAPLS risk score reflects an amalgamation of both changing features (e.g., age, cognition, symptom severity, and global function drop) with stable features of risk (e.g., family history of psychosis). As a result, an individual with relatively stable ratings in all areas may “progress” in risk over time by virtue of age alone.
The SIPS-RC calculator included participants with at least a 1-year follow-up assessment (Zhang et al., Reference Yates, Egeland and Sroufe2019). The SIPS-RC derives risk estimates from four dimensions: (a) current positive symptom severity, (b) current negative symptom severity, (c) low levels of dysphoric mood (general symptoms), and (d) deterioration of functioning over the past 4 months.
When examining the relative performance of the two risk calculators in an independent sample, the NAPLS-2 and SIPS-RC demonstrated moderate to fair discrimination performance for distinguishing CHR converters and nonconverters, respectively (Osborne & Mittal, Reference Oliver, Wong, Bøg, Jönsson, Kinon, Wehnert and Fusar-Poli2019). However, when examining the calculators’ abilities for predicting positive symptom progression, both calculators provided moderate discrimination ability, with the SIPS-RC (AUC = .76) providing qualitatively better prediction of positive symptom progression than the NAPLS-2 (AUC = .71). Differences in discrimination performance between the two calculators may reflect cross-culture differences in risk prediction: when validating the NAPLS-2 risk calculator in the Chinese SHARP sample, a similar reduction in discrimination performance was observed for the NAPLS-2 (AUC = .63) (Zhang et al., Reference Worthington, Walker, Addington, Bearden, Cadenhead, Cornblatt and Cannon2018).
The existing risk calculators for predicting psychosis have shown promise for adopting a developmental perspective. For example, the NAPLS-2 calculator is based on a study that reduced hundreds of possible variables to a small number, including developmentally relevant constructs (e.g., childhood trauma). A developmentally informed RDoC approach has the potential to build on these foundations. For example, currently, in the NAPLS-2 calculator, life events and trauma do not offer a high predictive value for later conversion to psychosis. Rather than summing Childhood Trauma and Abuse Questionnaire items to yield an index (Cannon et al., Reference Cannon, Yu, Addington, Bearden, Cadenhead, Cornblatt and Kattan2016; Janssen et al., Reference Janssen, Krabbendam, Bak, Hanssen, Vollebergh, de Graaf and van Os2004), a potentially more powerful approach might be to capture the timing, duration, severity, or degree of impact of trauma or stressors. Moreover, rather than treating trauma as a uniform construct, researchers could consider distinguishing between conceptually and empirically distinct types (e.g., neglect or threat) (Vargas, Conley, & Mittal, Reference Studerus, Beck, Fusar-Poli and Riecher-Rössler2020; Vargas & Mittal, Reference Teti, O'Connell and Reiner2018; Vargas, Zou, Conley, & Mittal, Reference Vargas, Conley and Mittal2019). As such, given the relevance of sensitive and critical periods of neurodevelopment in informing degrees of impact of environmental exposures, the predictive value of trauma and life events could be underestimated in the current NAPLS-2 calculations. Further incorporating a developmental perspective to assessing risk could be helpful in this regard.
Predicting internalizing and externalizing problems in early childhood
Although existing calculators predicting adult clinical outcomes have considered events in childhood, there is a growing need to expand this risk evaluation during early development. Meehan et al. (Reference McGorry, Purcell, Goldstone and Amminger2020) used a risk calculator model to identify significant risk predictors of psychiatric outcomes in environmental risk populations aged 7–18 years, identifying individuals of victimization being the most at risk for the development of any psychiatric disorder. Risk calculators predicting depression and ADHD in youth and young adults have also been tested, demonstrating c statistics above 0.7 in test samples (but poorer discrimination in some validation samples; e.g., Caye et al., Reference Caye, Agnew-Blais, Arseneault, Gonçalves, Kieling, Langley and Rohde2020; Rocha et al., Reference Roberts, Attkisson and Rosenblatt2021). Research has also shown that some of the problem behaviors identified in these older age groups (e.g., the majority of severe and chronic internalizing and externalizing problems; Meehan et al., Reference McGorry, Purcell, Goldstone and Amminger2020; Damme, Wakschlag, Briggs-Gowan, Norton, & Mittal, Reference Damme, Wakschlag, Briggs-Gowan, Norton and Mittal2021) onset with symptoms or vulnerability patterns in early childhood, laying the groundwork for identification far earlier in the developmental sequence (Wakschlag et al., Reference Wakschlag, Estabrook, Petitclerc, Henry, Burns, Perlman and Briggs-Gowan2018).
The first few years of life are a time of heightened neurodevelopmental plasticity, and interventions delivered during this time may be most effective at both the cost and impact level (Campbell et al., Reference Campbell, Conti, Heckman, Moon, Pinto, Pungello and Pan2014). The significant overlap between the normative misbehaviors of early childhood and problems in self-regulation has been a major impediment to early identification (Wakschlag, Tolan, & Leventhal, Reference Wakschlag, Perlman, Blair, Leibenluft, Briggs-Gowan and Pine2010). To differentiate normative variation from clinical risk markers presaging chronic psychopathology, researchers must identify the constellation of modifiable risk factors that identify the subset of young children with early problems in self-regulation that will advance to persistent clinical problems (Luby et al., Reference Luby2019). This includes functioning in other domains of development and social–ecological factors that may modify the likelihood that risk will result in adaptive or maladaptive outcomes.
Although risk calculators have yet to be developed for psychopathology prevention in early childhood, our preliminary data indicate the potential utility of this approach. Using data harmonized from two early childhood data sets, we examined the utility of multiple-level early childhood behavioral and ecological risk factors in preschool to predict internalizing/externalizing disorders at preadolescence. The transdiagnostic behavioral indicator, irritability, provided the most discriminative power, and behavioral risk was amplified by exposure to environmental adversity (Wakschlag et al., Reference Wakschlag, Luby, Pool, MacNeill, Adam, Barch and Allen2021). To achieve the power and precision necessary for generating a tool for dissemination, we are conducting the Mental Health, Earlier Synthetic Cohort (MHESC) Study. This is a new initiative designed to generate and validate a developmentally based risk calculator, for which we are currently in the early development stages. The goal is to compare innovative computational and epidemiologic data science methods to accelerate clinical translation of neurodevelopmental discovery during infancy toward generalizable risk prediction for preschool psychopathology. We are pooling three independent, extramural data sets to form the first clinically enriched “synthetic” neuroimaging cohort for the generation of neurodevelopmentally based clinical risk algorithms (N = 1,020, followed from birth to age 54 months). Unlike prototypical risk calculators for psychosis prediction, risk calculators in early childhood, such as the MHESC Study, are designed to operate in a relatively normative population. The MHESC Study will focus on transdiagnostic indicators of emotion dysregulation, as emotion dysregulation underlies the development of common and modifiable internalizing and externalizing problems and is measurable early in life.
A key emotion dysregulation indicator of emerging psychopathology in the MHESC Study is irritability. Irritability is detectable in infancy and early childhood, can be reliably assessed at survey, behavior, and paradigm levels in these developmental periods, and has established neural correlates (Wakschlag et al., Reference Wakschlag, Roberts, Flynn, Smith, Krogh-Jespersen, Kaat and Davis2019). Noninvasive neural measures such as electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) have provided important insights into the development of brain structure and function underlying emotion dysregulation, including irritable behaviors (Brotman, Kircanski, Stringaris, Pine, & Leibenluft, Reference Brotman, Kircanski, Stringaris, Pine and Leibenluft2017; Morawetz et al., Reference Mittal and Walker2020; Nielsen, Wakschlag, & Norton, Reference Nielsen, Wakschlag and Norton2021). For instance, fMRI studies demonstrate that young children who are moderately irritable but not impaired recruit the prefrontal cortex for regulating frustration, but children who have severe impairment show less activation in this region (Grabell et al., Reference Grabell, Li, Barker, Wakschlag, Huppert and Perlman2018). Similarly, structural MRI has revealed that severe irritability in early school-age children predicted reduced gray matter volumes in areas related to emotion (e.g., amygdala, medial prefrontal cortex) at preadolescence (Damme et al., Reference Damme, Wakschlag, Briggs-Gowan, Norton and Mittal2021). With respect to EEG and event-related potentials, children who are persistently irritable at age 3 years have shown a heightened error-related negativity (ERN) component (associated with error monitoring) at age 6, which in turn predicted greater internalizing problems at age 9. Further, a blunted ERN was related to more externalizing problems (Kessel et al., Reference Kessel, Meyer, Hajcak, Dougherty, Torpey-Newman, Carlson and Klein2016).
An RDoC-Informed Approach for Selection of Neurodevelopmental Markers and Environmental Indicators
Practical guidelines for incorporating markers into a risk calculator have been established, such as defining the population and outcomes of interest, as well as modeling and testing selection to evaluate incremental value (D'Agostino, Reference D'Agostino2012). The most rudimentary but necessary condition for including a marker is its statistical significance (Pencina et al., Reference Pencina and D'Agostino2008), but that alone does not guarantee that the marker will improve the discrimination of a risk prediction tool. Given the dearth of developmental mental health risk calculators that exist, RDoC principles can be particularly helpful for guiding initial decision making for risk indicator inclusion.
A core tenet of the RDoC framework is the use of multiple units of analysis when measuring a construct within a particular domain. The RDoC delineates behavioral and neural domains of positive valence, negative valence, cognitive, social processes, arousal/regulatory, and sensorimotor systems. Each of these domains has a set of constructs that can be measured across multiple units. For example, a researcher may measure the construct of fear in the negative valence system at the circuit unit (e.g., ventromedial prefrontal cortex), the physiological unit (e.g., pupillary dilation), the survey unit (e.g., parent report of temperament), and the paradigm unit (e.g., Laboratory Temperament Assessment Battery). The researcher could then use these scores in a risk calculator algorithm to assess the added predictive value of each unit for the development of an anxiety disorder.
As previously mentioned, the MHESC Study will use indicators of emotion dysregulation to predict probabilistic risk for psychopathology. Emotion regulation indicators span RDoC domains that include negative valence, positive valence, arousal and regulatory, and cognitive systems. Negative valence systems reflect how an individual responds to stressful or aversive situations (e.g., fear), while positive valence systems pertain to positive motivational contexts (e.g., responses to reward). For example, highly irritable children show alterations in brain regions associated with reward processing, error monitoring, and emotion regulation when they expect a reward that is not received (Perlman et al., Reference Pencina, D'Agostino, D'Agostino and Vasan2015). Arousal and regulatory systems pertain to an individual's sensitivity to external and internal stimuli, functionally facilitating engagement with the environment in a context-specific manner. These systems could facilitate or hamper emotion regulatory efforts. Cognitive functions intrinsic to emotion regulation dissociate components of RDoC cognitive system domains. The cognitive domains most relevant to emotion regulatory capacities would include cognitive control, particularly the inhibition subconstruct, and working memory, particularly the active maintenance and flexible updating subconstructs.
Irritability, when conceptualized within an RDoC framework, stems from alterations across a swathe of domains as well, including negative valence domain (e.g., frustrative nonreward), as well as positive valence domain (e.g., errors in reward prediction), and cognitive domain impairments including attention and language subdomains (Bell, Bryant, Boyce, Porter, & Malhi, Reference Bell, Bryant, Boyce, Porter and Malhi2021). In addition, irritability may engage emotion regulation indicators, thus comprising parts of the above discussed arousal/regulatory, cognitive control/inhibition, and working memory/active maintenance and flexible updating. Using an RDoC approach, a key goal of the MHESC Study is to measure constructs such as irritability and its correlates across multiple units of analysis, including circuits (e.g., natural sleep MRI), physiology (e.g., EEG, eye-tracking), behaviors (e.g., in-lab observations), parent reports, and paradigms (e.g., in-lab experimental tasks).
These indicators and their units of analysis have been extensively studied in developmental populations and are considered robust predictors of child mental health outcomes (Cicchetti, Reference Cicchetti, Beauchaine and Hinshaw2008). Of note, the bulk of evidence on neural circuitry is from older children. However, there is an accruing body of literature on neural circuitry in infancy and early childhood underlying risk for developing psychopathology. For instance, amygdala resting-state functional connectivity in neonates has predicted internalizing symptoms at 2 years of age (Rogers et al., Reference Rocha, Fisher, Caye, Silva, Arseneault, Barros and Kieling2017). Less amygdala reactivity has been related to greater depression in preschoolers and aberrant default mode network connectivity has been identified in preschoolers with a history of depression (Gaffrey et al., Reference Gaffrey, Barch, Bogdan, Farris, Petersen and Luby2018; Gaffrey, Luby, Botteron, Repovš, & Barch, Reference Gaffrey, Luby, Botteron, Repovš and Barch2012). On the whole, however, there are high levels of variability in findings and a lack of standardization of neural measures in young populations. For the MHESC Study, we will synthesize the neural circuitry findings across the life span with extant knowledge from early childhood to select appropriate neural biomarkers for inclusion in the risk algorithms. It is important to include these neural indicators in the development of the MHESC given that risk calculators may have the greatest clinical value in these younger age groups. Neural measures are seldom incorporated in risk calculators due to increased burden and low reproducibility of findings. Thus, it is imperative that any findings from the MHESC Study – particularly with respect to neural correlates – be robust, replicated, and externally validated before they are included in a risk calculator for public health use (see the section Strategies for Generalizing Risk Prediction).
Although the RDoC framework spans a wide range of constructs examined on a spectrum of normal to abnormal, it focuses on capturing deviations of degree. For instance, anxiety becomes a disorder when an individual demonstrates excessive fear or anxiety, but some anxiety – particularly in response to stress – is normal and adaptive (e.g., vigilance toward impending threat). However, less emphasized in the RDoC are deviations of kind, or phenomena that are not evident in normative development (Mittal & Wakschlag, Reference Mittal and Wakschlag2017). Deviations of kind include compulsive behaviors evident in obsessive–compulsive disorder. Deviations of degree and kind should both be considered when choosing risk indicators and should align with the outcome of interest.
Accounting for environmental factors that amplify or attenuate risk
An individual's dysregulation is nested within dynamic environmental contexts and is an important predictor of later adaptation; a tiered approach is thus necessary for risk prediction. Neurodevelopmental risk is probabilistic, such that multiple individual and contextual mechanisms shape early characteristics and behaviors over the course of development to influence outcomes (Mittal & Wakschlag, Reference Mittal and Wakschlag2017). These risk factors do not occur in isolation, but rather operate together to inform development (Rutter, Reference Rothbart, Bates, Damon and Eisenberg1998). Further, given the large intraindividual and interindividual variability in the development of mental health problems, particularly in early development, an analysis across multiple systems and modalities is essential for identifying profiles of risk that can inform screening and intervention (Finlay-Jones et al., Reference Finlay-Jones, Varcin, Leonard, Bosco, Alvares and Downs2019).
Pediatric care settings have only recently started considering environmental factors in the picture of a child's health, even though there is robust evidence supporting that early adversity and environmental stress influence mental health problems and the efficacy of interventions (Shonkoff, Reference Shonkoff2010; Shonkoff, Boyce, Levitt, Martinez, & McEwen, Reference Sheldrick, Merchant and Perrin2021). Epidemiological perspectives suggest that multiple ecological subsystems of the individual are the key predictors for mental health cumulative risk scores, such that accrual of adversity over time decreases developmental competence (Sameroff, Seifer, & McDonough, Reference Rutter2004). These subsystems include parent mental illness and psychological wellbeing, parent attitudes and beliefs about child development, responsive and sensitive parenting, and social determinants of health such as racial discrimination (Fowler, Tompsett, Braciszewski, Jacques-Tiura, & Baltes, Reference Fowler, Tompsett, Braciszewski, Jacques-Tiura and Baltes2009; Goodman et al., Reference Goodman, Rouse, Connell, Broth, Hall and Heyward2011; Sameroff, Seifer, Barocas, Zax, & Greenspan, Reference Sameroff, Seifer, Barocas, Zax and Greenspan1987; Teti, O'Connell, & Reiner, Reference Steyerberg, Moons, van der Windt, Hayden, Perel, Schroter and Altman1996). Although many of these factors amplify risk for psychopathology, there has been less of an emphasis in mental health risk factors on resilience-promoting factors (Yates, Egeland, & Sroufe, Reference Wilson-Mendenhall, Barrett, Simmons and Barsalou2003). For example, warm and responsive parenting interacts with early regulatory abilities in a bidirectional manner, and this type of parenting is the main target of most evidence-based early prevention/intervention programs (Kochanska & Kim, Reference Kochanska and Kim2013; Smith et al., Reference Simon2020; Waller et al., Reference Walker, Trotman, Goulding, Holtzman, Ryan, Macdonald and Brasfield2014). Further, even though developmental psychopathology research has traditionally included factors both intrinsic and extrinsic to the child in predicting mental health outcomes (Cicchetti & Dawson, Reference Cicchetti and Dawson2002), mental health risk calculators typically do not include these extrinsic, contextual factors. An important goal of the MHESC Study is to test the added value of social determinants of health, parenting, and family context indicators. In our proof-of-concept risk calculator predicting preadolescent psychopathology from preschool indicators (Wakschlag et al., Reference Wakschlag, Luby, Pool, MacNeill, Adam, Barch and Allen2021), adding environmental adversity to the model accounting for demographics and behavioral risk improved model discrimination.
RDoC conceptualizations stress the value of environmental, social, and developmental context for emotional experience, regulation, and symptom presentation (Barrett, Reference Barrett2012; Barrett, Mesquita, & Gendron, Reference Barrett, Mesquita and Gendron2011; Wilson-Mendenhall, Barrett, Simmons, & Barsalou, Reference Wilson-Mendenhall, Barrett, Simmons and Barsalou2011). The arousal domain, for example, enables sensitivity to internal or external stimuli such that context-sensitive action can be undertaken. In this case, chronic high arousal preparing the organism for context-sensitive action in the face of repeated exposure to environmental threats could constitute an adaptive response. However, chronic environmental threat exposure occurring during sensitive developmental periods could ultimately contribute to emerging vulnerability for psychopathology. As such, constructs that relate to emotion regulation could interact with and be impacted by both development and environment.
Which methods when?
Multiple levels of analysis are important for best identifying risk for psychopathology, but assessment at all levels for all children may not be feasible, practical, or translatable. Indicators of emotion dysregulation are measurable in the first year of life, pervasive across mental health disorders, and associated with disruptions in the prefrontal cortex (Beauchaine, Reference Beauchaine2015; Beauchaine & Cicchetti, Reference Beauchaine and Cicchetti2019; Finlay-Jones et al., Reference Finlay-Jones, Varcin, Leonard, Bosco, Alvares and Downs2019). Yet questions remain regarding which or what combination of indicators are sufficient for predicting mental health risk, as well as when and for whom these indicators are most predictive. In this context, “sufficient” means the most parsimonious and least intensive/burdensome set of indicators that add clinically meaningful explanatory value. By leveraging multiple units of analysis in a risk calculator approach to developmental psychopathology research, we can determine the units that are needed for identifying children across the spectrum of risk to streamline measurements and reduce burden at the clinical level.
Risk prediction models prioritize low-burden or less-intense measures, warranting the inclusion of more burdensome or intense measures only when they evidence substantial added predictive value (Lloyd-Jones, Reference Lloyd-Jones2010). For example, the MHESC Study will aim to better understand whether more cost- and resource-intensive methods have significant added value to early mental health risk prediction. The first risk calculator algorithm will be derived solely from commonly used survey data to optimize feasibility for future use in primary care settings. Then, we will investigate the statistical and clinical incremental utility of more intensive assessment for future use in mental health specialty settings. This algorithm sequentially tests the added predictive value of methods of intermediate–high intensity (from direct observational or performance-based assessments to MRI) for the most precise, least burdensome risk prediction. This sequential method will enable us to establish for which children surveys alone are adequate, when behavioral methods have added value, and/or whether EEG and/or MRI have sufficient added value for children with higher clinical uncertainty (e.g., those that fall in the middle of the dimensional spectrum) after replication and validation to warrant consideration of use. In addition, dimensional approaches and capturing heterogeneity lead to the question of what works for whom. As RDoC and developmental psychopathology frameworks point away from a “you have it or you don't” approach to clinical risk towards a vulnerability spectrum conceptualization, it is possible that children at neither clinical extreme will need more intensive measurements or additional domains to precisely determine probabilistic risk. For instance, using a stoplight metaphor (Smith et al., Reference Silva Ribeiro, Pereira, Salagre, Coroa, Santos Oliveira, Santos and Vieta2018), children in green (low risk) may only receive additional testing as part of their regular well-child visit, children in red (high risk) may receive immediate referral for prevention/intervention or mental health specialty, and children in yellow (higher clinical uncertainty) may prompt the additional intermediate–high intensity measures.
An alternative approach in risk algorithm development is to incorporate tools that are practical for implementation across all potential risk levels. For example, these calculators may only use demographic information and clinical/neurocognitive functioning measures in the forms of surveys and short tests (Cannon et al., Reference Cannon, Chung, He, Sun, Jacobson, van Erp and Heinssen2015). Digital technologies for capturing behavioral assessments may be optimal for reaching under-represented and low-resourced families. Online platforms such as Lookit (Scott & Schulz, Reference Sameroff, Seifer, Barocas, Zax and Greenspan2017) have pushed past the typical constraints of in-person data collection by capturing a wide range of child behaviors from participants representative of the US population with respect to race, parent education, and income. Prioritizing cognitive tasks that adults can complete online could help the critical effort of reaching non-help-seeking individuals. Using lower-burden, cost-effective, and easily accessible assessments may increase the likelihood of widespread adoption in the real world. Calculators that leverage these types of assessments will also be more accessible to adolescents and young adults who have concerns about their mental health, facilitating their ability to make informed decisions for seeking information and support.
Strategies for Embedding Developmental Principles in Risk Calculator Designs
Developmental psychopathologists use individual characteristics to predict which individuals will grow into healthy, prosocial, competent adults, as well as those for whom mental health problems might arise. Importantly, some of these characteristics change across seconds, days, and months, while some remain relatively stable across the life course. Change, or the absence of it, is woven into the fabric of risk for psychopathology, but is not always prioritized in the actual calculation of risk. There is recent evidence that accounting for longitudinal change in blood pressure does improve the precision of probabilistic risk prediction in cardiovascular disease (Pool, Ning, Wilkins, Lloyd-Jones, & Allen, Reference Perlman, Jones, Wakschlag, Axelson, Birmaher and Phillips2018). The RDoC framework, while providing a comprehensive system for identifying risk factors that promote psychopathology, has not yet attempted to account for such intraindividual variability and change.
Risk status is situated in trajectories of biobehavioral change across development, rather than a characteristic or behavior at one point in time (Wakschlag et al., Reference Vargas and Mittal2015). Capturing all risk indicators at one point in time can lead to an over- or under-identification of impairment. On the one hand, treating normative variation in behavior as a clinical marker may squander limited resources. On the other hand, clinically concerning behavior may not occur until later in development for some children, which may lead to an under-identification of disorder. Further, even when change is considered, many studies have simplified development to a change score in the two-time-point approach, which can smooth out important discontinuities inherent to development (Adolph, Robinson, Young, & Gill-Alvarez, Reference Adolph, Robinson, Young and Gill-Alvarez2008). Longitudinal designs with time points that are spaced out over long periods may also mask development as it emerges (Adolph et al., Reference Adolph, Robinson, Young and Gill-Alvarez2008; Boker & Nesselroade, Reference Boker and Nesselroade2002; Collins, Reference Collins2006; Damme et al., Reference Damme, Wakschlag, Briggs-Gowan, Norton and Mittal2021; Hertzog & Nesselroade, Reference Hertzog and Nesselroade2003). Developmental processes and skill acquisition occur at rapid rates (e.g., days), are highly variable, and change depending on the construct, person, and time period assessed. Incorporating repeated measures in a risk calculator framework can facilitate the differentiation between transient maladaptive behaviors and stable impairment.
Examples from emotion dysregulation and psychosis research demonstrate that measurement of risk at only one time point can lead to biased results. Irritable behavior is a risk factor for the development of psychopathology, but only when it is pervasive, intense, and occurs in developmentally unexpectable contexts (Wakschlag et al., Reference Wakschlag, Roberts, Flynn, Smith, Krogh-Jespersen, Kaat and Davis2019). Further, pathogenic forms of dysregulated tantrums, such as destruction, are far less common, but they can be important indicators for sensitive and specific clinical identification (Wiggins et al., Reference Waller, Gardner, Viding, Shaw, Dishion, Wilson and Hyde2018). However, the presence of tantrums per se in early childhood is normative and in approximately a quarter to a third of young children do not persist into the school years (Wakschlag et al., Reference Vargas and Mittal2015). Although children who score high in irritability are at higher risk of chronic irritability and less than 1% of children with low irritability develop higher levels later in childhood (Wiggins, Briggs-Gowan, Brotman, Leibenluft, & Wakschlag, Reference Walkup, Mathews and Green2020), the children who are at intermediate risk may require repeated assessments to determine event rates for psychopathology at later time points (Wiggins et al., Reference Walkup, Mathews and Green2020).
With respect to psychosis, 15%–35% of CHR individuals between the ages of 12 and 35 years develop fully psychotic symptoms across 2 years (Fusar-Poli et al., Reference Fusar-Poli, Bonoldi, Yung, Borgwardt, Kempton, Valmaggia and McGuire2012). Of note, this is also a population that is typically in the adolescent and young-adult developmental period, as the onset of schizophrenia characteristically occurs between mid to late adolescence and late adulthood (i.e., early thirties) (Jones, Reference Jones2013; McGorry, Purcell, Goldstone, & Amminger, Reference McGlashan, Walsh and Woods2011; Patel, Leathem, Currin, & Karlsgodt, Reference Ozonoff2021; Rajji, Ismail, & Mulsant, Reference Pool, Ning, Wilkins, Lloyd-Jones and Allen2009; Walker et al., Reference Wakschlag, Roberts, Flynn, Smith, Krogh-Jespersen, Kaat and Davis2013). Given the large heterogeneity in risk factors and the dynamic developmental window for developing fully psychotic symptoms, a developmentally informed calculator leveraging multiple measurement occasions could have significant potential. For example, Worthington et al. (Reference Wiggins, Ureña Rosario, Krogh-Jespersen, MacNeill, Walkup, Smith and Wakschlag2021) included salivary cortisol as a new predictor in the established NAPLS-2 risk calculator and observed marked improvement from the original model. This finding speaks to the relevance of stress-sensitivity for emerging psychosis. However, importantly, increases in stress-sensitivity are developmentally normative during the psychosis-risk period, as cortisol increases each year in adolescence to play an important role in modulating pubertal development (Mittal & Walker, Reference Mittal and Wakschlag2019). Because sensitivity is a moving target during the risk period, there is an opportunity here for detecting obvious and conceptually relevant divergence. The CHR period is a time of clinical change (by definition, most meeting criteria have recently emergent and/or escalating attenuated psychosis symptoms). It is also a time of profound developmental change, overlaying the adolescent and young-adult period. Future risk calculators that take into account developmental trajectories (e.g., using variables that capture a subgroup of youth who are showing a steep incline in stress-sensitivity) may reveal the subgroup at highest risk for transition to psychosis (Dean et al., Reference Dean, Orr, Bernard, Gupta, Pelletier-Baldelli, Carol and Mittal2016).
Further, the degree to which a behavioral measure is dysfunctional changes depending on the developmental stage in question. As previously mentioned, irritable behavior is normative in early childhood, as this is a point in development when children are learning to regulate their own emotions and behavior with guidance from parents (Rothbart & Bates, Reference Rogers, Sylvester, Mintz, Kenley, Shimony, Barch and Smyser1998). Therefore, young children are still novices and are prone to tantrums and outbursts. These behaviors become clinically concerning when they are frequent, high in intensity, occur throughout multiple contexts, and impair other functioning (Wakschlag et al., Reference Vargas and Mittal2015). However, with age and experience, self-regulation is typically well integrated in children's emotional, cognitive, and behavioral responding. Thus, the presence of a tantrum per se at older ages may be cause for concern, where quality of behavior (e.g., dysregulated tantrum) is especially discriminative at younger ages (Wakschlag et al., Reference Wakschlag, Perlman, Blair, Leibenluft, Briggs-Gowan and Pine2010). Regarding psychosis, motor behaviors such as clumsiness may disappear as children reach other milestones, but the latent vulnerability for psychosis remains and can be detected with sensitive paradigms in adults (Mittal & Wakschlag, Reference Mittal and Wakschlag2017). Researchers can better incorporate developmental change processes into their risk algorithms by choosing meaningful sampling rates when collecting data, modeling change in a theory-informed way, and considering a risk indicator's capacity for change.
Choosing sampling rates in the context of developmental change
Many factors can be considered for determining the optimal sampling rate, including the number of time points, the spacing between time points, and when to begin sampling. If meaningful, theoretically driven choices are not made at the data collection level, the risk algorithm may yield biased results with little predictive value. First, researchers should consult theoretical models regarding the shape and timescale of change for informing their decisions (Boker & Nesselroade, Reference Boker and Nesselroade2002). If theory dictates a more complex anticipated pattern of change, more frequent measurement occasions are often required (Ram & Grimm, Reference Ram and Gerstorf2007). For instance, to model quadratic change, at least three time points are needed. Second, the sampling rate should capture the age of onset for the behavior(s) in question and, ideally, start sampling before the expected age of onset. Too large a time interval and the model will erroneously estimate the age of onset by missing when most variability occurs. For instance, Adolph et al. (Reference Adolph, Robinson, Young and Gill-Alvarez2008) found that when changing the sampling interval of infant motor behavior from daily to monthly level, there were increased errors in estimating how early infants reached stable expression. Third, data should be collected as frequently as possible. This is particularly true for binary risk indicators as distortions in their trajectories are especially susceptible to gross sampling intervals (Adolph et al., Reference Adolph, Robinson, Young and Gill-Alvarez2008). Once researchers use theory to inform the sampling rate and collect the data at these time points, they can use the scores to determine how many measurement occasions are needed to reduce false positives, the optimal interval length for optimizing risk prediction, and how early to begin sampling. Refining the risk calculator in this way will help researchers identify the minimum number of occasions requisite for sensitive and specific risk prediction, as there will come a point when additional occasions will not improve the model. Furthermore, limiting the number of assessments will mitigate the burden for both families and care providers when the calculator is ready for public health use.
Capturing the shape of developmental change
Repeated measures data open up possibilities for examining multiple change processes that can be used to predict mental health disorders. In particular, growth curve analyses can be leveraged for describing interindividual differences in intraindividual change (Ram & Grimm, Reference Ram and Gerstorf2007). Two common parameters that can be extracted from these growth curve models to predict risk are the individual's intercept (i.e., initial or average level of behavior) and their linear slope (rate of change). However, development is ripe with both continuities and discontinuities in behavior, from gradual and incremental changes in development of white matter in the brain (Giedd et al., Reference Giedd, Blumenthal, Jeffries, Castellanos, Liu, Zijdenbos and Rapoport1999) to surges in cognitive performance bookended by periods of little to no change (e.g., MacNeill, Ram, Bell, Fox, & Pérez-Edgar, Reference Luby, Allen, Estabrook, Pine, Rogers, Krogh-Jespersen and Wakschlag2018). Therefore, researchers should use their knowledge of developmental theory to consider using parameters from nonlinear trajectories (e.g., polynomial, exponential, and logistic).
Statistical methods that identify different trajectories of development can be used to form heterogeneous groups meaningful for risk prediction (Nagin, Reference Morawetz, Riedel, Salo, Berboth, Eickhoff, Laird and Kohn2005). For instance, Allen et al. (Reference Allen, Siddique, Wilkins, Shay, Lewis, Goff and Lloyd-Jones2014) used latent mixture modeling to identify five unique groups of blood pressure trajectories through early adulthood to predict the development of atherosclerosis in middle age. They found that groups with elevated blood pressure over time had greater odds of having a high score of coronary artery calcification in middle age compared with the group with low, stable blood pressure. Allen et al. (Reference Allen, Krefman, Labarthe, Greenland, Juonala, Kähönen and Lloyd-Jones2020) also examined cardiovascular health trajectory groups beginning in childhood, finding relations with subclinical atherosclerosis levels in middle age. In their study of child socioemotional development, Liu et al. (Reference Liu, Moore, Beekman, Pérez-Edgar, Leve, Shaw and Neiderhiser2018) identified six patterns of child anger from 9 months to 7 years of age: low/stable rank, average/stable rank, average/decreasing rank, average/increasing rank, high/decreasing rank, and high/stable rank. They found that children in the high/stable angry group had greater levels of internalizing and externalizing problems at age 8 years compared with those in the average/stable, average/decreasing, and high/decreasing groups. These studies demonstrate that latent group modeling can inform our understanding of the development of disease and disorder over time, where such scores could be incorporated into a mental health risk calculator.
Two other aspects of within-person change that can be considered for identifying disorder are an individual's timing and rate of change. Timing refers to how mature an individual is in relation to their same-age peers and rate is how quickly or slowly an individual progresses. Individuals can be considered early, average, or late maturers based on their progress relative to others, and they can also be considered fast, average, or slow depending on their pace from one point to another. Findings from logistic growth curve models demonstrate relations among timing and rate that inform between-person differences that have potential downstream consequences for developmental outcomes. For instance, earlier timing (at what age youth reach inflection point, or 50% growth) and faster rate (slope of curve at the inflection point) in pubertal development have been associated with youth's internalizing and externalizing problems (Marceau, Ram, Houts, Grimm, & Susman, Reference MacNeill, Ram, Bell, Fox and Pérez-Edgar2011). Across the second half of the infant's first year, the timing and rate of cognitive task performance are correlated, such that infants growing faster reach the inflection point (50% of the way between no ability and mastery) earlier. This rate of change is related to linear change in EEG power across this period (MacNeill et al., Reference Luby, Allen, Estabrook, Pine, Rogers, Krogh-Jespersen and Wakschlag2018). In a study of dysregulated fear, fear sensitivity scores (50% fear) across fear contexts were related to inhibition ratings across early childhood (Buss, Davis, Ram, & Coccia, Reference Buss, Davis, Ram and Coccia2018). In sum, how children compare to their peers in their timing of milestones and the rates at which they reach such milestones contribute to physiological change and problem behavior. There is a lack of established norms for the development of many emotional features relevant to risk for mental disorder. Thus, identifying prototypical and normative variation in milestone acquisition may be a logical next step for the examination of developmental psychopathology.
Capturing capacity to change
In addition to rate of change, risk indicators can also reflect an individual's capacity to change, or the dynamic characteristics of the individual (Ram & Gerstorf, Reference Rajji, Ismail and Mulsant2009). They may capture trait-like capabilities of the individual or their potential to change, such as their plasticity, lability, rigidity, and robustness. For instance, within-family lability in parent–adolescent connectedness has been linked to risk for depression, anxiety, antisocial behavior, drunkenness, and marijuana use 1 year later (Fosco, Mak, Ramos, LoBraico, & Lippold, Reference Fosco, Mak, Ramos, LoBraico and Lippold2019).
Importantly, some variables may not change at all during the period of study or throughout the individual's life course. These time-invariant, between-person predictors of risk include demographic characteristics, such as sex and crime status. They are typically easier to measure than psychological or social constructs that change over time, making them easy additions to the risk calculator. They also may serve as important moderators in the relations between risk indicators and mental health outcomes, as constellations of risk indicators may only inform risk for particular demographic subgroups.
Although such demographic information is important to account for and can help identify those individuals who are most at risk, it may not be advantageous to include multiple stable measures in a risk prediction model if they cannot be targets of prevention/intervention efforts. For example, the current NAPLS risk calculator includes a range of indicators, but several will never change (e.g., history of trauma, first-degree relative with psychosis). Sensitivity to change is inherent in markers that prioritize development. The possibility of testing whether interventions lower a risk calculator score is currently limited by calculators that rely heavily on static markers or those with limited potential for tapping into clinically relevant variation and change.
Further, although demographic information may be relatively easy to obtain from participants, these variables likely reflect complex social constructs that may, in turn, perpetuate disparities. For instance, race and ethnicity have been historically included in risk prediction algorithms (Obermeyer, Powers, Vogeli, & Mullainathan, Reference Obermeyer, Powers, Vogeli and Mullainathan2019), yet social factors such as racism and forms of exclusion and mistreatment are what likely hold predictive power for mental health outcomes. Risk calculators that include racial and/or ethnic categories should be appropriately justified. Racial categories alone should not be used to interpret psychological processes because they do not provide conceptual information about the behaviors and attitudes of the study participants (Helms, Jernigan, & Mascher, Reference Helms, Jernigan and Mascher2005). Rather, risk calculators should include variables that directly measure minoritized individual's successes, experiences, and exposures to racism and mistreatment. Lastly, risk calculators should be tested and validated in samples that have adequate representation across racial and ethnic groups, which can be facilitated by leveraged integrative data strategies (see the section Strategies for Generalizing Risk Prediction).
Choosing Mental Health Outcomes
As previously mentioned, both individual- and environmental-level factors contribute to mental health problems across development and thus underlie many disorders. Although this nonspecific vulnerability can pose challenges for diagnosing a particular disorder while ruling out other disorders, population-based studies can take advantage of the transdiagnostic approach by identifying risk factors that cut across multiple disorders to impact “the greatest proportion of disease burden” (Finlay-Jones et al., Reference Finlay-Jones, Varcin, Leonard, Bosco, Alvares and Downs2019). These efforts may be particularly fruitful in early childhood, a time when mental health disorders are often undifferentiated and behavioral heterogeneity within a diagnostic group is high (Kim & State, Reference Kim and State2014). Researchers implementing risk calculators in early childhood may consider examining broadband self-regulation problems as outcomes, rather than narrow disorders, as they are common and cut across several domains of functioning (Wakschlag et al., Reference Wakschlag, Roberts, Flynn, Smith, Krogh-Jespersen, Kaat and Davis2019). Self-regulation problems also widely span the cognitive control and working memory RDoC constructs.
This transdiagnostic approach may not be fitting of all risk calculators, where some may be designed to predict risk for a specific disorder with a concerted effort to rule out other highly comorbid disorders. To improve specificity, a risk calculator may choose to include a measure on which patients of the target disorder perform normally, but people with related disorders evidence impairment. Risk calculators for psychosis in older adolescents and young adults have used this narrower approach to identify conversion to psychosis in particular. The priority has been to focus on the group of individuals who meet criteria for CHR syndrome. In doing so, they may leverage behavioral paradigms that distinguish CHR pathology from other forms of psychopathology that may manifest similarly yet require different treatment (Gold et al., Reference Gold, Corlett, Strauss, Schiffman, Ellman, Walker and Mittal2020). For instance, tasks that elicit hedonic responses may differentiate schizophrenia from mood disorders.
Traditionally, risk calculator outcomes are binary, such that risk indicators predict whether an individual either has or does not have a disorder. The RDoC framework adopts a transdiagnostic approach for developmental risk prediction, where dysfunction is a continuum. Initially, it may appear that the risk calculator framework with its binary diagnosis works to constrain diagnostic systems rather than broaden or deepen them. However, criteria for developing a disorder can be shifted in different iterations of the risk calculator algorithm for comparison. Varied thresholds can be derived via psychometric methods informed by clinically meaningful thresholds. For example, the threshold for “watch and wait” monitoring by pediatricians may be lower than the threshold for psychopharmacologic treatment. Further, risk calculators do not need to be limited to whether an individual does or does not have a diagnosis. Rather, researchers could consider developing a risk calculator for predicting whether or not an individual needs treatment. Treatment thresholds may be lower than diagnosis thresholds, and individuals who do not quite meet clinical diagnosis risk criteria may still benefit or need the treatment, depending on what resources are available and the risks and side effects of such treatment.
As an alternative to binary diagnostic criteria, which may not be optimal in early childhood given heightened within-person variability in preschool psychopathology (Kotov et al., Reference Kotov, Krueger, Watson, Achenbach, Althoff, Bagby and Zimmerman2017), researchers may consider using a bifactor modeling approach to generate a general psychopathology latent factor, or “p factor” (Caspi et al., Reference Caspi, Houts, Belsky, Goldman-Mellor, Harrington, Israel and Moffitt2014). The p factor reflects shared psychopathology variance similar to the general IQ “g factor” in that it reflects the co-occurrence of all measured psychopathology symptoms. Thus, using the p factor may be a more optimal way to capture and treat clinical phenomena at young ages compared to specific binary DSM (Diagnostic and Statistical Manual of Mental Disorders) disorders (Caspi et al., Reference Caspi, Houts, Belsky, Goldman-Mellor, Harrington, Israel and Moffitt2014; Caspi & Moffitt, Reference Caspi and Moffitt2018; Martel et al., Reference Marceau, Ram, Houts, Grimm and Susman2017; Michelini et al., Reference Michelini, Gair, Tian, Dougherty, Goldstein, MacNeill and Kotov2021). Treatment approaches would focus on transdiagnostic risk factors for psychopathology that emerge from the p factor. Further, transdiagnostic manuals may be easier for clinicians to use as opposed to training on several specified treatment protocols (Walkup, Mathews, & Green, Reference Wakschlag, Tolan and Leventhal2017). We will use a p factor approach in the MHESC Study to predict probabilistic risk of impairing internalizing/externalizing psychopathology at preschool age from risk factors during the infant and toddler periods. To generate clinical thresholds for the p factor for predictive utility to clinical outcomes, scores can be validated in relation to the dichotomized diagnosis score.
Strategies for Generalizing Risk Prediction
Individuals must be relatively healthy to capture the onset of the disease within the time period measured (Pencina & D'Agostino, Reference Patel, Leathem, Currin and Karlsgodt2012). Large samples sizes help to attain the necessary power and precision for the identification of a mental health disorder. Researchers using multiple methods to chart development over time often must sacrifice large sample sizes due to time and budgetary constraints. Therefore, integrative data analysis methods, such as pooling observed data from multiple cohorts to create a synthetic cohort, may be an ideal method for retaining richness in both data quality and quantity. This integration of data from independent studies has the potential to increase enrolment of under-represented populations in research studies and cross-validate or replicate across data sets to increase generalizability. These methods can bring together scientists from different domains to springboard interdisciplinary research and solve methodological challenges, as well as to pool resources to run a large study that would otherwise be too costly or unfeasible. These integrative methods do not compromise the aims of the individual studies, but rather leverage ongoing or existing studies that are similar in nature (Chandler et al., Reference Chandler, Kahana, Fletcher, Jones, Finger, Aklin and Webb2015; Fortier et al., Reference Fortier, Raina, Van den Heuvel, Griffith, Craig, Saliba and Burton2017).
Modern statistical approaches such as the synthetic cohort approach use a full-information multivariate framework for preserving the variability in development that might otherwise be erased if time points were missing. Further, individuals can be analyzed across all time points using multiple imputation by leveraging their data from other time points (Ning et al., Reference Nielsen, Wakschlag and Norton2021; Siddique et al., Reference Shonkoff2015). The data set in the risk calculator analysis is thus a combination of the observed and imputed data so that each participant has a data point for every measure at every time point (Siddique et al., Reference Shonkoff2015).
It is inevitable that independent study cohorts do not use all the same measures for assessing the same construct. Data can be harmonized, such that study-specific variables are systematically pooled into the analytical data set, resulting in a set of common variables across all the samples that are considered the same construct. Common methods for harmonization include algorithmic transformation (combinable variables or categories), simple calibration models (e.g., converting weight from pounds to ounces), standardization models (e.g., z score transformations), latent variable models (combining two different scales into a latent variable), item response theory methods, and multiple imputation models (Curran & Hussong, Reference Curran and Hussong2009; Fortier et al., Reference Fortier, Raina, Van den Heuvel, Griffith, Craig, Saliba and Burton2017; Siddique et al., Reference Shonkoff2015). To implement this approach, each cohort must have constructs in common that serve as anchors for the harmonization process. For instance, to examine the added value of responsive parenting, each study must have a measure of responsive parenting even if that measure differs across cohorts (Curran & Hussong, Reference Curran and Hussong2009; Luby et al., Reference Luby2019).
Successful harmonization initiatives of ongoing studies require that investigators work together to determine the overarching research questions and hypotheses, what domains of study are necessary for addressing these questions, and how data will be collected in each study (Chandler et al., Reference Chandler, Kahana, Fletcher, Jones, Finger, Aklin and Webb2015). The latter includes aligning on study protocols for measures that require pre-harmonization. Although survey measures can often be standardized to reflect similar scores and behavioral observations can be re-coded (as long as they elicit the desired behavior), experimental tasks and neural measures are sensitive to even small differences in the lab environment or stimulus presentation that can have major effects on the final results. Pre-alignment of such measures may involve travel to all study sites to set up the experiment, observe protocols, and train study staff (Norton et al., Reference Norton, MacNeill, Harriott, Allen, Krogh-Jespersen, Smyser and Wakschlag2021). Throughout the study, fidelity trips across sites may also be needed to ensure protocols are maintained. Other critical steps for the harmonization process include monitoring data collection and participation, creating measures documentation and data codebooks for the integrated data set, and determining how and when data will be deposited and harmonized for analysis (Chandler et al., Reference Chandler, Kahana, Fletcher, Jones, Finger, Aklin and Webb2015). Importantly, although agreeing on common research questions and psychological constructs is necessary for the creation of a synthetic cohort, the harmonization approach provides flexibility in how data are collected to leverage unique sample characteristics. Furthermore, harmonization does not require individual investigators to compromise on the unique scope of their work (Chandler et al., Reference Chandler, Kahana, Fletcher, Jones, Finger, Aklin and Webb2015).
To test for generalizability of model performance, the risk calculator algorithms should be internally validated, as well as externally validated in an independent data set (Simon, Reference Siddique, Reiter, Brincks, Gibbons, Crespi and Brown2006). Internal validation involves splitting one's sample into derivation and test samples to estimate the reproducibility of the algorithm. Various methods of internal validation can be applied (e.g., sample splitting, bootstrapping, and cross-validation), but – regardless of the method – the test is performed in the original population (König, Malley, Weimar, Diener, & Ziegler, Reference König, Malley, Weimar, Diener and Ziegler2007). The test set can be used to define thresholds for determining the level of clinical risk. The predicted and observed risks can then be compared in the test sample to determine the accuracy of the new model. After this internal validation, researchers should validate the risk algorithms in an external data set to examine model accuracy before they apply the algorithm in real-world settings. External validation is required for rigorous assessment of the generalizability for any proposed risk prediction tool (König et al., Reference König, Malley, Weimar, Diener and Ziegler2007).
The MHESC Study is an example of the use of synthetic cohorts in risk calculator creation. The study will use a synthetic cohort approach by combining multiple, ongoing extramural cohorts at Northwestern University and Washington University in St. Louis. Investigators at each site have pioneered neurodevelopmental approaches to early childhood psychopathology, multimodal infant neuroimaging, computational and deep learning methods, and epidemiologic risk prediction. Activities in both studies were extensively pre-aligned to capture the central transdiagnostic infant risk phenotype of emotion dysregulation, as well as the central transdiagnostic clinical outcome of general psychopathology risk (internalizing/externalizing problems) using multiple methods, including surveys, behavioral observations, eye-tracking, EEG, and MRI. On-site training and oversight by study investigators have enabled each site to collect multimodal data. The MHESC Study capitalizes on the neural measures that were part of the original studies; thus, rather than needing to introduce new neural measures, we increased the number of occasions at which we collected these measures. Each study uses identical neural and behavioral measures. Although some survey measures differ, all studies capture the same main constructs of interest, allowing for harmonization of the survey data post-collection. This synthetic approach adjusts for sample heterogeneity and tests its empirical salience. Using diverse pooled samples with methods that rigorously account for such differences enhances the generalizability of the risk calculator for future public health use. The risk calculator algorithm will be internally validated, and externally validated in the normative sample collected in the Baby Connectome Project (Howell et al., Reference Howell, Styner, Gao, Yap, Wang, Baluyot and Elison2019).
Accelerating Clinical Translation to Mental Health Disorder Prevention
Risk prediction algorithms should be useful across settings, from public health (e.g., screening for a disorder) to mental health treatment (e.g., diagnosis, prognosis, clinical decision making). The final risk calculator for pragmatic clinical use should be the most reliable and parsimonious algorithm, which may be based on the intended end user, ambiguity of clinical presentation, and cost and/or risk of the intervention.
Primary care
Risk calculator algorithms, taking into account both individual- and context-level factors, can be implemented in routine care at children's pediatric check-ups to facilitate quick decision making for prevention or clinical services. Behavior in early childhood is highly variable, such that health care professionals often tell parents to watch and wait for more pronounced problem behaviors to develop and remain stable over time. Using a personalized risk estimate for their patients, health care providers can make a faster, empirically informed decision to identify and act when faced with clinical uncertainty (e.g., Ascend, https://www.telethonkids.org.au/ascend). If providers could easily differentiate youth who are at risk from those who are not, based on a risk algorithm score, they could refer patients to mental health services more efficiently. For mental health measures to be successfully implemented in routine primary care, they must balance brevity with the ability to capture modifiable, transdiagnostic risk factors for the outcome of interest (Smith et al., Reference Smith, Wakschlag, Krogh-Jespersen, Walkup, Wilson, Dishion and Shaw2019). Therefore, survey only or a combination of survey and online cognitive assessments that generate instantaneous scores may be most feasible for primary care.
Mental health specialty settings
For clinical practice, mental health risk calculators have the potential to provide empirically informed parameters for therapeutic decision making based on multilevel risk makers in a standardized, cost-effective manner. These guidelines are currently under-developed and establishing them could ameliorate clinical and public health practice. A risk calculator in a mental health specialty clinic may need to be more comprehensive than a risk calculator for primary care because individuals coming to the clinic may already have an indication that problems exist and are thus seeking further diagnostic support. In this setting, a risk calculator may also be of particular value for determining whether groups of intermediate risk (higher clinical uncertainty) need added evaluation to increase clinical certainty. In such cases, clinicians may employ intensive methods (e.g., behavioral observations, EEG, MRI) to acquire a more complete understanding of risk.
Public access
Although many children and adolescents have a primary care physician, these visits can be infrequent, particularly as children get older. Furthermore, they may not always serve as a space where parents and youth feel they can speak openly about potential mental health problems. Risk calculators could be made publicly available in order to reach vulnerable individuals and families who would benefit most. Risk calculators for physical health risk, such as the Framingham risk calculator for coronary heart disease, are already available for anyone on the internet to use. Researchers may also consider assessments for the risk calculator that can be completed in the home of any young person at risk for mental health problems. For instance, an individual can use an online platform that delivers surveys and cognitive batteries that feed into the risk calculation score. Such online distributions will shift risk prediction from the few clinical sites in town to any individual with access to the internet.
An important caveat is that once reliable developmental risk calculators are established in research, resources for accessing evidence-based treatment must be made easily available before the risk calculators can be used in practice. Without access to this infrastructure, those with a high-risk score will not receive the help they require and deserve. Unfortunately, the availability of these resources and their support in payment systems lag far behind the empirical evidence base. Further, a number of structural constraints (e.g., financial costs, parent availability, transportation) and attitudinal barriers (e.g., stigma, thoughts that problems will get better on their own) prevent timely access to treatment (Reardon et al., Reference Ram and Grimm2017). Children from low-income families and historically disadvantaged populations, as well as children whose parents have mental health issues, may disproportionately face more barriers to initial mental health treatment (e.g., Ofonedu, Belcher, Budhathoki, & Gross, Reference Obermeyer, Powers, Vogeli and Mullainathan2017). An important resource that could be developed alongside the calculator is a nonacademic, web-based dissemination platform for strategic health messaging and evidenced-based information on primary-care-based mental health risk and promotion. These materials must be accessible to the population at risk and informed by stakeholders.
Risk of stigma
The ultimate goal of many risk identification algorithms is to implement them at a large scale in existing health care systems during critical and sensitive periods when interventions are likely to be most successful (e.g., early childhood, transition to adolescence). Although early identification can prevent the onset of disorder or mitigate symptoms, labeling children with a disorder early on can lower their social and academic standing and foster stereotypes, prejudice, and discrimination (Kaushik, Kostaki, & Kyriakopoulos, Reference Kaushik, Kostaki and Kyriakopoulos2016). Seeking out this support requires that individuals and caregivers disclose their mental health problems, which can exacerbate the stress of all involved (Karnieli-Miller et al., Reference Karnieli-Miller, Perlick, Nelson, Mattias, Corrigan and Roe2013). Children and adolescents rarely seek out mental health resources on their own; therefore, their caregivers, broader family, peers, and culture play a role in cultivating expectations that can lead to or prevent treatment (Roberts, Attkisson, & Rosenblatt, Reference Reardon, Harvey, Baranowska, O'Brien, Smith and Creswell1998). In many cases, parents blame themselves for their child's disorder, which contributes to their efficacy as a parent and affective distress (Eaton, Ohan, Stritzke, & Corrigan, Reference Eaton, Ohan, Stritzke and Corrigan2016; Eaton, Stritzke, Corrigan, & Ohan, Reference Eaton, Stritzke, Corrigan and Ohan2020).
Many parents, however, still see the value in early identification. For instance, studies have found that parents of Head Start children perceive screening as important and that this early identification has benefits, such as closer parent–teacher relationships and better parenting (Nelson et al., Reference Nagin2011). Developmental risk calculator approaches must consider the bioethics behind risk communication. Culturally informed framing that focuses on a child's strengths, as opposed to a deficit-based approach, should be used to guide families through mental health treatment (Finlay-Jones et al., Reference Finlay-Jones, Varcin, Leonard, Bosco, Alvares and Downs2019). Before a risk calculator is employed, researchers should consider collecting qualitative data from individuals and/or their families to better understand the stigma surrounding mental health diagnoses, identify key barriers and motivators to seeking and receiving treatment, and understand cultural variation in norms and expectations surrounding mental health. These data can be collected through individual interviews and focus groups.
Reducing false positives
Given both the burden of intense assessments and the potential stigma attached to mental health problems, risk for false positives is a large ethical concern tethered to early identification of mental health (Ozonoff, Reference Osborne and Mittal2015). In addition to misallocation of valuable mental health resources, false positives may cause parents undue worry and difficulty in maintaining a healthy relationship with their child (Hewlett & Waisbren, Reference Hewlett and Waisbren2006). Developmental risk calculators that differentiate children and youth with a high probability of mental health problems versus those with naturally remitting problems will reduce the likelihood of false positives. This personalized approach is designed to limit over-pathologizing of normative variation, particularly in early development. Prevention/intervention resources may be limited and those who will benefit the most (e.g., those at highest risk) should be the receivers of such resources. The risk calculator model should have high discrimination ability to form such a subgroup (Steyerberg et al., Reference Smith, Wakschlag, Krogh-Jespersen, Walkup, Wilson, Dishion and Shaw2013). High discrimination is also essential if prevention efforts could potentially have adverse effects on those individuals with lower risk of future diagnosis (Cannon et al., Reference Cannon, Yu, Addington, Bearden, Cadenhead, Cornblatt and Kattan2016). Importantly, treatment in the form of medication, when inappropriately applied, can have harmful consequences.
When resources are abundant and/or cost-effective, one may consider lowering the threshold for outcome identification, particularly if the intervention can be beneficial for all or side effects are benign (Cannon et al., Reference Cannon, Yu, Addington, Bearden, Cadenhead, Cornblatt and Kattan2016). For instance, strategies for directly improving self-regulation may be beneficial for all children (Smith & Dishion, Reference Smith, Berkel, Jordan, Atkins, Narayanan, Gallo and Bruening2014). Interventions centered on positive parenting in the domains of positive behavior support, monitoring and limit setting, and family relationship building can enhance children's self-regulation, ultimately leading to the prevention of later internalizing and externalizing disorders (Smith & Dishion, Reference Smith, Berkel, Jordan, Atkins, Narayanan, Gallo and Bruening2014). Although the concern of false positives is an important one for reducing burden at the systems level, interventions that improve parenting and/or promote children's self-regulation are likely to be beneficial for all children and families regardless of risk status.
Conclusions
This conceptual paper has provided a framework for incorporating a developmental perspective into risk calculator algorithms for use in a wide range of settings to promote child and adolescent mental health. The recommendations highlighted throughout the paper are summarized in Figure 1. The development and broad dissemination of developmental mental health risk calculators will help primary care physicians and specialty clinicians provide personalized risk estimates to their patients and their families to assist them in making complicated treatment (prevention and intervention) decisions. The risk calculator approach incorporates several key RDoC tenets (e.g., utilizing multiple units of analysis as predictors across each of the domains, examining transdiagnostic outcomes). Adopting a developmental risk calculator approach can help translate RDoC science into clinically useful metrics to strengthen (a) our understanding of how individual and environmental risk indicators predict the developmental course of mental health, (b) our prevention/intervention efforts by reaching children earlier, and (c) the knowledge base and decision making for children and youth with high clinical uncertainty. A developmental risk calculator that is pragmatic, personalized, and developmentally sensitive can optimize the characterization of neurodevelopmental vulnerability for identification of mental health risk to improve prevention and early intervention toward enhancing child and adolescent wellbeing.
Summary of recommendations for generating developmentally informed mental health risk calculators.
Funding Statement
Preparation for this manuscript was supported in part by National Institute of Mental Health grants R01MH121877 (to Luby, Rogers, and Wakschlag) and R01 MH120088 (to Mittal).
Conflicts of Interest
None.
