Introduction
Irritability – an elevated proneness to anger relative to peers (Brotman, Kircanski, Stringaris, Pine, & Leibenluft, Reference Brotman, Kircanski, Stringaris, Pine and Leibenluft2017) – is increasingly the focus of clinical and translational research (Brotman et al., Reference Brotman, Kircanski, Stringaris, Pine and Leibenluft2017; Leibenluft, Reference Leibenluft2017). The inclusion of frustrative nonreward, a related construct, in the National Institute of Mental Health's Research Domain Criteria (RDoC) (Insel et al., Reference Insel, Cuthbert, Garvey, Heinssen, Pine, Quinn and Wang2010) has facilitated recent work in irritability: preliminary evidence suggests that behavioral and neural function in the context of frustrative nonreward is impaired in youth with high levels of irritability (Meyers, DeSerisy, & Roy, Reference Meyers, DeSerisy and Roy2017; Tseng et al., Reference Tseng, Deveney, Stoddard, Kircanski, Frackman, Yi and Leibenluft2019). Consistent with the RDoC principles, irritability is a transdiagnostic symptom in developmental psychopathology that spans a continuum of severity. Irritability is common in many pediatric mood, anxiety, and disruptive behavior disorders and can confer risk for multiple negative outcomes in adulthood (Vidal-Ribas, Brotman, Valdivieso, Leibenluft, & Stringaris, Reference Vidal-Ribas, Brotman, Valdivieso, Leibenluft and Stringaris2016). Severe, impairing irritability is the defining symptom of disruptive mood dysregulation disorder (DMDD) (American Psychiatric Association, 2013) and predicts later unipolar depression, anxiety, suicidality, and functional impairment (Brotman et al., Reference Brotman, Schmajuk, Rich, Dickstein, Guyer, Costello and Leibenluft2006; Copeland, Angold, Costello, & Egger, Reference Copeland, Angold, Costello and Egger2013; Orri, Perret, Turecki, & Geoffroy, Reference Orri, Perret, Turecki and Geoffroy2018; Stringaris, Vidal-Ribas, Brotman, & Leibenluft, Reference Stringaris, Vidal-Ribas, Brotman and Leibenluft2018; Vidal-Ribas et al., Reference Vidal-Ribas, Brotman, Valdivieso, Leibenluft and Stringaris2016).
Notwithstanding research progress on the brain and behavioral mechanisms of irritability, important gaps in the measurement and phenotyping of irritability remain and may impact further advancements. The emergence of new technologies assessing irritability symptoms naturalistically in real time allows investigators to answer phenomenological questions and to link laboratory measures and clinical interventions with youth's real-world functioning. This is particularly important for pediatric irritability as studies suggest that irritable youth may retrospectively underreport levels of symptoms relative to their parents (Pan & Yeh, Reference Pan and Yeh2019; Stoddard et al., Reference Stoddard, Stringaris, Brotman, Montville, Pine and Leibenluft2014). In the current study we developed a naturalistic, smartphone-based measurement of pediatric irritability using ecological momentary assessment (EMA) (Myin-Germeys et al., Reference Myin-Germeys, Oorschot, Collip, Lataster, Delespaul and van Os2009; Russell & Gajos, Reference Russell and Gajos2020) to answer specific questions about irritability. Here, we evaluate EMA-assessed irritability with respect to feasibility, convergent validity with other levels of analysis, and transdiagnostic phenomenology in a clinical sample.
Irritability is a multifaceted clinical construct. Many conceptualizations of irritability, including its formulation in DMDD, include both a “phasic” component of temper outbursts and a “tonic” component of irritable mood. Temper outbursts refer to acute expressions of anger, verbal aggression, and/or physical aggression, typically in response to triggering stimuli (Avenevoli, Blader, & Leibenluft, Reference Avenevoli, Blader and Leibenluft2015). Irritable mood refers to lower-intensity but longer-lasting crankiness, grouchiness, or annoyance (Cardinale et al., Reference Cardinale, Freitag, Brotman, Pine, Leibenluft and Kircanski2021; Moore et al., Reference Moore, Lapato, Brotman, Leibenluft, Aggen, Hettema and Roberson-Nay2019). Recent studies have compared tonic and phasic irritability; for example, Cardinale et al. (Reference Cardinale, Freitag, Brotman, Pine, Leibenluft and Kircanski2021) explored the expression of phasic and tonic irritability in a sample of youth with varying levels of attention-deficit/hyperactivity disorder (ADHD) symptoms. The findings showed that phasic, but not tonic, irritability was significantly associated with ADHD symptoms. In another study, Moore et al. (Reference Moore, Lapato, Brotman, Leibenluft, Aggen, Hettema and Roberson-Nay2019) reported that phasic and tonic irritability were influenced by different, unrelated subsets of genetic variants. EMA may be well-suited to further examine phasic versus tonic manifestations by capturing this multifaceted, complex symptomatology of irritability through the use of multiple assessment items.
As noted, frustrative nonreward is a translational research construct in the RDoC framework that is closely related to irritability (Insel et al., Reference Insel, Cuthbert, Garvey, Heinssen, Pine, Quinn and Wang2010). Frustrative nonreward was originally conceptualized in animal research (Amsel, Reference Amsel1958), defined as a response to the omission of an expected reward involving increased motor activity and aggression (e.g., Burokas, Gutierrez-Cuesta, Martin-Garcia, & Maldonado, Reference Burokas, Gutierrez-Cuesta, Martin-Garcia and Maldonado2012; Deveney et al., Reference Deveney, Connolly, Haring, Bones, Reynolds, Kim and Leibenluft2013; Leibenluft, Reference Leibenluft2011). Irritability relates to frustrative nonreward as irritable symptoms are often triggered by blocked goal attainment (Brotman et al., Reference Brotman, Kircanski, Stringaris, Pine and Leibenluft2017; Kircanski et al., Reference Kircanski, Craske, Averbeck, Pine, Leibenluft and Brotman2019). In multiple laboratory studies, higher levels of irritability in youth have been linked to altered neural (e.g., Deveney et al., Reference Deveney, Connolly, Haring, Bones, Reynolds, Kim and Leibenluft2013; Grabell et al., Reference Grabell, Li, Barker, Wakschlag, Huppert and Perlman2018; Perlman et al., Reference Perlman, Jones, Wakschlag, Axelson, Birmaher and Phillips2015; Tseng et al., Reference Tseng, Deveney, Stoddard, Kircanski, Frackman, Yi and Leibenluft2019) and subjective (e.g., Deveney et al., Reference Deveney, Connolly, Haring, Bones, Reynolds, Kim and Leibenluft2013; Rich et al., Reference Rich, Schmajuk, Perez-Edgar, Fox, Pine and Leibenluft2007, Reference Rich, Carver, Holroyd, Rosen, Mendoza, Cornwell and Leibenluft2011) responses to frustration.
To date, irritability has been quantified predominantly using retrospective questionnaires and clinician inventories (e.g., Haller, Kircanski, et al., Reference Haller, Kircanski, Stringaris, Clayton, Bui, Agorsor and Brotman2020; Lindgren & Koeppl, Reference Lindgren, Koeppl and Prinz1987; Stringaris, Goodman, et al., Reference Stringaris, Goodman, Ferdinando, Razdan, Muhrer, Leibenluft and Brotman2012; Wakschlag et al., Reference Wakschlag, Briggs-Gowan, Choi, Nichols, Kestler, Burns and Henry2014). While useful, the limitations of these methods include potential memory biases and social desirability (Bradburn, Rips, & Shevell, Reference Bradburn, Rips and Shevell1987). The naturalistic, momentary nature of EMA may help to improve ecological validity and reduce memory biases and social desirability. EMA also has the ability to capture more fine-grained within-person variability over time (Russell & Gajos, Reference Russell and Gajos2020). For example, previous EMA studies in pediatric samples have successfully probed within-person variability across complex clinical phenomena such as reactive and proactive aggression in youth with negative emotional lability (Slaughter, Leaberry, Fogleman, & Rosen, Reference Slaughter, Leaberry, Fogleman and Rosen2020), anger following violence exposure (Odgers & Russell, Reference Odgers and Russell2017), and emotion regulation in youth with ADHD (Babinski & Welkie, Reference Babinski and Welkie2020).
Previous research links EMA-based and converging laboratory findings, underscoring EMA as a compelling methodology to probe irritability in line with the RDoC initiative. In a recent study from our group, Smith et al. (Reference Smith, Kircanski, Brotman, Do, Subar, Silk and Pine2019) examined EMA ratings in relation to functional magnetic resonance imaging (fMRI) data in a pediatric anxiety sample (n = 18). Participants rated the subjective valence of recent interactions with peers and completed fMRI task probing error monitoring in social versus nonsocial contexts. The results indicated significant associations between EMA ratings and neural responses to social errors. Other studies have linked EMA-acquired data with laboratory-based measures of pupillary reactivity (Silk et al., Reference Silk, Stroud, Siegle, Dahl, Lee and Nelson2012), attention to threat (Price et al., Reference Price, Allen, Silk, Ladouceur, Ryan, Dahl and Siegle2016), and neural responses to monetary reward (Flores et al., Reference Flores, Eckstrand, Silk, Allen, Ambrosia, Healey and Forbes2018). As described later in the paper, this is the first study to test associations between EMA-assessed irritability and laboratory-assessed frustrative nonreward.
In the present study, we evaluated the feasibility and validity of EMA measures of pediatric irritability. We utilized a transdiagnostic sample of youth aged 8–18 years diagnosed with primary DMDD, ADHD, or anxiety disorders (ANX), along with healthy volunteers (HV). Irritability is common in both ADHD (e.g., Mulraney et al., Reference Mulraney, Schilpzand, Hazell, Nicholson, Anderson, Efron and Sciberras2016; Shaw, Stringaris, Nigg, & Leibenluft, Reference Shaw, Stringaris, Nigg and Leibenluft2014) and ANX (e.g., Comer, Pincus, & Hofmann, Reference Comer, Pincus and Hofmann2012; Cornacchio, Crum, Coxe, Pincus, & Comer, Reference Cornacchio, Crum, Coxe, Pincus and Comer2016; Stoddard et al., Reference Stoddard, Stringaris, Brotman, Montville, Pine and Leibenluft2014). Thus, there may be some shared mechanisms of irritability as a symptom dimension across diagnoses (Hommer et al., Reference Hommer, Meyer, Stoddard, Connolly, Mogg, Bradley and Brotman2014; Kircanski et al., Reference Kircanski, White, Tseng, Wiggins, Frank, Sequeira and Brotman2018; Stoddard et al., Reference Stoddard, Tseng, Kim, Chen, Yi, Donahue and Leibenluft2017; Tseng et al., Reference Tseng, Deveney, Stoddard, Kircanski, Frackman, Yi and Leibenluft2019).
First, we evaluated the convergent validity of EMA reports of irritability with gold-standard questionnaires and clinician instruments. We predicted that EMA ratings of irritability symptoms would correlate significantly with gold-standard assessments, to a similar degree as has been shown for anxiety symptoms (Smith et al., Reference Smith, Kircanski, Brotman, Do, Subar, Silk and Pine2019). Second, as an initial test of convergence across levels of analysis, we tested associations of EMA-reported irritability symptoms with frustration ratings obtained during a laboratory Stop Signal Task (SST) (Logan, Cowan, & Davis, Reference Logan, Cowan and Davis1984). The SST is a canonical assessment of inhibitory control (Buzzell et al., Reference Buzzell, Troller-Renfree, Barker, Bowman, Chronis-Tuscano, Henderson and Fox2017; Cardinale et al., Reference Cardinale, Subar, Brotman, Leibenluft, Kircanski and Pine2019; Nigg, Reference Nigg2017), and modifications of this task have been used to elicit frustrative nonreward (Scheinost et al., Reference Scheinost, Dadashkarimi, Finn, Wambach, MacGillivray, Roule and Tseng2021; Tseng et al., Reference Tseng, Deveney, Stoddard, Kircanski, Frackman, Yi and Leibenluft2019). We predicted that EMA ratings of irritability symptoms would correlate significantly with SST frustration ratings. Together, these analyses serve as a critical first step toward validating the current EMA as a tool for quantifying irritability symptoms, which can then be further examined in relation to behavioral, physiological, and neural circuit levels of analysis. Third, to provide naturalistic evidence for irritability as a transdiagnostic construct, we leveraged the sample composition to compare irritability symptoms across diagnostic groups. We focused on the levels of and interrelations among irritability symptoms (e.g., temper outbursts, irritable mood) across diagnoses. We hypothesized that irritability symptoms would show a graduated pattern across diagnostic groups, being highest and most strongly interrelated in participants with DMDD, followed by participants with ADHD and ANX, and lastly by HV.
Given our prior use of EMA in pediatric ANX (Smith et al., Reference Smith, Kircanski, Brotman, Do, Subar, Silk and Pine2019) and the documented links between irritability and anxiety, we included several EMA-reported anxiety symptoms as relevant comparisons with which to benchmark the irritability results. Thus, parallel to the irritability analyses, we evaluated the convergent validity of EMA-assessed anxiety symptoms and levels of anxiety symptoms across diagnostic groups.
Method
Participants
To examine irritability transdiagnostically, recruitment focused on youth aged 8–18 years (M age = 12.55 years, SD = 2.53 years; 75.20% male) meeting criteria for a primary diagnosis of DMDD (n = 26), ADHD (n = 28), or ANX (generalized, social, and/or separation anxiety disorder) (n = 28), and youth with no psychiatric diagnosis (HV) (n = 27). A total of 109 youth thus participated in the study. Demographic and clinical characteristics by diagnostic group are provided in Table 1 (see Table S1 of the Supplementary Material for details of all psychiatric comorbidities).
Demographic and clinical characteristics by diagnostic group
Note: ADHD = attention-deficit/hyperactivity disorder; ANX = anxiety disorder; ARI = Affective Reactivity Index; CL-ARI = Clinician Affective Reactivity Index; DMDD = disruptive mood dysregulation disorder; EMA = ecological momentary assessment; HV = healthy volunteers; PARS = Pediatric Anxiety Rating Scale; SCARED = Screen for Child Anxiety Related Emotional Disorders
a IQ data were missing for 12 participants
Participants were recruited via direct mailings and online advertisements. Participants were evaluated for eligibility and diagnostic status by a doctoral- or master's-level clinician using the Schedule for Affective Disorders and Schizophrenia for School-Age Children – Present and Lifetime version (Kaufman et al., Reference Kaufman, Birmaher, Brent, Rao, Flynn, Moreci and Ryan1997). Primary diagnosis was based on the chief presenting complaint and clinician judgment of the most severely impairing diagnosis. Recruitment focused on youth whose irritability was chronic and not clearly related to another ongoing or episodic diagnosis (e.g., major depressive disorder, bipolar disorder). Exclusion criteria were: IQ < 70, assessed using the Wechsler Abbreviated Intelligence Scale (Wechsler, Reference Wechsler1999); a diagnosis of posttraumatic stress disorder, schizophrenia, neurological disorder, developmental disorder, bipolar disorder, or obsessive–compulsive disorder; a current major depressive episode; or substance abuse within 3 months of participation. Participants with a primary anxiety disorder were additionally excluded if they were taking psychotropic medication, based on their simultaneous recruitment for a study of pediatric anxiety that involved treatment. HV participants were free of any current or past psychiatric disorder. Two participants were excluded from the current study due to EMA compliance below the predetermined cutoff (less than five prompts completed). For the current study, ANX and HV participants were drawn from a larger sample that completed the same EMA and were reported in Smith et al. (Reference Smith, Kircanski, Brotman, Do, Subar, Silk and Pine2019) (sample overlap: ANX n = 15; HV n = 7). ANX and HV participants were randomly selected from this larger sample with the constraint that age and distribution by sex did not differ significantly between either group and the DMDD or ADHD group. Participants and their parents provided written assent and consent, respectively. Participants were compensated and offered a monetary bonus for completing ≥ 75% of prompts. The study was approved by the National Institute of Mental Health Institutional Review Board.
Procedure
Participants and parents completed a clinical evaluation visit before being enrolled in the research protocol. Participants subsequently completed a standardized EMA training session during which a research assistant familiarized the participant with the smartphone and protocol and reviewed each EMA item by guiding the participant through a practice prompt. To enhance feasibility and compliance, for each day during the upcoming 7 days of EMA, participants preselected 60-min periods during standardized time windows within which prompts would be delivered: morning/before school (6:00–9:00 a.m.), afternoon/after school (3:00–6:00 p.m.), and evening/before bedtime (7:00–10:00 p.m.). The actual prompt times were randomized within these time periods. EMA was administrated via ReTAINE technology (http://retaine.org/). Participants used either a personal or a study-provided smartphone (iPhone 7). Access on study-provided smartphones was limited to the website that delivered the items.
Following the training session, participants were prompted three times per day for seven consecutive days. At each prompt, participants received a text message with a link to the website through which the items were delivered. Once the prompt was received, participants had 60 min to complete the assessment before it expired and was considered incomplete. At the end of the 7-day period, participants and their parents completed retrospective questionnaires assessing psychopathology symptoms, including irritability. With the aim of evaluating convergent validity of the EMA items, every attempt was made to assess symptoms via questionnaires as close as possible to the EMA period. Of note, parents of participants with DMDD and ADHD also completed EMA as part of a separate project. However, to enable direct comparisons across diagnostic groups, only youth-report EMA data were included in the current study.
Measures
EMA measures
The full EMA protocol assessed various dimensions of mood and anxiety symptoms and their situational context. The current analyses examined irritability and anxiety symptoms.
Irritability symptoms
Irritability symptoms were assessed using four items (temper outburst, irritable mood, frustration, and momentary anger), with a focus on irritability chronometry (e.g., phasic vs. tonic irritability). Aiming to capture irritability throughout the entire day, three items assessed symptoms since the previous prompt (thereby encompassing the entire day in total): (a) temper outburst – “Since the last beep, I felt really, really angry and out of control” (categorical yes or no); (b) irritable mood – “Since the last beep, aside from being really, really angry and out of control, I was feeling generally grouchy or cranky” (5-point Likert scale, 1 = none of the time; 5 = the whole time); (c) frustration – “Since the last beep, I felt frustrated” (5-point Likert scale, 1 = not at all; 5 = extremely). One item assessed irritability at the time of the prompt: (d) momentary anger – “At the time of the beep, I felt annoyed or angry” (5-point Likert scale, 1 = not at all; 5 = extremely).
To enable further evaluation of convergent validity, we also included one item querying impairment related to irritability since the previous prompt, as an important component of in-clinic assessments. This item was irritability-related impairment – “Since the last beep, my grouchy mood, or being angry and out of control, got me in trouble” (categorical “with my parent,” “at school,” and/or “with other kids” [multiple selections allowed], or “none of the above”). The total number of domains of impairment was calculated for each participant (range = 0–3).
Anxiety symptoms
For the present analyses, anxiety symptoms were indexed using three items (anxious affect, anxious avoidance, and momentary anxiety). Parallel to the irritability items, anxiety symptoms were assessed either since the last prompt or at the time of the prompt. Two items assessed anxiety symptoms since the previous prompt: (a) anxious affect – “Since the last beep, I felt worried or scared” (5-point Likert scale, 1 = not at all; 5 = extremely); (b) anxious avoidance – “Since the last beep, I avoided doing things because I felt worried or scared” (categorical yes or no). One item assessed anxiety at the time of the prompt: (c) momentary anxiety – “At the time of the beep, I felt worried or scared” (5-point Likert scale, 1 = not at all; 5 = extremely).
Youth- and parent-report questionnaires
Irritability symptoms
After completing the EMA protocol, irritability symptoms over the past week were assessed using Affective Reactivity Index (ARI) 1-Week version completed separately by youth and parents (Stringaris, Goodman, et al., Reference Stringaris, Goodman, Ferdinando, Razdan, Muhrer, Leibenluft and Brotman2012). The ARI includes six items related to irritable feelings and behaviors that are computed in the total score, and one additional item assessing impairment due to irritability. Each item uses a 3-point Likert scale ranging from 0 = not true to 2 = certainly true. The ARI has demonstrated strong construct validity and reliability (DeSousa et al., Reference DeSousa, Stringaris, Leibenluft, Koller, Manfro and Salum2013; Mulraney, Melvin, & Tonge, Reference Mulraney, Melvin and Tonge2014; Stringaris, Goodman, et al., Reference Stringaris, Goodman, Ferdinando, Razdan, Muhrer, Leibenluft and Brotman2012). In the current study, internal consistency reliability was .89 for the ARI youth-report and .92 for the ARI parent-report. The ARI was completed as close as possible to the completion of EMA (number of days from completing EMA: ARI youth-report Mdn = 3.00, M = 8.14, SD = 14.68; ARI parent-report Mdn = 3.00, M = 7.50, SD = 13.70).
Anxiety symptoms
After completing the EMA protocol, anxiety symptoms were assessed using the Screen for Child Anxiety Related Emotional Disorders (SCARED) parent- and youth-report forms, which probe symptoms over the past 3 months (Birmaher et al., Reference Birmaher, Brent, Chiappetta, Bridge, Monga and Baugher1999). The SCARED includes 41 items grouped into categories of panic/somatic symptoms, generalized anxiety, social anxiety, separation anxiety, and school avoidance. Each item uses a 3-point Likert scale ranging from 0 = not true to 2 = very true. The SCARED has demonstrated strong construct validity and reliability (Birmaher et al., Reference Birmaher, Khetarpal, Brent, Cully, Balach, Kaufman and Neer1997, Reference Birmaher, Brent, Chiappetta, Bridge, Monga and Baugher1999). In the current study, internal consistency reliability was .85 for the SCARED youth-report and .84 for the SCARED parent-report. The SCARED was completed as close as possible to the completion of EMA (number of days from completing EMA: SCARED youth-report Mdn = 3.00, M = 8.17, SD = 14.64; SCARED parent-report Mdn = 3.00, M = 7.87, SD = 13.85). One participant (ADHD) was missing the SCARED youth-report and two participants (HV) were missing the SCARED parent-report.
Clinician-administered ratings
Irritability symptoms
After completing the EMA protocol, participants in the DMDD and ADHD groups were further assessed for irritability symptoms over the past week using the Clinician Affective Reactivity Index (CL-ARI) (Haller, Kircanski, et al., Reference Haller, Kircanski, Stringaris, Clayton, Bui, Agorsor and Brotman2020). Only the DMDD and ADHD groups completed the CL-ARI due to constraints of the research protocols for the different diagnostic groups. The CL-ARI is a 12-item semi-structured interview with both parent and child assessing irritability along three subscales: temper outbursts, irritable mood, and impairment. Items are scored on Likert scales. A total score is completed by weighting each subscale equally (transforming to a proportion of the total possible score for the subscale), averaging across the three subscales, and multiplying by 100 (score range = 0–100). The CL-ARI has demonstrated strong validity and adequate reliability (Haller, Kircanski, et al., Reference Haller, Kircanski, Stringaris, Clayton, Bui, Agorsor and Brotman2020). The CL-ARI was completed as close as possible to the completion of EMA (number of days from completing EMA: Mdn = 2.00, M = 3.37, SD = 9.14). Three participants (ADHD) were missing the CL-ARI.
Anxiety symptoms
After completing the EMA protocol, all participants were further assessed for anxiety symptoms over the past week using the clinician-rated Pediatric Anxiety Rating Scale (PARS) (Research Units on Pediatric Psychopharmacology Anxiety Study Group, 2002). The PARS is a 50-item symptom checklist encompassing categories of physical symptoms, generalized anxiety, social anxiety, separation anxiety, and specific phobia. Each symptom is scored as present or absent. Endorsed symptoms are then rated by the clinician on dimensions of severity, frequency, avoidance, and interference using Likert scales. A total score is calculated by summing most items. The PARS has demonstrated strong psychometric properties (Research Units on Pediatric Psychopharmacology Anxiety Study Group, 2002). The PARS was completed within 3 months of the completion of EMA (number of days from completing EMA: Mdn = 3.00, M = 15.65, SD = 29.32). Four participants (two HV and two ADHD) were missing the PARS.
Laboratory task frustration ratings
To begin to probe the associations of EMA-assessed symptoms with laboratory-assessed frustrative nonreward, the current study utilized data from a subset of participants (n = 79; M age = 12.60 years, SD = 2.43 years; 73.70% male) across the groups (DMDD n = 16, ADHD n = 23, ANX n = 15, HV n = 25, total n = 79) who completed a behavioral SST in the laboratory (Verbruggen, Logan, Liefooghe, & Vandierendonck, Reference Verbruggen, Logan, Liefooghe and Vandierendonck2008) (number of days from completing EMA: Mdn = 40.00, M = 75.86, SD = 99.66). The SST has been used widely to assess response inhibition in youth (Alderson, Rapport, & Kofler, Reference Alderson, Rapport and Kofler2007; Lipszyc & Schachar, Reference Lipszyc and Schachar2010) and, as noted, has been leveraged as one way to elicit frustration (Scheinost et al., Reference Scheinost, Dadashkarimi, Finn, Wambach, MacGillivray, Roule and Tseng2021; Tseng et al., Reference Tseng, Deveney, Stoddard, Kircanski, Frackman, Yi and Leibenluft2019). Briefly, the task includes “go” trials and “stop” signal trials. On go trials, participants are presented with a standard two-choice reaction time task (e.g., an X or O) and instructed to press a corresponding key for each target as it appears on the screen. Stop signal trials randomly occur on 25% of the total trials, in which participants are instructed to inhibit pressing a key in response to the target if they hear an auditory stop signal. Critically, the onset time of the stop signal is adjusted based on the participant's performance in order to titrate accuracy on inhibition trials as close as possible to 50% (for details, see Logan, Schachar, & Tannock, Reference Logan, Schachar and Tannock1997). The SST consisted of five experimental blocks. After each block, participants rated their subjective level of frustration on a 9-point Likert scale (1 = not at all; 9 = extremely). Average frustration level across the blocks was then computed.
Data Analysis
Given the nested structure of the EMA data (prompts within participants), statistical analyses were performed using multilevel modeling (HLM software, version 8.0; Raudenbush, Bryk, Cheong, & Congdon, Reference Raudenbush, Bryk, Cheong and Congdon2019). Level 1 included the within-subject repeated data and Level 2 included the between-subject variables. Level 1 continuous predictors were person-centered and Level 2 continuous predictors were grand-mean centered. Level 1 and Level 2 categorical predictors were uncentered. The models specified outcome variables as continuous with the exception of temper outbursts and anxious avoidance, which were analyzed using logistic multilevel modeling. Where noted below, standardized variables at Level 1 and Level 2 were computed to enable more direct interpretation of coefficients (Snijders & Bosker, Reference Snijders and Bosker2012). All models included a random intercept and, where applicable, a random slope. To handle missing data, the HLM software performed listwise deletion of missing data at Level 1 when running each analysis (i.e., within-person missing data were deleted when the analysis was conducted, as opposed to when the data set was constructed). This enabled us to include all available time points for each participant for each analysis.
(a) Convergent validity of EMA items: A series of analyses tested the associations between each EMA item assessing irritability or anxiety and the external measure of irritability (e.g., ARI) or anxiety (e.g., SCARED), respectively.
(b) Associations with laboratory task frustration ratings: A series of analyses tested the associations between each EMA item and frustration during the SST.
(c) Transdiagnostic phenomenology of symptoms: A series of means-as-outcomes models (Raudenbush & Bryk, Reference Raudenbush and Bryk2002) was conducted for each EMA item to examine group differences in levels of EMA-assessed irritability and anxiety. In addition, a series of analyses evaluated within-prompt associations between the different irritability symptoms.
All the models are described in full later in the paper. Supplementary analyses evaluating within-prompt associations between irritability and anxiety symptoms are detailed in the Supplementary Material.
Given the multiple tests conducted, the results for each series of analyses were subjected to false discovery rate (FDR) correction (Benjamini–Hochberg procedure) with the expected proportion of false positives set to q = .05. Thus, the results below represent raw coefficients and standard errors from the multilevel modeling and FDR-corrected p values. Note that the Benjamini–Hochberg method is a step-up procedure in which all corrected values below q = .05 are considered significant.
Results
Participant characteristics
As shown in Table 1, there were no significant differences across the diagnostic groups in age, F (3, 106) = 0.56, p = .644, IQ, F (3, 93) = 0.15, p = .928, or distribution by sex, χ2 (3) = 3.03, p = .388, race, likelihood ratio χ2 (9) = 7.77, p = .557, or ethnicity, χ2 (3) = 5.13, p = .162. We examined potential associations between each of the irritability- or anxiety-related EMA items and age. Table S2 of the Supplementary Material presents the full results of these models, which were nonsignificant.
As expected, symptom questionnaire scores and clinical ratings differed significantly across groups – ARI youth-report, F (3, 105) = 12.50, p < .001; parent-report, F (3, 105) = 35.41, p < .001. Post-hoc tests indicated that the DMDD group had consistently higher scores than all other groups, and the ADHD and ANX groups had consistently higher scores than the HV group (all p values ≤ .025). The CL-ARI was completed with the DMDD and ADHD groups only; the DMDD group again exhibited higher scores than the ADHD group, t (49) = 3.63, p = .001. For the SCARED youth-report, omnibus F (3, 104) = 7.33, p < .001, SCARED parent-report, omnibus F (3, 103) = 19.73, p < .001, and PARS, omnibus F (3, 104) = 53.36, p < .001, post-hoc tests indicated that the ANX group had consistently higher scores than the ADHD and HV groups, and the DMDD group had consistently higher scores than the HV group (all p values ≤ .010). Of note, the ANX group had higher scores on the SCARED parent-report and PARS than the DMDD group (both p values ≤ .002) but did not differ significantly from the DMDD group in SCARED youth-report scores (p = .074).
Supporting feasibility of the EMA protocol, participant compliance with the protocol was high (proportion of prompts completed: M = 78.94%, SD = 16.02%). Compliance rate did not differ significantly across groups, F (3,106) = 2.08, p = .107.
Convergent validity of EMA items
To evaluate the convergent validity of each EMA item, a series of models specified each EMA item as the outcome variable at Level 1 and each external measure (youth-report, parent-report, or clinician-report) as the predictor at Level 2. Variables were standardized at both levels. However, the categorical variables “temper outburst” and “anxious avoidance” were not standardized. Consistent with the work of Smith et al. (Reference Smith, Kircanski, Brotman, Do, Subar, Silk and Pine2019), the models also included the number of days between completion of EMA and the external measure as a covariate at Level 2.
At Level 1, EMA item ratingij denotes the rating for participant j at prompt i, β0j represents the within-person mean rating, and rij denotes the within-person random effect. At Level 2, γ00 denotes the average rating in the sample, γ01 denotes the between-person association of the score on the external measure with the mean rating, and u 0j denotes the between-person random effect.
Irritability symptoms
Supporting convergent validity, as shown in Table 2, the ratings on all four EMA items assessing irritability symptoms (temper outburst, irritable mood, frustration, momentary anger) were significantly associated with ARI youth-report total score, ARI parent-report total score, and CL-ARI total score (all standardized coefficients ≥ .25, all corrected p values ≤ .015). Further, ratings on the EMA item irritability-related impairment were significantly associated with scores on the impairment item on both the ARI youth- and parent-report and the impairment subscale of the CL-ARI (all standardized coefficients ≥ .13, all corrected p values ≤ .014).
Convergent validity of EMA-assessed irritability symptoms
Note: ARI = Affective Reactivity Index; CL-ARI = Clinician Affective Reactivity Index; EMA = ecological momentary assessment; FDR = false discovery rate; SE = standard error. Results reflect models in which all continuous variables were standardized
a Models included the disruptive mood dysregulation disorder (DMDD) and attention-deficit/hyperactivity disorder (ADHD) groups only; three participants (ADHD) were missing the CL-ARI
Anxiety symptoms
To a similar degree, as shown in Table S3 of the Supplementary Material, ratings on all three EMA items assessing anxiety (anxious affect, anxious avoidance, momentary anxiety) were significantly associated with SCARED youth- and parent-report total scores and PARS total score (all standardized coefficients ≥ .20, all corrected p values ≤ .009).
Associations with laboratory task frustration ratings
A series of models was conducted to examine associations between EMA ratings and frustration ratings obtained while a subset of participants (n = 79) completed the SST. The Level 1 and Level 2 equations were analogous to those examining convergent validity, and continuous variables were again standardized at both levels. As shown in Table S4 of the Supplementary Material, average frustration ratings during the SST were significantly associated with mean levels of all EMA-assessed irritability symptoms, including temper outburst, irritable mood, frustration, and momentary anger (all standardized coefficients ≥ .24, all corrected p values ≤ .007). In addition, average frustration ratings were significantly associated with mean levels of all EMA-assessed anxiety symptoms, including anxious affect, anxious avoidance, and momentary anxiety (all standardized coefficients ≥ .23, all corrected p values ≤ .012).
Transdiagnostic phenomenology of symptoms
Symptom levels across diagnoses
To examine group differences in levels of EMA-assessed symptoms, a series of means-as-outcomes models (Raudenbush & Bryk, Reference Raudenbush and Bryk2002) was conducted. For each EMA item, the Level 1 equation was the same as presented earlier in the paper. At Level 2, diagnostic group was entered as a categorical predictor using dummy coding (0 = participant not in that group; 1 = participant in that group):
With respect to the new notation, γ00 denotes the mean rating in the HV group, while γ01, γ02, and γ03 denote the differences in mean rating between the HV group and the DMDD, ADHD, and ANX groups, respectively. For each EMA item, parallel Level 2 equations tested the remaining group comparisons (e.g., DMDD vs. ADHD) in which the reference group was changed. FDR-corrected p values were calculated separately for irritability symptoms and anxiety symptoms.
Irritability symptoms
The full results for irritability symptoms across diagnostic groups are presented in Table 3. In general, irritability symptom levels exhibited a graduated pattern across the groups. As expected, the DMDD group endorsed a temper outburst on a significantly greater proportion of prompts than the ANX and HV groups. However, the DMDD group did not differ significantly from the ADHD group in the frequency of temper outbursts. In turn, both the ADHD and ANX groups endorsed a temper outburst on a significantly greater proportion of prompts than the HV group. With respect to both irritable mood and frustration, the DMDD group reported significantly higher levels than all three of the other groups (ADHD, ANX, and HV). The ANX group reported significantly higher irritable mood and frustration than the HV group. No other group differences were significant. For momentary anger, again, the DMDD group reported significantly higher levels than all the other groups (ADHD, ANX, and HV). In turn, the ADHD and ANX groups reported significantly higher momentary anger than the HV group. No other group differences were significant.
Irritability symptom levels across diagnostic groups
Note: ADHD = attention-deficit/hyperactivity disorder; ANX = anxiety disorder; DMDD = disruptive mood dysregulation disorder; FDR = false discovery rate; HV = healthy volunteers; SE = standard error. All group means for all variables were significantly greater than 0 (all corrected p values < .05)
a For illustration purposes only, γ00 (SE) reflects multilevel modeling in which categorical outcome variable was run as continuous, thus representing the estimated % of prompts on which the item was endorsed; group comparisons were computed using logistic multilevel modeling
Anxiety symptoms
The full results for anxiety symptoms across groups are presented in Table S5 of the Supplementary Material. Similar to irritability symptoms, anxiety symptom levels exhibited a graduated pattern across groups. As expected, the ANX group reported significantly higher levels of anxious affect and endorsed anxious avoidance on a significantly greater proportion of prompts than both the ADHD and HV groups. However, the ANX group did not differ significantly from the DMDD group on anxious affect or anxious avoidance. Consistent with this, the DMDD group reported significantly higher anxious affect than the HV group and endorsed significantly greater anxious avoidance than both the ADHD and HV groups. For momentary anxiety, the ANX group reported a significantly higher level than the HV group, and the DMDD group reported a significantly higher level than the HV group. No other group differences were significant.
Within-prompt associations between irritability symptoms
A series of analyses was conducted to examine within-prompt associations between irritability symptoms across groups. At Level 1, each EMA irritability item was entered as a person-centered, continuous predictor to test whether and how within-person fluctuations in that symptom predicted fluctuations in the other irritability symptom. Analyses were constrained such that symptoms assessed since the previous prompt were tested as predictors of other symptoms since the previous prompt (i.e., temper outburst, irritable mood, frustration) and of momentary symptoms (i.e., momentary anger). Momentary symptoms were never predictors of symptoms since the previous prompt. Further, no association was tested twice. At Level 2, diagnostic group was again entered as a categorical predictor:
Level 2 (participant level):
Here, β1j denotes the association between irritability item ratings rating for participant j. At Level 2, γ10 denotes the association between irritability ratings in the HV group, while γ11, γ12, and γ13 denote the differences in this association between the HV group and the DMDD, ADHD, and ANX groups, respectively. FDR-corrected p values were calculated separately for group means and comparisons.
Table 4 presents the full results for within-irritability associations across groups. In general, all four groups showed significant associations among irritability symptoms. There were no significant group differences in within-irritability, within-prompt associations (i.e., no difference in slopes across groups).
Within-prompt associations between irritability symptoms
Note: ADHD = attention-deficit/hyperactivity disorder; ANX = anxiety disorder; DMDD = disruptive mood dysregulation disorder; FDR = false discovery rate; HV = healthy volunteers; SE = standard error. There were no significant group comparisons
Supplementary analyses: within-prompt associations between irritability and anxiety
For a full description and results of these analyses, which were generally nonsignificant, see the Supplementary Material.
Discussion
The aim of the present study was to use smartphone-based EMA to measure pediatric irritability expressed naturalistically and transdiagnostically. Supporting feasibility, rates of compliance with the EMA protocol were high and similar across diagnostic groups (Babinski & Welkie, Reference Babinski and Welkie2020; Glenn et al., Reference Glenn, Kleiman, Kearns, Santee, Esposito, Conwell and Alpert-Gillis2020; Rodríguez-Blanco, Carballo, de León, & Baca-García, Reference Rodríguez-Blanco, Carballo, de León and Baca-García2020). These compliance rates are comparable with those reported in previous similar studies. According to a recent meta-analysis that included 42 EMA studies of youth across clinical and nonclinical settings (Wen, Schneider, Stone, & Spruijt-Metz, Reference Wen, Schneider, Stone and Spruijt-Metz2017), most studies defined compliance as the proportion of prompts to which participants responded. In these studies, the weighted average compliance rate was 78.3%, with a similar average compliance across studies with clinical (76.9%) and nonclinical (79.2%) samples.
Overall, the results indicated significant, consistent correlations between youth's EMA reports of irritability symptoms in their natural environments and in-clinic measures of irritability as rated by youth, parents, and clinicians. EMA-assessed irritability symptoms were also related to laboratory task frustration ratings, documenting links between EMA and laboratory-elicited frustrative nonreward. The diagnostic groups differed in levels of irritability symptoms, as expected, while the interrelations among irritability symptoms appeared comparable across groups. Below, we discuss each of these findings in detail and propose key directions for future research.
All four EMA items assessing irritability symptoms were significantly associated with established youth-, parent-, and clinician-report measures of irritability. The strength of these associations was in the small to medium range, the same range as that found for the EMA anxiety items (see also Smith et al., Reference Smith, Kircanski, Brotman, Do, Subar, Silk and Pine2019), and, importantly, was comparable across the different irritability items. These results suggest that the EMA items used in this study might serve as similarly strong estimates of the overall construct of irritability, while also reflecting its multifaceted nature and fluctuations over time. Convergent validity was also shown for irritability-related functional impairment, which is an important outcome measure in treatment studies. The feasibility and convergent validity of these items build upon previous EMA research in child and adolescent anxiety (e.g., Oppenheimer et al., Reference Oppenheimer, Silk, Lee, Dahl, Forbes, Ryan and Ladouceur2020; Smith et al., Reference Smith, Kircanski, Brotman, Do, Subar, Silk and Pine2019), ADHD (e.g., Babinski & Welkie, Reference Babinski and Welkie2020; Slaughter et al., Reference Slaughter, Leaberry, Fogleman and Rosen2020), and mood disorders (Gershon, Kaufmann, Torous, Depp, & Ketter, Reference Gershon, Kaufmann, Torous, Depp and Ketter2019; Hamilton et al., Reference Hamilton, Chand, Reinhardt, Ladouceur, Silk, Moreno and Bylsma2020), providing naturalistic indices that can be used in future pathophysiological and treatment research on irritability and, as discussed below, could be integrated with real-time interventions.
Additional evidence of convergence across levels of analysis was found in the subset of transdiagnostic participants who completed the SST. Specifically, the findings documented consistent links between EMA-assessed irritability symptoms and frustration levels reported during the SST. This was shown despite the relatively long average time span between the completion of EMA and the SST (median of 40 days). Of note, however, EMA-rated anxiety symptoms were also significantly associated with SST-reported frustration. The associations of both irritability and anxiety symptoms with SST frustration ratings may reflect a common element of clinical severity or negative affectivity that cuts across symptom dimensions. These results also may be consistent with accumulating evidence on the clinical (e.g., Cornacchio et al., Reference Cornacchio, Crum, Coxe, Pincus and Comer2016; Stoddard et al., Reference Stoddard, Stringaris, Brotman, Montville, Pine and Leibenluft2014), genetic (Savage et al., Reference Savage, Verhulst, Copeland, Althoff, Lichtenstein and Roberson-Nay2015; Stringaris, Zavos, Leibenluft, Maughan, & Eley, Reference Stringaris, Zavos, Leibenluft, Maughan and Eley2012), behavioral (e.g., Hommer et al., Reference Hommer, Meyer, Stoddard, Connolly, Mogg, Bradley and Brotman2014), and neural (Cardinale et al., Reference Cardinale, Subar, Brotman, Leibenluft, Kircanski and Pine2019; Kircanski et al., Reference Kircanski, White, Tseng, Wiggins, Frank, Sequeira and Brotman2018) links between irritability and anxiety. That is, extant studies suggest that there are both shared and specific correlates of irritability and anxiety. The current findings for laboratory-assessed frustration underscore the value of further examination of common versus specific mechanisms of these two symptom dimensions. Future studies should examine associations between EMA measures and neural circuit, physiological, and behavioral levels of analysis, in line with the RDoC framework.
As predicted, the findings for irritability symptom levels by primary diagnosis evidenced a graduated pattern across the clinical groups. First, the DMDD, ADHD, and ANX groups all exhibited higher levels of temper outbursts and momentary anger than youth with no psychiatric diagnosis, and the DMDD and ANX groups also exhibited higher levels of irritable mood and frustration than healthy youth. These results highlight the transdiagnostic nature of irritability in clinical populations. Second, the DMDD group exhibited higher levels of all irritability symptoms relative to both the ADHD and ANX groups, with the exception of a nonsignificant difference in temper outbursts between the DMDD and ADHD groups. These findings support the higher threshold of overall symptomatology required for DMDD, as well as the role of temper outbursts in ADHD (Cardinale et al., Reference Cardinale, Freitag, Brotman, Pine, Leibenluft and Kircanski2021; Gisbert et al., Reference Gisbert, Vilar, Rovira, Sanchez-Mora, Pagerols, Garcia-Martinez and Ribases2019; Karalunas, Gustafsson, Fair, Musser, & Nigg, Reference Karalunas, Gustafsson, Fair, Musser and Nigg2019). The observed discrepancy for “phasic” (behavior) versus “tonic” (mood) irritability in the ADHD group indicated that the ADHD group reported temper outbursts as frequently as the DMDD group, but the ADHD group did not report similarly elevated levels of irritable mood as the DMDD group. This relatively stronger association between ADHD and phasic irritability is consistent with recent findings reported by Cardinale et al. (Reference Cardinale, Freitag, Brotman, Pine, Leibenluft and Kircanski2021). Third, the ADHD and ANX groups both showed levels of irritability intermediate between DMDD and HV, replicating and extending previous reports on the significance of irritability within primary ADHD and ANX (e.g., Cornacchio et al., Reference Cornacchio, Crum, Coxe, Pincus and Comer2016; Haller, Stoddard, et al., Reference Haller, Stoddard, Pagliaccio, Bui, McGillivrary and Brotman2020; Shaw et al., Reference Shaw, Stringaris, Nigg and Leibenluft2014).
Interestingly, somewhat less diagnostic specificity was suggested for EMA-assessed anxiety symptoms between the ANX and DMDD groups, which did not differ significantly in levels of any anxiety item. An equivalent pattern was observed for SCARED youth-report scores, although the ANX group did exhibit higher SCARED parent-report and PARS scores than the DMDD group. One potential source of non-specificity in youth reports might involve the influence of ANX comorbidity in the DMDD group (see Table S1 of the Supplementary Material), consistent with previous findings (Brotman et al., Reference Brotman, Schmajuk, Rich, Dickstein, Guyer, Costello and Leibenluft2006; Hommer et al., Reference Hommer, Meyer, Stoddard, Connolly, Mogg, Bradley and Brotman2014; Mulraney et al., Reference Mulraney, Schilpzand, Hazell, Nicholson, Anderson, Efron and Sciberras2016). In addition, the overall clinical threshold for DMDD is high, reflecting a severe presentation that includes comorbid symptoms more often than not (Copeland et al., Reference Copeland, Angold, Costello and Egger2013; Freeman, Youngstrom, Youngstrom, & Findling, Reference Freeman, Youngstrom, Youngstrom and Findling2016). Thus, results might relate to the fact that the DMDD group was, by definition, clinically severe.
While irritability symptom levels differed significantly across groups, the interrelations among irritability symptoms were similar across groups. Specifically, irritability symptoms within prompts were significantly associated with one another in all groups, with one exception in the ANX group and two exceptions in the HV group. Thus, despite divergent levels of symptoms across diagnoses, the interrelations between different facets of irritability were preserved across diagnoses, providing insight into the transdiagnostic phenomenology of irritability. Future EMA studies may usefully explore the dynamics of irritability symptoms over time, for example, exploring how reporting irritable mood or frustration at one time point may influence the occurrence of a temper outburst at a subsequent time point. Future work might also consider moderating factors such as parental interactions. For example, parental responses to temper outbursts, such as increased attention, might influence the likelihood or intensity of future outbursts. Interrogating the antecedents and consequences of irritability symptoms in youth's daily lives will be valuable in both understanding and clinically targeting these symptoms.
In addition to using EMA as a phenotyping tool, recent and ongoing efforts are using EMA in conjunction with information processing technologies to develop “just in time” interventions tailored to psychiatric symptoms reported in daily life (e.g., Bidargaddi, Schrader, Klasnja, Licinio, & Murphy, Reference Bidargaddi, Schrader, Klasnja, Licinio and Murphy2020; Nahum-Shani, Hekler, & Spruijt-Metz, Reference Nahum-Shani, Hekler and Spruijt-Metz2015; Nahum-Shani et al., Reference Nahum-Shani, Smith, Spring, Collins, Witkiewitz, Tewari and Murphy2018). As empirically-supported measures of daily and momentary irritability symptoms, EMA items from the current study could be employed in future interventions for pediatric irritability, such as informing clinicians as to the clinical status of their patients to tailor interventions as needed. EMA reports could also be paired with real-time monitoring of youth's physiology and behavior to derive a more complete phenotype of irritability and assess the impact of interventions across levels of analysis (Chaspari et al., Reference Chaspari, Goodwin, Wilder-Smith, Gulsrud, Mucchetti, Kasari and Narayanan2014; Fletcher et al., Reference Fletcher, Dobson, Goodwin, Eydgahi, Wilder-Smith, Fernholz and Picard2010). This may be of a particular value in pediatric irritability, which is underserved with respect to evidence-based psychosocial treatments (Kircanski et al., Reference Kircanski, White, Tseng, Wiggins, Frank, Sequeira and Brotman2018). Overall, the feasibility, convergent validity, and transdiagnostic phenomenology of EMA-assessed irritability in the current study lay important groundwork for intervention efforts.
There are several limitations of the current study. First, we focused on irritability as it presents in DMDD, ADHD, and ANX. However, irritability is common in other disorders, such as major depression (Perlis et al., Reference Perlis, Fraguas, Fava, Trivedi, Luther, Wisniewski and Rush2005). We did not recruit participants with major depression in the current study because we were interested in chronic, not episodic, irritability. Future studies might examine these EMA items across additional diagnoses to further assess generalizability. Second, as participants were simultaneously being recruited for multiple ongoing studies with different inclusion criteria, there were several discrepancies across groups in relation to recruitment and procedures. In particular, only the DMDD and ADHD groups completed the CL-ARI, and participants in the ANX group had to be medication-free. Third, we matched the ANX and HV groups to the DMDD and ADHD groups based on age and sex distribution. Because males were particularly predominant in the ADHD group, this matching strategy resulted in the sample being predominately male. In addition, the sample was largely composed of White non-Hispanic participants. As such, future research should increase sample diversity with respect to race and ethnicity in order to further examine generalizability. Fourth, based largely on the constraints of school and smartphone use, the youth were prompted three times per day and the majority of the EMA items assessed symptoms that had occurred since the previous prompt. While this enhanced our ability to capture irritability symptoms (relative to querying only momentary experience), this protocol may have not fully captured the intensity of symptoms at their peak. Future EMA studies might also consider event-contingent reporting. Finally, while this study utilized frustration ratings obtained during a laboratory task in an initial examination of frustrative nonreward, future studies might include other external validators such as biological and behavioral indices.
In sum, in accordance with the call for transdiagnostic, developmentally-sensitive assessment methods consistent with the RDoC framework, the present study leveraged digital technology and EMA methodology to assess pediatric irritability in daily life. Demonstrating convergent validity with established measures and laboratory-elicited frustrative nonreward, as well as transdiagnostic phenomenology across a range of pediatric conditions, the results provide tools that can be used in future research. In particular, pathophysiological and treatment studies should prioritize examinations across EMA, behavioral, and neural circuit levels of analysis in order to better understand and intervene on irritability symptoms as they naturalistically occur.
Supplementary Material
The supplementary material for this article can be found at https://doi.org/10.1017/S0954579421000717
Acknowledgments
This work was supported by the Intramural Research Program (IRP) of the National Institute of Mental Health/National Institutes of Health (NIMH/NIH), ZIAMH002781 (Pine), ZIAMH002786 (Leibenluft), ZIAMH002778 (Leibenluft), and conducted under NIH Clinical Study Protocols 01-M-0192 and M-00-M-0021 (ClinicalTrials.gov identifiers: NCT00018057 and NCT00025935).
Funding Statement
This research received no specific grant from any funding agency, commercial or not-for-profit sectors.
Conflicts of Interest
None.
