The Benefits and Costs of Involving Clinical Judgment in the Diagnostic Process, Particularly With Regard to Diagnosis and Mood Severity Ratings

Clinician ratings play a central role in research on bipolar disorder. Research diagnoses are typically made via a semistructured diagnostic interview (Fristad, Teare, Weller, Weller, & Salmon, 1998; Geller et al., 2001; Kaufman et al., 1997; Orvaschel, 1995), which provides more latitude for clinical judgment than is allowed in more structured interviews. Clinician ratings of the severity of depressed (Poznanski, Miller, Salguero, & Kelsh, 1984) and manic symptoms (Axelson et al., 2003; Fristad, Weller, & Weller, 1992, 1995; Young, Biggs, Ziegler, & Meyer, 1978) are also central to characterizing the phenomenology of cases and to measuring treatment outcomes. The arguments in favor of emphasizing clinician ratings focus on the improved validity of the ratings. Mood disturbances in youths can be difficult to differentiate from symptoms of other conditions (Bowring & Kovacs, 1992; Kim & Miklowitz, 2002), and clinical judgment might help in teasing these components apart. Clinical judgment also could play a role in discerning whether a behavior is developmentally “within normal limits.” Similarly, raters are often in a position to gauge whether an adult is overreporting symptoms, pathologizing developmentally appropriate behaviors, or offering naïve responses to questions about clinical behaviors (see Drotar, Stein, & Perrin, 1995, for a discussion of the role of follow-up clinical probing in scoring responses on checklists).

These potential benefits are offset to some degree by increases in cost and decreases in the interrater reliability of instruments. Semistructured interviews are more expensive because they require the use of staff to conduct the interview, whereas parents and teenagers are often able to complete rating scales independently. The cost also rises because the flexibility in interview structure and scoring requires increased training if the procedures are to be done with adequate interrater reliability, and it also becomes more important to add mechanisms to prevent rater drift. The best-validated semistructured interviews also add considerably to the length of time required for the interview, with sequential interviews of the parent and youth resulting in total interview times in the range of 1.5 to 6 hr (Geller et al., 2001; Kaufman et al., 1997).

A less discussed, but more pernicious, problem is that raters might use different anchors for ratings in a way that makes direct comparisons across sites or groups much more difficult to interpret. Anchoring effects could occur either because of formal differences in the implementation of the rating system, or because of differences in the range of behavior that raters encounter in a given setting (Epley & Gilovich, 2004). One example of differences in anchoring comes from the way that the Kiddie Schedule for Affective Disorders and Schizophrenia (K-SADS; Geller et al., 2001; Kaufman et al., 1997) interview is rated across sites. Geller et al. (2002) provide thorough descriptions of clinical presentations and how they were scored. This level of detail helps calibrate ratings across sites, and illuminates potential methodological differences that may drive apparent differences in findings. For instance, consider a vignette of an adolescent female telling a school principal to go “screw himself.” Some groups would most likely consider this an example of irritable mood, and more context would be needed to determine whether this would be scored in the depression, mania, or some other section of the K-SADS. According to Geller et al. (2002), raters adequately trained in the use of the Washington University (WASH-U)-K-SADS would consider the vignette as evidence of irritable mood, but also consider it evidence of hypersexuality, because the profanity contained sexual terminology. Further, the WASH-U-K-SADS guidelines specify that this vignette exemplifies severe grandiosity, because the behavior shows a loss of contact with the reality that there would be consequences for insulting a school principal. It also is noteworthy that the WASH-U algorithms consider severe grandiosity or severe elated mood (scores of 5 or 6 on the WASH-U-K-SADS) to be evidence of psychosis due to the loss of contact with reality. Thus, this vignette would potentially be considered psychotic behavior in one study, but not in another. The differences in anchors make it easier for a behavior to be considered “threshold” or “severe,” and also make it easier to be considered psychotic. The WASH-U algorithm also provides more gateways into being labeled “psychotic” than are being used by some other groups. It is not clear without external validators that particular set of anchors or scoring algorithm is better. Clearly, one strategy will be more sensitive to mania and psychosis, and the other will be more specific, at the possible cost of missing bipolar cases. However, these methodological differences clearly help understand how groups are documenting markedly different rates of particular symptoms or psychosis (Kowatch, Youngstrom, Danielyan, & Findling, 2005).

Data from the Young Mania Rating Scale (YMRS; Young, Biggs, Ziegler, & Meyer, 1978), the most widely used clinical rating scale for the severity of manic symptoms, also provide an instructive example of how differences in setting and modal clinical presentations may contribute to anchoring effects. Ten sites provided YMRS data from youths with research diagnoses of bipolar spectrum disorders (aggregate N = 824 bipolar spectrum cases; Youngstrom, Findling, Sachs, et al., 2003). The average scores for bipolar cases differed significantly across sites, F (9, 776) = 30.39, p < .00005, with site means ranging from 10.0 to 34.5. Counterintuitively, the average YMRS score from the inpatient unit was among the lower means (17.2), and the three highest averages all came from outpatient settings (all three means greater than 32.0). Furthermore, there were Site × Item interactions, indicating that that there were differences either in the clinical presentation of bipolar cases, or else differences in the way that raters were scoring items at different sites, F (90, 5212) = 25.62, p < .00005. Overall, site differences accounted for 15% of the variance in YMRS scores, representing a medium to large effect size. Both the fact that there were significant differences in item scoring and the relative ranking of the site averages suggest that anchoring effects may play a large role in determining clinical ratings of mania.

Some of these differences in ratings across sites may be due to the use of different algorithms for generating a summary score when clinical raters are confronted with inconsistent data from the parent versus the child. Changes in the scoring algorithm could have substantial effects on final scores, and lead directly to shifts in diagnostic sensitivity and specificity. Taking the higher score (a “disjunctive” strategy) maximizes sensitivity (i.e., the percentage of true bipolar cases correctly identified) but penalizes specificity (i.e., the percentage of nonbipolar cases correctly identified; Youngstrom, Findling, & Calabrese, 2003). Multiple groups have experimented with a “structured” YMRS that used clearer scoring guidelines and more detailed anchors in an effort to improve interrater reliability. Structured YMRS ratings were consistently higher than typical YMRS ratings, even when both sets of ratings were completed by the same raters. Clinical judgment aims to improve the validity of the decision, but typically involves a reduction in the interrater reliability of scoring decisions that sets an upper limit on the potential validity of the resulting scores. Research groups have made different choices in terms of how to integrate data from multiple informants. Some have tended to interpret the higher score as being accurate (Tillman et al., 2004) or at least generally more clinically valid (Fristad et al., 1998); others have tended to use an average of the scores, and yet other groups have placed more emphasis on clinical judgment resolving each discrepancy separately (Findling et al., 2001; Youngstrom, Meyers, et al., 2005). In many published studies, the method used to resolve discrepancies is not specified.

The Validity of Parent, Teacher, and Youth Self-Report of Manic Symptoms

Although it is a truism in child assessment that assessors should gather data from multiple informants (Sattler, 1998), there has been much debate about the relative value of parent, teacher, and youth report of hypomanic and manic symptoms. It has been uncertain who might be the optimal informant with regard to mania, and also who might be sufficiently accurate to provide reliable diagnostic information by themselves. This is a central question, because some research has relied entirely on adolescent self-report to establish diagnoses (Lewinsohn, Klein, & Seeley, 1995; Teplin, Abram, McClelland, Dulcan, & Mericle, 2002), whereas other investigations have relied almost entirely on parent report, especially in prepubertal children (Wozniak et al., 1995). Those who incorporate both parent and child interviews into the diagnostic process have also used different algorithms for resolving discrepancies, such as consistently taking the more severe score as the summary score, regardless of source (Geller et al., 2001), versus relying on best clinical judgment or discussion with both informants to achieve consensus (Findling et al., 2001). If there are differences in the sensitivity of parents, teachers, or youths to mania, then this will result in differences in the rate of cases identified, and it could also generate differences in apparent clinical features of the condition. Teacher ratings have also featured prominently in discussions of the validity of the PBD diagnosis: there have been discussions about whether revised diagnostic criteria should require demonstration of impairment in multiple settings before making a diagnosis of mania, versus parent report being sufficient to establish a diagnosis even in the absence of corroborating information (Carlson & Youngstrom, 2003; Leibenluft et al., 2003).

What is necessary to evaluate the relative validity of different informants would be a design where they report on similar behaviors and are evaluated against the same diagnostic criterion (cf. Richters, 1992). The Achenbach System of Empirically Based Assessment (Achenbach, 1991a, 1991b, 1991c) offers an opportunity to compare parent, teacher and youth impressions, because eight of the core syndrome scales and 89 of 118 behavior problem items are identical across all three versions. The CBCL has been the most widely used parent-reported scale in research in PBD to date, with published results from at least eight different research groups on three continents (Hazell et al., 1999; Kahana, Youngstrom, Findling, & Calabrese, 2003; Mick, Biederman, Pandina, & Faraone, 2003, provides a meta-analysis of prior studies; Tramontina et al., 2003). However, fewer studies have reported teacher report on the Teacher Report Form (TRF) in the same sample (Carlson, Loney, Salisbury, & Volpe, 1998; Geller, Warner, Williams, & Zimerman, 1998; Hazell et al., 1999; Kahana et al., 2003; Youngstrom, Findling, Calabrese, Gracious, et al., 2004), and yet fewer have included youth self-report data from the Youth Self-Report (YSR) Form (Hazell et al., 1999; Kahana et al., 2003; Youngstrom, Findling, Calabrese, Gracious, et al., 2004).

What consistently emerges from these articles is that parent CBCLs show elevations on multiple scales, and that the magnitudes of these elevations compared to the scores of youths diagnosed with other disorders (most typically attention-deficit/hyperactivity disorder [ADHD]) are larger based on parent report as opposed to teacher report or self-report. YSR scores did not differ significantly between bipolar and ADHD cases in one sample (Hazell et al., 1999), and failed to show any incremental predictive value after controlling for parent CBCL in another sample (Kahana et al., 2003). Thus, effect sizes for bipolar versus nonbipolar comparisons consistently appear to be smallest for self-report on the YSR, and largest on the CBCL.

To compare the diagnostic value of each potential informant directly, Youngstrom, Findling, Calabrese, Gracious, et al. (2004) compared parent, teacher, and youth report on six measures in a sample of 324 youths ages 11 years 0 months to 17 years 11 months. The measures included the parent, teacher, and youth versions of the Achenbach, parent report on the General Behavior Inventory (PGBI; Youngstrom, Findling, Danielson, & Calabrese, 2001), parent report on a questionnaire version of the YMRS (P-YMRS; Gracious, Youngstrom, Findling, & Calabrese, 2002; Youngstrom, Gracious, Danielson, Findling, & Calabrese, 2003), and adolescent self-report on the GBI (Danielson, Youngstrom, Findling, & Calabrese, 2003; Reichart et al., 2004). The same publication also presented a comparison of parent and teacher ratings in a younger sample using the same instruments, with 318 youths ages 5 years 0 months to 10 years 11 months old. The criterion diagnoses were based on a K-SADS with sequential interviews of the parent and child by the same rater, and with discrepancies resolved using best clinical judgment. All cases were reviewed with a licensed medical doctor clinician before final diagnoses were assigned, and more than 60% of the cases were subsequently reinterviewed by a medical doctor before continuing into other research protocols.

The diagnostic efficiency of each test was quantified using receiver operating characteristic (ROC) analyses (Figure 1), which examine the trade-off between diagnostic sensitivity and specificity across the full range of test scores, captured by the area under the curve (AUC) statistic (Kraemer, 1992).

A receiver operating characteristic (ROC) plot of the diagnostic efficiency of a 10-item PGBI discriminating any bipolar spectrum diagnosis (n = 291) from all other diagnoses (n = 346). The dashed line represents the threshold imposed by the criterion diagnosis having a kappa reliability of .90.

In the adolescent sample, the three parent report measures all performed significantly better than the teacher or youth report. On the Achenbach instruments, the parent CBCL Externalizing score earned an AUC of .78, the PGBI hypomanic/biphasic score earned an AUC of .84, and the P-YMRS earned an AUC of .80. The TRF fared significantly worse than all three parent measures (p < .005), with an AUC of .70. The adolescent self-report on both measures was comparable to the diagnostic validity of teacher report on the TRF, with AUCs of .71 on the YSR and .67 on the GBI; thus, youth report was also significantly worse than parent report in discriminating bipolar cases.

In the younger sample, the three parent measures performed similarly well. The CBCL Externalizing had an AUC of .82, the PGBI an AUC of .81, and the P-YMRS an AUC of .83 (95% confidence intervals = 9–12%). The TRF not only performed significantly worse than any of the parent measures, but it also did not succeed in discriminating bipolar cases significantly better than at a chance level, with an AUC of .57. Neither youth nor teacher report helped identify additional cases with bipolar disorder in either age group after controlling for parent report on any of the measures.

The better performance of parent report versus youth report on the GBI has been replicated in a new sample, with many of the participants drawn from an urban community mental health center (Youngstrom, Meyers, et al., 2005). Furthermore, in the new sample parent report outperformed YSR on a two other measures, the Mood Disorder Questionnaire (MDQ; Hirschfeld et al., 2000), and a self-report version of the YMRS adapted for comparison to the P-YMRS. In the sample of 124 adolescents, the PGBI had an AUC of .80 versus .65 for adolescent self-report; the Parent MDQ (P-MDQ) had an AUC of .75 versus .63 for self-report, and the P-YMRS had an AUC of .70 versus .50 for self-report. In each instance, the parent report provided statistically significant classification superiority.

These results are consistent with prior findings that PBD elevates the parent report more than the teacher or youth report (Carlson et al., 1998; Geller et al., 1998; Hazell et al., 1999), but the findings sharply contradict the prevailing clinical wisdom that assessments should rely on self-report for information about mood disorders (Loeber, Green, & Lahey, 1990). The advantage of relying on parent report becomes even larger when using measures that concentrate on symptoms of mania: The CBCL and YSR AUCs differed by .07, but the measures including more manic symptoms yield differences of .12 to .20 in performance. Several factors are likely to be operating here. One is that manic symptoms tend to manifest as externalizing behavior problems, which are readily observed and reported by collateral informants (Youngstrom, Loeber, & Stouthamer-Loeber, 2000). Another is that lack of insight into one's behavior is an associated feature of mania (Dell'Osso et al., 2002; Pini, Dell'Osso, & Amador, 2001), with self-report of manic symptoms having less validity as a consequence (Ghaemi et al., 2005; Miller, Klugman, Berv, Rosenquist, & Ghaemi, 2004; Youngstrom, Findling, & Calabrese, 2004).

How Much Cross-Situational Consistency Is Typically Shown in Mood and Behavior?

An important and highly controversial issue is how much agreement should be expected between parents, youths, and other informants such as teachers about manic symptoms. A frequent clinical presentation involves a highly distressed and concerned parent, along with a youth and teacher who report few if any of the same concerns. This constellation of opinions might reflect greater sensitivity to the symptoms of mania by parents, or possibly that the problems associated with mania are sometimes limited to the home environment; or that the parent's perceptions are inaccurate and exaggerated. There are mechanisms that could explain distorted parent report, including the effects of the parent's own distress or mood disorder on their ratings of child behavior (Richters, 1992; Youngstrom, 1999; Youngstrom et al., 2000), or the recent popularity of the bipolar diagnosis causing parents to interpret behaviors as evidence of mania instead of other more common processes. A consistent finding has been that when parent data are compared to youth or teacher data on a similar instrument, levels of parent-endorsed mania, and other behavior problems are significantly higher than the levels reported by the youth or teacher (e.g., Carlson & Youngstrom, 2003; Findling et al., 2002; Geller et al., 1998; Hazell et al., 1999; Youngstrom, Findling, Calabrese, Gracious, et al., 2004), corroborating the clinical impression that parents are often the most worried about mania as well as other aspects of their child's functioning. Some suggest that future revisions of diagnostic criteria go so far as to stipulate that manic symptoms must be evident in multiple settings before considering a diagnosis of bipolar disorder, especially in a child (see Leibenluft et al., 2003, for discussion of narrow versus broad definitions of PBD).

When judging the degree of agreement between parent report and other informants' descriptions of manic symptomatology, it is vital to bear in mind the general level of agreement between different informants about child behavior. The average level of youth, teacher, and parent agreement is modest about behavior problems in general. An early meta-analysis found that youths agreed with adult informants r = .22 on average, and parents and teachers agreed r = .28 with each other (Achenbach, McConaughy, & Howell, 1987). Data from standardization samples typically indicates moderately higher levels of agreement, such as the correlations from the restandardization of the Achenbach instruments (see table 9-3 of Achenbach & Rescorla, 2001). Table 1 reports these benchmarks as well as parent–youth agreement about manic symptoms based on multiple independent samples (Youngstrom et al., 2004; Youngstrom, Findling, Calabrese, Gracious, et al., in press). Results indicate that parent–youth agreement about manic symptoms is somewhat lower than agreement about externalizing problems (which was r = .53 on the Achenbach scale in the Youngstrom, Findling, Calabrese, Gracious, et al., 2004, sample), but well within the range of agreement reported about other dimensions and measures.

Interrater agreement about manic symptoms and other behavior problems

Referral patterns and regression artifacts

It is the rare child or teenager that self-refers for mental health services, with the possible exception of school-based mental health services. Typically, the youths presenting at outpatient mental health centers arrive because parents are concerned about their behavior or functioning. Overall, there is surprisingly little agreement between the parent and youth about which (if any) behaviors are cause for concern (Yeh & Weisz, 2001).

The fact that outpatient referrals are most often driven by parental concerns, combined with the modest to moderate levels of agreement between informants about youth functioning, sets up a situation that has profound implications for clinical presentation. Essentially, outpatient clinics are selecting for high levels of parent reported problems: if the parent is not worried, they will not bring the youth to the clinic. When referrals are driven by one source, the predicted levels of symptoms reported by other sources can be established via simple regression equations. If parent-reported externalizing problems are extremely high, for example, then self-reported levels of externalizing should be higher than average, but also lower than the level of parent-reported problems, based on the imperfect correlation of parent and youth report (Campbell & Kenny, 1999).

The standardization sample of the Achenbach measures (Achenbach & Rescorla, 2001) provides benchmarks for the degree of anticipated agreement between parents, teachers, and youths on the Achenbach forms. Two previously unpublished sets of different analyses indicate that the level of externalizing problems reported by teachers and youths in cases with K-SADS diagnoses of bipolar disorder are actually significantly higher than would be expected based on the typical correlation found across informants. One analysis (Table 2) used the cross-informant correlations from the standardization sample to predict the average TRF externalizing and YSR externalizing scores based on the level of CBCL externalizing scores in the bipolar cases. These analyses showed that the youth self-reported levels of externalizing problems were two points higher than would be predicted based on the level of parent-reported problems, and teacher-reported problems were 5 points higher than would be expected based on parent report alone.

Predicted levels of teacher- and youth-reported externalizing problems

The second set of analyses investigated whether K-SADS bipolar cases had significantly more youth- or teacher-reported problems after statistically controlling for parental concerns. These analyses regressed teacher or youth report of externalizing on diagnosis (bipolar spectrum, yes or no) after controlling for CBCL externalizing (using the data described in Youngstrom, Findling, Calabrese, Gracious, et al., 2004). Findings again showed that cases with K-SADS bipolar diagnoses evinced higher levels of both self-reported (+3.3 points, p < .01) and teacher-reported (+4.4 points, p < .01) externalizing problems. Both sets of analyses strongly indicate that the level of impairment (measured as externalizing problems) shown by cases with K-SADS diagnoses of bipolar disorder are actually significantly higher than would be expected based on the level of parent-reported problems alone.

The Extent to Which a Parents' Mental Health Status Influences Their Report of Child Behavior

Perhaps the most vexing concern about relying on parent report of youth mood symptoms is the possibility that the parent's mental health history (Youngstrom et al., 2000) or current mood status (Youngstrom, Ackerman, & Izard, 1999) might influence their description of child functioning. The issue is especially pernicious given the high degree of heritability of bipolar disorder (McGuffin et al., 2003). As a consequence, parent report is most likely to be influenced by mood in precisely those cases where youths are at greatest risk of developing bipolar disorder (i.e., affected youths are more likely to have affected parents, whose judgment might be compromised as a result of their own mood states). Although there is a large literature examining the effects of depression on parent ratings (Richters, 1992), much less has been published on the effects of bipolar disorder or mania in particular on ratings of other people's behavior.

The one study of which we are aware directly looking at the effects of parental diagnoses of bipolar or unipolar disorder on cross-informant agreement found that parents with unipolar diagnoses reported significantly more Internalizing problems than did the adolescents, and parents with bipolar diagnoses reported significantly more youth manic symptoms than the youths reported themselves (Youngstrom, Findling, & Calabrese, 2004). However, even after controlling for parental diagnostic status, youths with K-SADS bipolar diagnoses still reported significantly fewer manic symptoms than did parents (p < .0005), strongly suggesting psychological “minimization” by the affected youths.

How Different Measures Compare in Terms of Detecting Bipolar Disorder, the Challenges in Comparing the Performance of Measures Across Research Groups, and Leading Candidates for Research or Clinical Use

As the number of potential measures for detection of PBD swells, it becomes increasingly important to be able to compare the measures and establish which have the greatest validity for clinical decision making as well as defining phenotypes for research. In theory, it should be possible to compare the diagnostic efficiency of different measures based on published findings, using a consistent metric to calibrate performance such as the AUC from ROC analyses (Zhou, Obuchowski, & McClish, 2002). The AUC is preferable to other measures such as test sensitivity and specificity, because it does not depend upon a single test threshold, and it also is unrelated to the base rate of the target diagnosis (unlike indices such as the positive and negative predictive values, or the overall percent correct; Zhou et al., 2002). It is possible to statistically test differences in AUC estimates of different tests based on the same sample, or the same test based on different samples (Hanley & McNeil, 1983).

Two factors complicate the agenda of comparing test performance based on published articles. One issue is the fact that different studies have used different operational definitions of bipolar disorder. More narrow definitions may tend to result in higher estimates of diagnostic sensitivity, particularly for index tests whose content includes more manic symptoms. Similarly, inclusion criteria that emphasize greater levels of impairment in the manic group (such as ascertainment from inpatient versus outpatient settings, or ascertainment from clinical versus community settings) will tend to yield higher estimates of diagnostic sensitivity because a larger percentage of bipolar cases will tend to score above threshold on both measures of general impairment as well as measures of mania (Zhou et al., 2002).

The second issue is that the composition of the comparison (i.e., nonbipolar) sample can vary dramatically across studies. Differences in the general level of impairment of the comparison group will have a direct effect on specificity estimates. Less impaired groups will be easier to separate from the target group, yielding higher estimates of specificity (Zhou et al., 2002). It is also important to consider the rate of occurrence of diagnoses that are difficult to differentiate from bipolar disorder. For example, both unipolar depression and ADHD demonstrate high degrees of symptom overlap with the clinical presentation of bipolar disorder (Bowring & Kovacs, 1992; Kim & Miklowitz, 2002). Samples that include higher rates of unipolar depression or ADHD would be expected to show lower levels of diagnostic specificity as a result, because nonbipolar cases would have a greater tendency to show spuriously high scores on measures intended to capture bipolar disorder.

The possibility that these two factors could markedly influence the apparent performance of diagnostic tests was empirically confirmed in a reanalysis of published data using two different sets of inclusion and exclusion criteria. The original analysis used a broad definition of bipolar disorder, lumping together youths who met strict criteria for bipolar I with youths meeting criteria for bipolar II, cyclothymia, and bipolar not otherwise specified (NOS; mostly due to inadequate duration of mood states to meet strict DSM-IV criteria for the other bipolar categories). The comparison group in the original analysis consisted of all other youths completing the assessment protocol, regardless of diagnosis (Youngstrom, Findling, Calabrese, Gracious, et al., 2004). As a result, the comparison group included a large percentage of youths with ADHD, unipolar depression, or both concurrently, along with a variety of other diagnoses.

The reanalysis limited the bipolar cases to those that satisfied the more restrictive inclusion criteria documented in Geller et al. (2003). Similarly, the reanalysis distilled the comparison group according to Geller et al.'s (2003) exclusion criteria, creating a group comprised entirely of youths meeting criteria for ADHD without any comorbid mood disorder or youths with no Axis I diagnosis at all.

Comparing the two sets of analyses clearly indicated that that diagnostic efficiency of all index tests was significantly better under the more distilled conditions for sample construction (Youngstrom, Meyers, Youngstrom, Calabrese, & Findling, in press). The AUCs of tests under the distilled conditions replicated the estimates reported in an independent sample (Tillman & Geller, 2005), providing excellent convergent evidence that similar tests perform comparably well under similar conditions, even across research groups. Unfortunately, the findings also indicated that not all tests were equally influenced by changes in sample characteristics. Although all index tests showed significantly degraded performance under more clinically generalizable sampling conditions, the tests that were optimized in samples under distilled conditions showed the largest decrement (Youngstrom, Meyers, et al., in press). Both the sensitivity and specificity of index tests changed markedly as a function of changing the sample composition.

These empirical findings reinforce the point that studies comparing the diagnostic performance of multiple measures in the same sample not only have greater statistical power, but also greater internal validity for comparing the relative performance of index tests (Hanley & McNeil, 1983). To date, we are aware of five published articles that compare multiple measures within the same sample (Findling et al., 2002; Youngstrom, Findling, Calabrese, Gracious, et al., 2004; Youngstrom, Gracious, et al., 2003; Youngstrom, Myers, et al., 2005, in press). Two of these publications (Findling et al., 2002; Gracious et al., 2002) have been superseded by a publication that subsumes the original dataset in a larger sample and more comprehensive analyses (e.g., Youngstrom, Findling, Calabrese, Gracious, et al., 2004). The pattern that consistently emerges across all of the studies is that parent report measures significantly outperform youth report and teacher report measures. Parent report measures that include more manic item content have a slight but sometimes statistically significant advantage over other parent-reported measures in terms of AUC. The superior performance of instruments such as the Hypomanic/Biphasic Scale of the GBI (Youngstrom et al., 2001), the Parent-Report Mood Disorders Questionnaire (adapted from Hirschfeld et al., 2000), and the Parent YMRS (Gracious, Youngstrom, Findling, & Calabrese, 2002) is mostly a function of their improved specificity, reducing the number of false positive test results in nonbipolar cases. It is worth noting that virtually no large studies to date have included teacher-reported mania scales, making it difficult to determine whether they also would show superior specificity compared to more global measures of behavior problems.

Within the realm of self-reported measures of mania or externalizing behaviors, there appears to be much less distinguishing the different index tests in terms of performance. Measures with more manic content do not significantly outperform the YSR externalizing scale. This is probably in part a result of the lessened insight into illness and behavior associated with manic states, but it also may be exacerbated by the reactive nature of many of the items in the scales. Questions that ask about irritable mood, for example, tend to produce significantly lower scores in self-report than collateral report. For example, the GBI item asking about times “when almost everything got on his/her nerves and made him/her irritable or angry” (#54) was substantially more endorsed by parents than youths with bipolar diagnoses (Cohen's d = 1.5). It is worth speculating that rephrasing of the questions so that endorsement seemed less pejorative to the respondent might not only yield higher mean scores, but also more valid (albeit indirect) scores. For example, in clinical interviews, it often is productive to assess irritable mood by asking the patient if there have been more arguments with parents, teachers, or significant others, or if the patient has noticed an increase in the “friction” of interpersonal interactions. It might be feasible to develop a self-report measure that assesses manic symptoms in a way that avoids defensive response sets, similar to some innovative measures of antisocial personality traits (Andershed, Gustafson, Kerr, & Stattin, 2002).

Based on the results of analyses comparing multiple measures, it appears that the PGBI (particularly the 10-item mania form; Youngstrom, Myers, et al., 2005) and the Parent MDQ (P-MDQ) are the two best-performing measures diagnostically. The P-YMRS appeared promising in initial studies, but has performed slightly less well than the PGBI and P-MDQ, perhaps due to the inclusion of two psychometrically weak items (#10, bizarre appearance, and #11, lack of insight; Gracious et al., 2002). Other parent measures with adequate representation of manic symptoms are likely to perform well (e.g., Gadow & Sprafkin, 1994; Pavuluri, 2002), although the few studies published have tended to use samples that might exaggerate diagnostic efficiency compared to what the instrument would deliver in most clinical settings.

The Evidence-Based Strategies for Interpreting Measures as Diagnostic Aids

Sensitivity and specificity

The sensitivity and specificity of a test are the most commonly reported features that have direct clinical application. Tests with high sensitivity are more effective at ruling diagnoses “out”: subthreshold scores on such tests will rarely occur in cases that have the disorder. Conversely, tests with high specificity are more helpful at ruling a diagnosis “in.” By establishing a stringent threshold that would be exceeded by few cases without the target condition, highly specific tests make the likelihood of a true diagnosis higher when the threshold is exceeded (Sackett et al., 2000).

There are several shortcomings to sensitivity and specificity from a practical viewpoint. One is that they are not intrinsic properties of a test, but instead are parameters that can change in different settings (Kraemer, 1992). This contradicts a fairly widespread belief that sensitivity and specificity are invariant across samples (Baldessarini, Finklestein, & Arana, 1983), but data reviewed above have demonstrated exactly these sorts of shifts in performance in measures detecting bipolar disorder (e.g., Youngstrom, Meyers, et al., in press). For this reason, it is imperative that more research examine the diagnostic efficiency of multiple tests across different age groups, demographic groups, and clinical conditions, so that researchers and clinicians can select estimates that more closely approximate the populations with which they work.

A second issue is that neither sensitivity nor specificity provides a direct gauge of the accuracy of a test result for a specific case, nor for a group of cases sharing positive tests results. These probabilities, that a test result is accurate and correctly indicates the true diagnostic status, are what would be most useful both clinically and in defining research samples. Such probabilities can be estimated, and they are variously referred to as the positive (PPP) and negative predictive values or predictive (NPP) powers (Kraemer, 1992). The PPP and NPP are Bayesian probability estimates that combine information from the test result with the prior risk of having the target condition. Put another way, the likelihood of a youth having a diagnosis is not just a function of their test result, but also of other factors that contribute to their risk. One of the most widely discussed factors is the base rate, or the frequency with which a disorder presents at a particular setting. PPP and NPP can be calculated from a combination of the base rate, sensitivity, and specificity of the test. The PPP and NPP, although much easier to interpret and apply than sensitivity and specificity, are commonly not reported because they are directly dependent on the base rate. The effects of base rate on PPP are illustrated in detail in the following section. Until recently, the lack of base rate estimates for PBD further complicated efforts to estimate PPP and NPP; but base rate estimates from different infrastructures are now becoming available (see Table 3 for a partial listing; Youngstrom, Findling, et al., 2005).

Examples of the effects of base rate on the positive predictive value (PPV) of high scores on two screening tests for adolescent bipolar disorder

A third issue is that sensitivity and specificity both change as a direct function of the threshold chosen on the index test. For example, considering CBCL externalizing T scores of 80 or higher a “test positive” result for bipolar disorder will have higher specificity and lower sensitivity than setting the test positive threshold at a T score of 70. Fewer nonbipolar cases would score in the 80+ range than in the 70+ range, but using the lower threshold would reduce the number of true bipolar cases missed. Published articles evaluating the same index test will produce different estimates of sensitivity and specificity if different thresholds are used. It then becomes unclear whether differences in diagnostic efficiency are due to changes in test performance, or simply because different thresholds were used. The problem of comparing test performance across samples can be solved by reporting a global measure of performance such as the AUC. However, the more challenging issue is deciding where the optimal decision threshold is for a test, and how to pick a threshold that would be robust and generalizable.

Quality calibrated test performance

Kraemer (1992) has developed a method for calibrating the sensitivity, specificity, and total percentage correct (which she calls “efficiency”) so that optimal thresholds become more obvious. The calibration process algebraically adjusts the sensitivity and specificity according to the rate of cases testing positive or negative at that threshold. Put another way, it is weighting the sensitivity, specificity, and efficiency estimates based on the marginal distributions. Intuitively, a sensitivity of 90% is much more impressive if only 20% of a sample tested positive than if 90% of the sample tested positive. Indeed, a sensitivity of 90% could be achieved by a random or useless test if one were willing to treat 90% of cases as “test positives.” Cohen's kappa, the percentage of correctly identified cases after adjusting for the marginals, is identical to Kraemer's calibrated efficiency.

It is possible to graph calibrated sensitivity as a function of calibrated specificity, producing what Kraemer calls a “Quality ROC” (QROC) plot. Unlike an ROC plot, the QROC plot can make it visually obvious what might be optimal places to cut a test to maximize sensitivity, specificity, and efficiency, adjusting each for chance performance. The QROC transformation changes the ROC curve into a shape that Kraemer describes as a “leaf” or the “hull” of a boat (see Figure 2). When calibrated sensitivity is plotted as a function of calibrated specificity, the point most closely approaching the top of the chart maximizes sensitivity above chance performance, and the point furthest to the right on the QROC curve maximizes specificity. The point closest to the top right corner maximizes Cohen's kappa, which would be the optimal place to set the test threshold if the costs and benefits of positive and negative results are equal.

A quality receiver operating characteristic (QROC) curve for a 10-item PGBI discriminating any bipolar spectrum diagnosis (n = 291) from all other diagnoses (n = 346). The diamond in the upper right-hand corner denotes the diagnostic efficiency of the criterion diagnosis (κ = .90).

Kraemer's approach is appealing because it offers a mathematical framework for identifying decision thresholds. It also allows for the incorporation of costs and benefits, which can shift the choice of threshold (although these sorts of patient preferences and clinical utilities have not been formally operationalized in the area of PBD yet). Another important feature is that it identifies multiple thresholds for a test, each optimized for a different purpose (i.e., maximizing sensitivity, specificity, or overall accuracy). This can help remind test users to think about the purpose for which they are using the test. However, despite these advantages, the QROC approach has not led to identification of robust decision thresholds on measures examined in the area of PBD. Instead, QROC strongly indicates that it is unlikely that robust decision points can be identified for tests discriminating PBD. Figure 2 presents a QROC plot for the 10-item version of the Hypomanic/Biphasic Scale from the PGBI. Of the measures evaluated in this particular sample, this index test presents a “best case” scenario (i.e., largest AUC, .86 in this sample versus a maximum potential AUC of .95 given the interrater reliability of the gold standard diagnosis). The shape of this QROC curve is quite different from the prototypic QROC curves presented in Kraemer (1992), and at the same time is it representative of the shape of QROC plots for the CBCL, TRF, YSR, parent and youth report on the GBI, parent and youth report on the MDQ.

Inspection of Figure 2 indicates that the statistically optimal threshold to maximize sensitivity is so low as to be clinically trivial. Setting the score at a 1 or higher results in an uncalibrated sensitivity of 1.00 and specificity of .79, and a calibrated sensitivity of .97. The problem is that setting the threshold this low creates a huge number of test positives (89% of this sample), with the consequence that there is little advantage to giving the test versus just treating everyone as a “test positive” for bipolar disorder. Similarly, the statistically optimal threshold to maximize specificity is extremely stringent. The recommended threshold score of 23 or higher, yielding a calibrated specificity of .85, is close to the absolute maximum score of 30. Although most cases scoring this high are likely to have bipolar disorder, the majority of bipolar cases will appear as false negatives (sensitivity = .19). Thresholds this high would require screening huge numbers of cases to identify a large enough number of cases to enable statistical analysis also: the threshold is high enough that it might miss four out of five true bipolar cases, and less than 10% of the validation sample scored in this range.

Finally, the QROC plot makes clear that there is no obvious best threshold in terms of maximizing kappa. Instead of there being a clear peak aimed at the top right corner of Figure 2, there is a long line of points nearly equidistant from the corner. In numeric terms, there is a wide range of threshold scores that produces nearly identical kappa scores. Threshold scores of 6+ to 15+ all offer kappa values greater than .50, with a sample specific maximum kappa of .55. Any advantage of a particular score relative to the rest of the pack is likely to be attributable to sampling error and not a robust improvement in performance. The QROC plot is indicating visually that there is little to choose between among a wide range of scores, at least when evaluating different measures in the context of PBD.

How Test Performance Changes When a Test Is Used in a New Setting and Implications for Research Samples and Clinical Practice

At least three major factors will affect test performance in predictable ways when tests are used in new settings. Broadly speaking, these are (a) the base rate of the condition, or how often it occurs at the new setting of interest; (b) factors that affect the definition or severity of the target condition; and (c) factors that affect the composition and severity of the nontarget group.

The Role of Family History of Mood Disorder Within an Assessment Framework

Family psychiatric history plays a large role in the clinical assessment of PBD, and in fact, is the only 1 of 30 risk factors evaluated in the literature that has proven robust enough to warrant routine clinical consideration (Tsuchiya, Byrne, & Mortensen, 2003). Although we need to “diagnose the child, and not the family,” family history is informative because of the strong contributions of both genes (e.g., McGuffin et al., 2003) and family environmental processes (e.g., Hammen, Adrian, Gordon, Jaenicke, & Hiroto, 1987; Miklowitz & Alloy, 1999) to the development and recurrence of bipolar disorder. Recent reviews have established that offspring of parents with bipolar disorder are at heightened risk for developing psychopathology in general, with the risk being somewhat higher for bipolar disorder in particular versus mood, anxiety, or behavior disorders in general (DelBello & Geller, 2001; Lapalme, Hodgins, & LaRoche, 1997). Meta-analysis indicates that on average 5% of the youths with a bipolar affected biological parent have developed bipolar disorder themselves at the time of participation in the included studies, versus 0% of the cases in the pooled comparison groups (Hodgins, Faucher, Zarac, & Ellenbogen, 2002).

It is possible to estimate an LR for the change in risk associated with having a first-degree relative with bipolar disorder. The recommended rule of thumb is to assign an LR of 5.0 to cases where a first-degree relative (e.g., biological parent or full biological sibling) has a confirmed history of bipolar disorder, and to assign half that risk (LR = 2.5) in instances where a second-degree relative has a history of bipolar disorder (Youngstrom, Findling, et al., 2005). This value was founded on the 5% prevalence of bipolar disorder in at-risk youths from the meta-analysis, compared to a 1% population prevalence. Several caveats need to be mentioned about the 5.0 LR estimate. The rates of bipolar disorder in the at-risk groups in the meta-analyzed studies varied a great deal, from 1 to 16%, with more heterogeneity than could be attributed just to sampling error. Perhaps more importantly, the 1% rate for the general population is higher than the rate of bipolar disorder reported in several epidemiological studies of youths (Costello, Mustillo, Erkanli, Keeler, & Angold, 2003; Lewinsohn, Klein, & Seeley, 1995). The 1% figure was chosen based on several considerations, including (a) epidemiological studies might be less sensitive to bipolar disorder because of reliance on self-report or because of issues of score anchoring or rater training; (b) epidemiological studies tend to underestimate the occurrence of bipolar spectrum disorders, because they often only inquire about episodes of mania, and not hypomanic episodes or symptoms (which are necessary for diagnoses of bipolar II or cyclothymia, in addition to bipolar NOS; Kessler, Rubinow, Holmes, Abelson, & Zhao, 1997); (c) bipolar spectrum illnesses besides bipolar I still appear to be quite impairing (Findling et al., 2005; Lewinsohn, Seeley, Buckley, & Klein, 2002), and thus deserving of consideration, even though not consistently included in epidemiological estimates; and (d) increasing the estimated prevalence in the general population is conservative, as it enlarges the denominator and therefore shrinks the LR and downsizes the degree of risk assigned to detecting a bipolar relative.

Given the controversy about labeling youths as having “bipolar disorder,” the authors of the evidence-based recommendations opted for a cautious stance with regard to the risk associated with positive family history. However, a good case could be made that the LR of 5.0 is too low. More conservative estimates of the prevalence of bipolar disorder in the general population would drive the LR much higher. More importantly, family histories of bipolar disorder themselves appear to be specific, but not very sensitive (Kendler, Prescott, Jacobson, Myers, & Neale, 2002; Kendler & Roy, 1995). In other words, research quality family histories, such as those gathered via the Family History–Research Diagnostic Criteria method (Andreasen, Endicott, Spitzer, & Winokur, 1977), tend to be accurate about the cases identified, but still tend to miss many cases that actually had mental health issues. The little evidence available suggests that typical clinical practices entail even less thorough assessment of family history, which would further erode the sensitivity of the assessment to familial bipolarity. The other issue that complicates clinical assessment yet further is that bipolar disorder appears to be underrecognized in minority populations (Bhatnagar et al., 2006; DelBello, Soutullo, & Strakowski, 2000; Neighbors, Trierweiler, Ford, & Muroff, 2003; Strakowski et al., 1997). Thus, family histories in non-White populations are likely to be even less sensitive to bipolar disorder.

Two further advantages of carefully assessing family history are both related to treatment. One issue is that response to pharmacological interventions is likely to be moderated by genes or genetic polymorphisms, such that the history of response or nonresponse of relatives could be valuable in guiding the treatment selection for the youth (e.g., Duffy et al., 2002). The more general advantage is that careful assessment of family history provides a sense of the context and interpersonal network surrounding the affected youth, which can help promote positive outcomes even in individually focused therapy modalities as well as family therapy (Geller, Tillman, Craney, & Bolhofner, 2004; Miklowitz, George, Richards, Simoneau, & Suddath, 2003; Miklowitz, Goldstein, Nuechterlein, & Snyder, 1988).

What Are the Implications of Assessment Research for the Understanding of the Phenomenology of Bipolar Disorder from a Developmental Framework?

One of the major potential contributions of assessment research to the understanding of developmental psychopathology comes from the use of consistent methods across different age cohorts. Only by using the same instrumentation does “homotypic continuity,” or the persistence of the same behavioral expressions of the same underlying trait, become clear. Furthermore, differences in symptom-level data when measured with a consistent methodology also provide strong evidence of developmental change, and provide a framework for using longitudinal data to gauge whether change is evidence of instability (e.g., remission or recovery, as well as later correction of initially false positive diagnoses) versus heterotypic continuity, whereby the same underlying process is expressed in the form of different behaviors due to the interaction of the process with developmentally determined environmental contexts (Cicchetti, Rogosch, & Toth, 1994). Of course, there are potential measurement issues that could also contribute to apparent change in the responses on psychometric items, such as cognitive limits to the reliability of self-report in younger samples (Anastasi & Urbina, 1997), changes in the developmental contexts in which behavior is observed (e.g., decreased need for sleep is something that could readily be observed by parents of school-aged children, but not of most college students), and heterotypic continuity making the content of certain items developmentally less appropriate even though the underlying mania still is manifesting.

To date, studies of phenomenology in PBD have concentrated on description of presentation within younger age groups, versus formal statistical comparison with adult samples. A recent meta-analysis of these studies (Kowatch et al., 2005) found that irritable mood was present in almost all cases of PBD. Elated mood, although not the most impairing symptom, was also present during at least one manic episode in the majority of cases across studies, regardless of whether elated mood was required as an inclusion criterion. In contrast, grandiosity appears to have lower sensitivity to bipolar disorder, occurring in only 60% of cases on average. Concerns have also been raised that irritability and grandiosity both may not be highly specific to bipolar disorder (Carlson, 2005). Hypersexuality had the lowest sensitivity of any of the symptoms examined, occurring in fewer than a third of cases. It is possible that physical maturation moderates the rate of hypersexuality, such that frequencies will increase with age.

The development of a 10-item measure to detect mania provides a statistical window into the features that best distinguish PBD from other disorders (Youngstrom, Frazier, Findling, & Calabrese, 2006). The 10 items were drawn from the pool of 73 items on the PGBI. Thus, the item pool included items describing depressed, manic, hypomanic, and mixed states. Additionally, the item pool consisted of clinically associated features as well as DSM symptoms, casting a broader descriptive net. Items selected for the short form demonstrated the largest statistical separation between two groups: Those with any bipolar spectrum diagnosis, versus all of the remaining participants in the protocol regardless of diagnosis. The best discriminating items were all members of the original Hypomanic/Biphasic Scale, despite the fact that unipolar depression and ADHD were included at roughly equal rates in the comparison group. The items assessing elated mood were among the 10 best discriminating items, whereas the items pertaining to grandiosity did not discriminate well enough for retention. Seven of the 10 best discriminating items included reference to periods of “extreme happiness or intense energy.” Irritable mood items were also among the most discriminating, but the PGBI items all embed irritable mood in the context of other mood symptoms. For example, one item asks, “Have there been times of several days or more when, although your child was feeling unusually happy and intensely energetic (clearly more than his/her usual self), he/she also had to struggle very hard to control inner feelings of rage or an urge to smash or destroy things?”

Examination of the most discriminating items also indicates that rapid shifts in mood and energy, although not DSM-IV symptoms strictly speaking, are highly indicative of PBD. The majority of the most discriminating items also were originally classified as “biphasic” or mixed state items by Depue et al. (1981). Both of these observations suggest that PBD is characterized by rapid shifts in mood and energy, with mixed episodes that involve rapid oscillation between different mood states. These data are consistent with clinical observations by others (Geller et al., 2003), and they are perhaps consistent with Kraepelin's observation that younger age cohorts showed more time in mixed states with a volatile and shifting mood presentation (Kraepelin, 1921). Interestingly, data from a large sample of adults with bipolar disorder found evidence of rapid mood shifts in a subset (44%) who, as a group, showed significantly earlier age of onset, higher rates of substance use and anxiety, more suicide attempts, and more relatives with a history of rapid mood switching (MacKinnon et al., 2003).

Recommendations

It is important that clinician judgment remain a component of research diagnosis and clinical case formulation. At the same time, even when using a standardized instrument, clinicians could form biased judgments about bipolar disorder, so parent and self-report should always be considered as well.

Research to date indicates that of all collateral informants, parents are the ones to pay attention to when the diagnostic consideration is bipolar disorder. Teacher and self-report are a distant second place and, at least with the state of the art instruments, either provides little incremental validity when a familiar parent's impressions are available.

Teachers and youth reports may still be useful. Given the reality that most clinical referrals are parent driven, it is no surprise that parent measure scores are higher than those from other informants. However, when parent report is unavailable (e.g., detention settings), cross-informant data will tend to prove useful.

The CBCL externalizing scale is sensitive to bipolar disorder (i.e., most cases will score high on it), but it is not specific to bipolar disorder (i.e., many cases will score high for reasons besides having bipolar disorder). The same appears true for the various “bipolar profiles” of scores derived from the CBCL. Thus, the CBCL is useful for identifying possible cases of bipolar disorder for further testing, but it should not be used in isolation to identify cases clinically or for research purposes. Bipolar-specific scales, such as the PGBI and P-MDQ will provide more specificity, while maintaining similar levels of sensitivity.

When evaluating the usefulness of a measure with respect to diagnostic efficiency, it is not enough to consider a single cut score. Rather, it is increasingly evident that, as base rates vary across settings, the LR approach gives test scores more meaning, because it enables estimation of the accuracy of the test result in each setting.

For any of the available assessment tools, a positive result should not be treated as if it were a definite diagnosis of bipolar disorder. Instead, “screen positives” should trigger more thorough and careful clinical evaluation.

Family history of illness and treatment response is an important domain for assessment.

The developmental continuity between what is being called PBD and the recognized adult version remains unclear. Ongoing longitudinal studies will continue to consider whether there are qualitative differences between the two phenomena. Emerging evidence suggests that complicated pediatric presentations will persist often into adolescence or young adulthood, without becoming more “classic” in bipolar presentation. This may actually represent homotypic continuity that has been obscured by changes in the diagnostic labels used with children versus adults experiencing the same symptoms of mood dysregulation.