Introduction
A major methodological issue in studying the epidemiology of personality disorders (PDs) is choice of assessment approach. There are two major clinical approaches for evaluating these disorders in the community. One approach is to evaluate PD criteria in the context of a systematic clinical examination; this is the typical clinical method of patient evaluation extended to include the comprehensive list of all PD criteria, each of which are defined and required to be assessed. The alternative approach is to use a semi-structured interview instrument that poses specific questions that target each PD trait (criterion) to be rated; this is the typical method in research studies. There is also a questionnaire approach (self-report), which is not discussed in this paper.
In a review of comparability studies, Perry (Reference Perry1992) reported poor concordance between direct-interview methods for diagnosing PDs in patients. He suggested that methods differ in their ability to adequately distinguish between long-standing patterns of affect, behavior and cognition that are characteristic of PDs, as opposed to sporadic occurrences of these traits. However, little is known about the agreement between different interview-based methods for evaluating PDs in the community. Therefore, in the current study, we evaluated the concordance between the above-mentioned two clinical assessment methods for diagnosing PDs in the community using item response analytic approaches designed to account for measurement error.
Method
Participants
As described previously (Samuels et al. Reference Samuels, Eaton, Bienvenu, Brown, Costa and Nestadt2002), individuals participating in the Hopkins Epidemiology of Personality Disorder Study (HEPS) were sampled from the Baltimore Epidemiologic Catchment Area Follow-up (Eaton et al. Reference Eaton, Anthony, Romanoski, Tien, Gallo, Cai, Neufeld, Schlaepfer, Laugharne and Chen1998). In brief, 3481 adult household residents of East Baltimore were sampled probabilistically in 1981 and interviewed using the Diagnostic Interview Schedule (DIS); 810 of these individuals were also examined by psychiatrists in the Clinical Reappraisal (Anthony et al. Reference Anthony, Folstein, Romanoski, Von Korff, Nestadt, Chahal, Merchant, Brown, Shapiro, Kramer and Gruenberg1985) in the same year. Between 1993 and 1996, 1920 (73%; 1920/3481) of the available participants were reinterviewed (many had died, as the original cohort was oversampled for elderly participants).
A stratified random sample of 1258 participants (of the aforementioned 1920 participants) were selected to participate in the HEPS, which was conducted between 1997 and 1999. These participants included: all available participants examined by psychiatrists in the Clinical Reappraisal in 1981; all participants identified by the DIS as having a lifetime diagnosis of mania, depression, panic disorder, obsessive–compulsive disorder, alcohol use disorders, or drug use disorders at follow-up, between 1993 and 1996; and a 25% random sample of the remaining participants who did not meet either of the two previous criteria (n=222). Of the 1258 subjects selected with these criteria, 516 could not be interviewed because they could not be traced, refused participation, had died, or were too ill or elderly to participate. A total of 742 subjects were included in the study, of whom 713 completed both personality assessments.
The gender and ethnic distributions of these 713 individuals were similar to those of the 516 who were not interviewed; however, the interviewed subjects were younger (mean age 47 years) than the non-interviewed subjects (mean age 61 years). The gender and ethnic distributions of the participants were similar to those of the 3481 subjects examined in 1981, although the study subjects were younger.
All participants provided signed, informed consent and the study was approved by the Institutional Review Board (IRB) at Johns Hopkins University.
Personality assessment
A total of 713 participants completed two independent, direct, in-person examinations, using the two above-mentioned methods for the assessment of PDs. One method involved a thorough examination of psychiatric symptoms and PD criteria conducted by psychiatrists, using the Personality Disorder Schedule (PDS) of the Standardized Psychiatric Examination (SPE) to assess DSM-IV PDs (Romanoski et al. Reference Romanoski, Nestadt, Chahal, Merchant, Folstein, Gruenberg and McHugh1998). Primarily an unstructured but systematic clinical method, the assessment of PD criteria relies on information from direct questioning about specific personality criteria, in addition to information emerging during the participant–examiner interaction. These examinations were conducted by five board-eligible psychiatrists. The psychiatrists collected biographical information including schooling, employment and relationship history, with an emphasis on interpersonal relationships, as an ingredient for PD assessment. The assessment is designed to ensure that these domains are completed in the examination. The aim was to identify emotional responses across various situations and circumstances. A glossary definition for each PD criterion was provided and, for certain criteria, questions to use as diagnostic prompts were provided. Each PD criterion was rated ‘0’ (absent), ‘1’ (accentuated or exaggerated), ‘2’ (severe) or ‘9’ (missing). The psychiatrists did not make PD diagnoses; these were computed based on the criteria endorsed using a DSM- IV algorithm.
The other method was an interview conducted by clinical psychologists, using the International Personality Disorder Examination (IPDE; Loranger et al. Reference Loranger, Sartorius, Andreoli, Berger, Buchheim, Channabasavanna, Coid, Dahl, Diekstra, Ferguson, Jacobsberg, Mombour, Pull, Ono and Regier1994), a semi-structured method that asks participants directly about each specific personality criterion, and queries informants about a subset of the personality criteria. Four masters-level clinical psychologists conducted these interviews. The psychologists evaluated PD criteria manifest over the participant's entire adult life. For each criterion, the definition was provided, in addition to one or more questions to be used as diagnostic prompts; the psychologists were encouraged to ask clarifying questions. Criteria were rated ‘0’, ‘1’, ‘2’ or ‘9’, as defined for the PDS. Following the interview, the psychologists interviewed at least one informant by telephone, using questions from the IPDE. The psychologists were required to evaluate more than half of the criteria for each of the PDs. Discrepancies between informants and participant ratings were resolved by the psychologists' judgment on the evidence available for each criterion when recording a final score. In 40 interviews of participants rated jointly by the psychologists, the intraclass correlation coefficients (ICCs) for the number of PD criteria rated as present were schizoid (0.81), schizotypal (0.58), paranoid (0.63), antisocial (0.80), borderline (0.76), histrionic (0.62), narcissistic (0.62), avoidant (0.89), dependent (0.76), and obsessive–compulsive (0.70) (Samuels et al. Reference Samuels, Nestadt, Romanoski, Folstein and McHugh1994).
The PDS and IPDE were conducted independently, and the psychiatrists and psychologists were blind to each others' assessments. Both methods rate each of the DSM-IV PD criteria and allow diagnoses for each of the 10 DSM-IV PDs (APA, 2000).
Statistical approach
The analytic approach evaluating concordance is based on, and parallels, that described in detail in Nestadt et al. (Reference Nestadt, Di, Samuels, Bienvenu, Reti, Costa, Eaton and Bandeen-Roche2010). Conceptually, we construe PD to be dimensional; that is, envision a given disorder as a manifestation of an underlying severity ranging along a continuum from low (no disorder) to high. The IPDE or PDS uses a set of criteria that are endorsed if the underlying severity exceeds a threshold level. We consider such thresholds to be potentially person specific, and hence subject to random uncertainty. The effect of such uncertainty is to attenuate associations and hence standard measures of concordance; the item response approach we propose here explicitly addresses and attempts to correct for such attenuation. Thresholds triggering criterion endorsement are almost certainly criterion and instrument specific. Both analytic approaches proposed here use measures of association, and hence concordance, that are robust to this sort of specificity. Finally, instruments may target fundamentally different constructs of disorder, so we conceptualize parallel severity scores for individuals, one for the IPDE construct and one for the PDS. We make concordance operational as the correlation between the two available representations of a person's disorder severity, one inferred through the IPDE and the other inferred through the PDS. Concordance as so defined should be high if the two representations rank individuals similarly as to their disorder severity.
In addition to the framework described above, there were also practical considerations. The two assessment methods differed considerably with respect to the distributions of the number of endorsed criteria. This suggested that there were systematic differences between the levels of severity that ‘triggered’ a positive endorsement of criteria. Moreover, the criteria count distributions were extremely skewed with a preponderance of 0 counts. These features may invalidate the use of standard analyses of concordance between criteria; the approach we now outline accommodates both of these practical complexities.
Our analytic approach was grounded in item response analysis (IRA; Lord, Reference Lord1980); to optimize our confidence in its findings, we implemented parallel versions of it. Take Y ijk to represent the presence (0 or 1) of the kth criterion (k=1, …, K) using the jth method (j=1 for IPDE, j=2 for PDS) for some ith person, for a given disorder. Let d ij be the associated severities of disorder for IPDE (j=1) and PDS (j=2) constructs; these are envisioned as z scores (normally distributed variables with mean 0 and variance 1). The model relating criteria to disorder severity is a random effects item response theory (IRT) model:
In IRT terminology, −(γj+βjk) gives the item k ‘difficulty’ parameter for each method j, λjk is the discrimination parameter, and d ij is the ‘ability’. In our context −(γj+βjk) distinguishes criteria by their severity, such that higher ‘difficulty’ identifies a more rarely exhibited criterion. In our context λjk distinguishes criteria by how precisely they measure disorder severity. In the standard diagnostic terminology of sensitivity and specificity, higher ‘discrimination’ identifies a criterion whose reporting is more sensitive and specific to the underlying disorder among two criteria of equal difficulty, lower difficulty identifies a criterion that is more sensitive among two equally discriminating criteria, and higher difficulty identifies a criterion that is more specific among two equally discriminating criteria. Our statement of (1) separates the criterion difficulty into a disorder-wide difficulty (−γj) and terms differentiating difficulty of each criterion from the disorder-wide difficulty (−βjk). This interpretation is achieved by constraining βj1+…+βjK=0, as in classical ANOVA. The resultant measure of concordance in an individual's assessments is the correlation between a person's two representations of disorder severity as inferred from the IPDE and PDS, d i1 and d i2.
Model (1) describes systematic differences in the assessment of disorder severity between the methods through γ2−γ1, β2k−β1k and λ2k−λ1k. We shall say that a method with higher γ, or a criterion with a higher β, tends to elicit disorder at a lower severity threshold than that with a lower such value, and a criterion with a higher λ has a higher precision for detecting the disorder. Exp(γj) can be interpreted as the average odds of exhibiting disorder for a person with mean underlying disorder severity, where the (geometric) average is taken over all the criteria defining the disorder. Similarly, exp(βjk) gives the factor by which the odds of exhibiting the kth trait is multiplied compared to the average, that is by which the kth criterion is relatively more or less frequently exhibited among the criteria defining the disorder.
Findings from an analysis based on equation (1) are potentially influenced by the assumption of normally distributed severity scores. To assure robustness of our findings to this assumption, we implemented a second item response model focused entirely on describing criterion prevalence, without explicitly building in an underlying disorder severity:
Here, γj and βjk once again denote disorder and criterion difficulties using method j, except that they represent population-wide averages rather than values for persons with the mean disorder severity. This second approach is a generalized estimating equations (GEE) model (Zeger & Liang, Reference Zeger and Liang1986), in which the concordance in an individual's assessments is estimated by a summary of the pairwise associations between a person's criterion ratings from the different methods (see equation 3 and associated text in Nestadt et al. Reference Nestadt, Di, Samuels, Bienvenu, Reti, Costa, Eaton and Bandeen-Roche2010), across all possible pairs of criteria representing the disorder. Similarly as with the criterion prevalence models (1) and (2), we decomposed this measure of concordance into an overall measure (averaged over traits, within disorder) and amounts by which concordance for different traits differs from the disorder-wide average. Importantly, concordance estimated by the approach just described is for measured disorder, not accounting for differences in the measurement error with which traits assess the underlying disorder. By contrast, model (1) estimates across-method concordance in assessing underlying disorder, correcting for random error in criterion measurement. We previously developed a method to estimate underlying disorder concordance using the approach described in this paragraph (see equations 4–7 in Nestadt et al. Reference Nestadt, Di, Samuels, Bienvenu, Reti, Costa, Eaton and Bandeen-Roche2010); we use the same method here.
Statistical analyses
All analyses were conducted for each of the 10 DSM-IV PDs, individually by disorder. Criteria ratings of ‘2’ were rare; therefore we counted a criterion as ‘present’ if rated at level ‘1’ or higher and absent otherwise. To permit comparability to standard analyses, counts of present criteria were cross-tabulated by the two assessment methods (IPDE and PDS) and ICCs between the IPDE and PDS counts were computed. κ agreement statistics were computed after merging criteria counts into categories of 0, 1–2 and 3. Pairwise odds ratios (ORs) were used to assess the association between a criterion assessed by one method and a criterion assessed using the other method for all possible criteria pairs (same and different); these assess the relative likelihood of the criterion assessed by PDS being present if the trait assessed by IPDE was, or was not, present. For several criteria spanning several disorders, no participants reported the criterion in either assessment, and these were eliminated from further analysis.
Latent trait analyses corresponding to equation (1) were fit using software written in Winbugs (www.mrc-bsu.cam.ac.uk/bugs/) to support the work of Nestadt et al. (Reference Nestadt, Di, Samuels, Bienvenu, Reti, Costa, Eaton and Bandeen-Roche2010). Analyses corresponding to equation (2) were conducted using GEEs (Huang et al. Reference Huang, Bandeen-Roche and Rubin2002) implemented in the R statistical computing language (www.r-project.org/); one to estimate (2) and a second to estimate the concordance (Heagerty & Zeger, Reference Heagerty and Zeger2000). Each analysis defined criteria as present or absent as described in the previous paragraph. For three disorders the algorithm for fit of the latent trait analyses was aborted by the software absent a successful fit, indicative of too low a frequency of reporting multiple criteria together to obtain a stable estimate of the latent trait model parameters for association; for these, we report the GEE analyses only.
Results
Demographic characteristics
The demographic characteristics of the 713 participants are shown in Table 1. About 60% of the participants were women, and 60% were white. The participants ranged in age from 34 to 94 years; most were younger than 55 years old, and the mean age was 51 years. Most participants were married; 25% were separated or divorced. About one-third of the participants were high-school graduates, and another third had post-high-school education. The annual household income was less than US$20 000 for approximately one-third of the participants, and more than US$50 000 for another third.
Demographic characteristics of participants examined with both personality disorder (PD) assessment methods (PDS and IPDE; n=713)
PDS, Personality Disorder Schedule; IPDE, International Personality Disorder Examination.
Values given as n (%).
Concordance between methods for counts of PD traits
There was substantial variation between PDs in the degree of concordance of criteria counts as assessed by IPDE and PDS (Table 2). For all disorders, participants were considerably more likely to be rated as having one or more criteria by the IPDE than by the PDS method. ICCs ranged from 0.06 to 0.27, and κ statistics ranged from 0.05 to 0.17; thus concordance between raw trait counts was poor. The best agreement occurred for histrionic personality, and the poorest agreement for avoidant PD. Median (over traits) pairwise ORs for association of trait presence between methods ranged from 1.7 (obsessive–compulsive) to 6.3 (antisocial), the latter indicating a more than sixfold higher odds for rating an antisocial trait as present in the PDS if a participant was rated, rather than if they were not rated, as having the trait in the IPDE. These can be roughly compared to the ICCs using the Digby transformation from ORs to correlation coefficients (Digby, Reference Digby1983); the corresponding range on a correlation scale is 0.20–0.60. That the ORs indicate a higher degree of concordance than the ICCs can be explained by the fact that the former are invariant to differences in trait prevalence across methods, whereas the latter penalize for such.
Concordance between diagnostic methods for counts of personality disorder (PD) criteriaFootnote a
PDS, Personality Disorder Schedule; IPDE, International Personality Disorder Examination; ICC, intraclass correlation coefficient; OR, odds ratio.
a Criteria were counted as positive if they were ‘present’ (rating of 1 or 2).
b κ was computed from 3×3 tables, with the number of endorsed criteria categorized into 0, 1–2 or ⩾3.
c Median cross-method pairwise OR.
Also provided in Table 2 is the proportion with any positive criterion, for each disorder. Taken as a measure of prevalence, this measure construes PD to be dimensional with no obvious cutoff, such that endorsement of any criterion constitutes a ‘hit’ reflecting the first detectable indication of disorder with the available data. However, in the present paper listing of these proportions mainly aims to convey the frequency of positive responses, as we have defined them, and a comparison of these frequencies across the two instruments. The data available for analysis, for most of the disorders, were not unduly sparse.
Systematic between-method differences in PD prevalence
Table 3 summarizes systematic differences in disorder prevalence as assessed by IPDE and PDS, for each disorder, as analyzed by the GEE method [equation (2)]. In that method, the ORs estimate the factors exp(γ2)/exp(γ1) by which the geometric average odds of trait presence across traits were higher or lower using the PDS as compared to the IPDE. For histrionic disorder there was no significant difference between the methods [OR 1.03, 95% confidence interval (CI) 0.85–1.26]. However, for all other PDs, the estimated disorder prevalence was significantly lower using the PDS than the IPDE. ORs ranged from 0.68 for obsessive–compulsive disorder, indicating a 32% decrease in odds of trait presence, to 0.11 for narcissism, indicating a 89% decrease in the odds; the odds for all other PDs were more than 60% lower using the PDS compared to the IPDE. However, within disorders there was considerable heterogeneity in individual traits about this global trend. For example, for obsessive–compulsive disorder, the trend to lower PDS trait prevalence was statistically significantly mitigated, and even reversed, for traits of perfectionism, excessive devotion to work and over-conscientiousness, and statistically significantly exacerbated for traits of inability to discard worthless objects, reluctance to delegate and miserliness. Table 3 provides data to identify items that oppose the prevailing distinction between the two methods (as in ‘perceives attacks’ for paranoid PD) and to characterize systematic differences in the modes of assessment.
Item response analysis (IRA) of specific personality disorder (PD) criteria using a GEE approach; evaluation of systematic differences in prevalence rates between clinical methods
GEE, Generalized estimating equation; CI, confidence interval; IPDE, International Personality Disorder Examination; PDS, Personality Disorder Schedule.
a Log odds averaged and exponentiated. 95% Confidence interval in parentheses.
b Factor by which odds of reporting criterion in PDS versus IPDE.
* Statistical significance at or below the α=0.05 level.
Primary concordance findings
Both latent trait and GEE analyses indicated moderate concordance between IPDE and PDS assessments of persons' underlying disorder severities (Table 4). Latent trait-based estimates of correlation between the IPDE and PDS representations of disorder severity ranged from 0.33 to 0.78; those for schizotypal, antisocial and borderline disorder exceeded 0.60. These directly estimate concordance between the underlying severity constructs targeted by the two methods and thus adjust for random, normally distributed measurement error, in addition to systematic error associated with different thresholds for registering the presence or absence of a trait; as a result they assess the degree to which the methods rank persons similarly in their degrees of PD. To a great extent, the corresponding models found the precision of PDS traits for assessing disorder to be higher than IPDE traits (higher λ estimates; data not shown). GEE-based estimates for concordance in traits as assessed by the two methods, assessing similarity of the methods' rankings of persons but not accounting for measurement error, ranged from 0.17 to 0.46. Their counterparts upon applying adjustment for measurement error so as to estimate correlations between the two available representations of underlying disorder increased to the same range as for latent trait analyses. These closely approximated the latent trait estimates of the same quantity, agreeing to within five hundredths for all but two disorders and falling within the 95% CI for the latent trait-based concordance for all disorders where a latent trait estimate was possible.
Item response analysis (IRA) of personality disorder (PD) concordance between clinical methods
GEE, Generalized estimating equation; CI, confidence interval; IPDE, International Personality Disorder Examination; PDS, Personality Disorder Schedule.
For each disorder and method, some PD criteria were identified that seemed to reflect the underlying disorder more or less precisely than others (Table 4). For instance in obsessive–compulsive PD, preoccupation with details, perfectionism and reluctance to delegate, as measured with the IPDE, exhibited enhanced precision; whereas inability to discard worthless objects had the least precision; the other criteria, whether measured by the PDS or the IPDE, had an intermediate performance.
Discussion
In this study the level of concordance between two PD clinical assessment methods was investigated in a population-based sample of 713 individuals. In a direct comparison of the methods, using traditional statistical approaches that compare methods (ICC, κ, and OR), the level of concordance is relatively poor. Based upon those analyses we should conclude that there is poor agreement between PD assessment methods, thus limiting their application in research studies and clinical practice. However, we recognize that there are inherent limitations in clinical measurement of personality attributes that do not necessarily invalidate their utility. First, there is no ‘gold standard’ to calibrate PD measurement. Second, the assessment requires the distillation of clinical judgments over time, circumstance, and with regard to related sets of emotion and behavior; this complicates the assessment beyond that required for the evaluation of a contemporaneous symptom. These aspects of assessment may lead to differing thresholds for criterion endorsement; this is prominently observed in this study.
Substantial differences in the two methods of assessment of trait prevalence indicate substantial systematic differences in the severity thresholds at which IPDE and PDS criteria detect disorder. Overall, PDS-based criteria prevalence was considerably lower than IPDE-based criteria prevalence for all disorders except histrionic PD. This indicates that IPDE-based criteria tend to be endorsed at a lower threshold of disorder severity than PDS-based criteria. However, there was also considerable heterogeneity in the degree to which individual criteria paralleled the overall trend.
These limitations need not be construed as necessarily an absence of validity of the measurement; and it is precisely for this reason that we adopted two IRT methodologies that take these factors into consideration. Both analytic strategies address measurement error; the latent IRT method considers that the manifest measures (scores endorsed in the diagnostic instrument) are approximations to an underlying or latent trait whose measurement represents the ‘true’ trait; the commonality-adjusted GEE requires fewer analytic assumptions in its procedure to adjust for measurement error. The findings that emerge after applying both statistical methods to the data are encouraging.
The concordance between the PDS and IPDE ranged between 0.32 and 0.78 for the PD scale measures, as estimated using IRT methods. For seven of the 10 PDs tested, the ICC was greater than 0.54. In practice, this means that the methods infer disorder severities that rank individuals moderately to highly similarly on their PD, provided that differences in endorsement thresholds and precision of criteria for detecting disorder are accounted for within and across methods. The PD scales with the lowest concordance estimates were histrionic, obsessive–compulsive and paranoid (and according to the latent IRT, avoidant). This suggests that these PDs are more difficult to assess.
The findings from this study are mixed. It is evident that the methods cannot be substituted directly, although they do correspond moderately once imprecision and differential endorsement thresholds are accounted for. Moreover, both approaches seem to be tapping the same (or similar) latent PD construct. This suggests that the two assessments are successfully identifying the construct based upon the individual units (criteria) of measurement, but that the individual units themselves are less reliable. It could be concluded that better assessment methods that account for measurement differences are needed. This could be accomplished by the identification of criteria that correspond more uniformly between assessment methods.
Nevertheless, the findings in this study suggest that endorsement, or lack thereof, of individual criteria is not necessarily reliable, but that endorsement of members of the group of criteria underlying each PD provides a viable diagnostic approach.
Limitations
The methods examined in this study are not the only available approaches. Other diagnostic interviews, such as the Structured Interview for DSM-IV Personality (SIDP-IV) or Structured Clinical Interview for DSM-IV Axis II Personality Disorders (SCID-II), may have given different results. The PDS was designed for this study in 1981, as there were no alternative instruments available at that time. Additional items were subsequently added to include changes in the criteria required by DSM-III-R and DSM-IV, although the original items were maintained for consistency. Psychiatrists used the PDS whereas masters-level psychologist conducted the IPDE. Both groups were trained extensively and ratings were reviewed by the respective group to maintain consensus across the study period. The examinations were not conducted simultaneously; however, the majority occurred with a 3-week period of each other. The PDS assessment was conducted as part of a complete diagnostic examination for Axis I psychopathology; this may have led to certain personality traits being construed as symptoms of an Axis I disorder and not endorsed for the PD assessment.
The sample, selected from the community-residing population of eastern Baltimore, was overselected for individuals with Axis I disorders, and therefore the results may differ from a true community survey population. Nevertheless, these participants were not selected from a treated population, and therefore are more representative of the performance of these measures without the influence of ‘training’ of patients that occurs in a treated sample. This ascertainment strategy identifies fewer individuals with frank PD resulting in a more strenuous test of the instrument performance.
This is an older cohort because of the long follow-up. The prevalence of certain PDs declines with age and the stability of the PDs is lower in early adult life. It is plausible that the findings may have been influenced by age.
This study focused on the estimation of concordance between IPDE and PDS assessment of specific DSM-IV PDs. It makes no attempt to establish that PD as so defined is well discriminated between disorders. Indeed, our prior work indicates that the criteria may be considerably associated across different disorders (Nestadt et al. 2006). Moreover, the methodology applied here may well heighten the estimates of association across PD relative to an analysis not endeavoring to correct for measurement error in assessment or differential thresholding by criterion or method. Ultimately, we suspect that the current 10-PD schema lacks strong discriminant validity and a different classification than the 10-PD schema currently used by DSM up to the present may lead to superior diagnoses for the identification and tracking of PD for clinical and epidemiological purposes. This important topic is of sufficient scope to merit a study in its own right.
Implications
The statistical approaches for the comparison of PD data assessed by different raters require careful consideration, given that the aforementioned estimates of concordance and methods applying standard ICC- and κ-based approaches to symptom counts differ greatly. Careful identification of the target of concordance assessment is also required. Concordance of underlying severity has to do with the extent to which inferred rankings of individuals by their severity agree, after accounting for differences in measurement properties; concordance of clinically assessed traits has to do with the extent to which rankings of individuals by their severity agree, as if all symptoms were equally reliably assessed; and the standard analyses measure agreement of symptom reports per se. Which of these is the most relevant will vary by situation and potentially will depend on the adjudication of the individual reader.
We interpret the findings of this study to suggest that concordance in the raw measurement of the individual PD criteria between the two clinical methods is lacking. However, based on two statistical methods that adjust for differential endorsement thresholds and measurement error in the assessment, we can deduce that the PD constructs themselves can be measured with a moderate degree of confidence regardless of the clinical approach used. It may be possible to strengthen this confidence in the constructs by refining the constructs themselves; for example by taking advantage of alternative PD constructs than those defined in the DSM-IV. Several studies have posited four or five PD dimensions that encompass the entire domain (e.g. Nestadt et al. 2006). The constructs can also be strengthened by identifying the specific traits that are most useful in the measurement of the PD constructs. The findings of this study suggest that the individual criteria for each PD are, in and of themselves, less specific for diagnosis, but as a group the criteria for each PD usefully identify specific PD constructs.
Acknowledgments
This study received support from National Institutes of Health grants MH50616, MH64543 and DA026652.
Declaration of Interest
None.
