Panic disorder (PD) is a common mental disorder defined by the occurrence of unexpected panic attacks and a persistent concern about their recurrence or consequences (American Psychological Association (APA), 2000). Many studies across the world have confirmed that PD can be a very debilitating condition with significant individual and societal costs (Bandelow & Michaelis, Reference Bandelow and Michaelis2015; Gadermann, Alonso, Vilagut, Zaslavsky, & Kessler, Reference Gadermann, Alonso, Vilagut, Zaslavsky and Kessler2012; Kessler, Petukhova, Sampson, Zaslavsky, & Wittchen, Reference Kessler, Petukhova, Sampson, Zaslavsky and Wittchen2012).
The Panic Disorder Severity Scale (PDSS; (Shear et al., Reference Shear, Brown, Barlow, Money, Sholomskas, Woods and Papp1997) is one of the most used scales for assessing panic severity. The PDSS is a clinician-administered, instrument, developed to assess overall PD severity as well as its associated features (panic attack frequency, associated distress, anticipatory anxiety, agoraphobic and interoceptive fear/avoidance and work and social impairment). The psychometric properties of the original English PDSS have been well investigated (Shear et al., Reference Shear, Rucci, Williams, Frank, Grochocinski, Vander Bilt and Wang2001) and it has been adapted to several languages including Korean, Turkish, or Japanese (Lim, Yu, & Kim, Reference Lim, Yu and Kim2007; Monkul et al., Reference Monkul, Tural, Onur, Fidaner, Alkin and Shear2004; Yamamoto et al., Reference Yamamoto, Nakano, Watanabe, Noda, Furukawa, Kanai and Otsubo2004), (see Table 1). However, there is not any Spanish validation of the scale, which limits the possibilities of cross-cultural research. Furthermore, some discrepancies have been found regarding the structure of the scale using factor analytic techniques, with some studies (Shear et al., Reference Shear, Rucci, Williams, Frank, Grochocinski, Vander Bilt and Wang2001; Yamamoto et al., Reference Yamamoto, Nakano, Watanabe, Noda, Furukawa, Kanai and Otsubo2004) suggesting a one-factor and others (Houck, Spiegel, Shear, & Rucci, Reference Houck, Spiegel, Shear and Rucci2002; Lim et al., Reference Lim, Yu and Kim2007; Monkul et al., Reference Monkul, Tural, Onur, Fidaner, Alkin and Shear2004; Shear et al., Reference Shear, Brown, Barlow, Money, Sholomskas, Woods and Papp1997) presenting data that fit a two-factor solution. Also, to the best of our knowledge, no previous research has investigated whether the degree of expertise of the interviewer may have an effect on inter-rater reliability.
Summary of Previous Studies on the Psychometric Properties of the PDSS
Note: ICC = Intraclass Correlation Coeficient; PD = Panic disorder; ASI = Anxiety Sensitivity Index; SSS = Subjective Symptoms Scale; ADIS-R-severity = Clinical severity rating of PD of the Anxiety Disorders Interview Schedule-Revised; APPQ-agora = agoraphobia factor of the Albany Panic and Phobia Questionnaire; APPQ-intero = interoceptive factor of the Albany Panic and Phobia Questionnaire; BAI = Beck Anxiety Inventory; FQ-Agora = Agoraphobia Subscale of the Fear Questionnaire; PAS-o = Panic and Agoraphobia Scale observer-rating; PAS-sr = Panic and Agoraphobia Scale self-report; HAM-A = Hamilton Anxiety Rating Scale; BDI = Beck Depression Inventory; CGI = Clinical Global Impression; FQ = Fear Questionnaire; MI = Mobility Inventory for Agoraphobia; SCL–90–R–anx = Anxiety Subscale of the Symptom Checklist–90 Revised; SCL–90–R–PhobAnx = Phobic Anxiety Subscale of the Symptom Checklist–90 Revised; ACG = Agoraphobic Cognitions Questionnaire; BSQ = Body Sensations Questionnaire; WHLS-WH = Work, Home Management, Social and Private Leisure Activities Scale, average of work and home subscales; WHLS-WH = Work, Home Management, Social and Private Leisure Activities Scale, social leisure subscale.
Our main goal was to conduct the first psychometric study of the Spanish version of the PDSS. We assessed several psychometric properties of the scale as well as its factor structure. Additionally, we evaluated whether a different degree of expertise in anxiety disorders influenced inter-rater reliability.
Method
Sample
The sample consisted of 94 adult PD patients (86 with and 8 without agoraphobia), who were assessed at a hospital-based anxiety disorders clinic in Barcelona (Spain). Their mean age was 35.9 (SD = 9.8). Thirty-three were males (35%), 61 females. The mean age at onset of PD was 27.1 years (SD = 9) and approximately one fourth (24,4%) had at least one comorbid mental disorder. The most frequent current comorbidities were Major Depressive Disorder (5.3%) and Social Phobia (5.3%). DSM-IV (APA, 2000) PD diagnosis was established using the validated Spanish version (Bobes, Portilla, Bascarán, Saiz, & Bousoño, Reference Bobes, Portilla, Bascarán, Saiz and Bousoño2002) of the Mini International Neuropsychiatric Interview (MINI) (Sheehan et al., Reference Sheehan, Lecrubier, Sheehan, Amorim, Janavs, Weiller and Dunbar1998) by two experienced clinicians. The Spanish version (5.0.0) of the MINI has shown to have sound psychometric properties (Bobes, Reference Bobes1998).
Exclusion criteria were age under 18 or over 69, current bipolar, psychotic, or substance abuse disorder other than nicotine; and intellectual disability or language barriers. Patients could have comorbid mood and anxiety disorders as long as PD was considered the primary source of distress as assessed by the MINI. The study was approved by the ethical committee of the Hospital del Mar. A brief description of the purpose of the study was given and written informed consent was obtained for all participants.
Procedure
Participants completed the self-report measures (see below) and were interviewed with the PDSS by a clinician with a high degree (> 10-year postgraduate experience) of expertise in anxiety disorders (“gold standard”). The PDSS interview was video-recorded and later independently rated by two different clinicians with a different level of expertise in anxiety disorders. The second rater was a clinical psychologist with 2-year experience in assessment/treatment of anxiety disorders. The third rater was a psychiatrist with general experience in psychopathology but no specific experience in anxiety disorders. Two weeks later, the PDSS was again assessed on the phone to test for test-retest reliability in a subsample of participants (n = 20).
Measures
The Panic Disorder Severity Scale (PDSS) (Shear et al., Reference Shear, Brown, Barlow, Money, Sholomskas, Woods and Papp1997) is a 7-item instrument, rated on five-point Likert scales (0 = not at all to 4 = most severe). The total score ranges from 0–28, with higher scores indicating greater PD severity. The original version of the PDSS was translated into Spanish and back translated, under the supervision of an English native speaker. Two of us (MAF and GF) then reviewed the translated version, and had it examined again by a Spanish philologist, in order to verify the accuracy of the translation. The Spanish PDSS is available from the corresponding author upon request.
The Anxiety Sensitivity Index–3 (ASI–3) is an 18-item self-report measure that assesses fear of anxiety symptoms (Taylor et al., Reference Taylor, Zvolensky, Cox, Deacon, Heimberg, Ledley and Bouvard2007). The total score ranges from 0 to 72 and consists of three dimensions (physical, social and cognitive, range 0–24 each). The Spanish version has shown to have excellent internal consistency, test-retest reliability and divergent validity, and adequate convergent validity (Sandín, Valiente, Chorot, & Santed, Reference Sandín, Valiente, Chorot and Santed2007).
The Panic and Agoraphobia Scale self-rating form (PAS) is a 13-item self-report measure assessing panic/agoraphobia symptoms (Bandelow, Reference Bandelow1995). The total score ranges from 0 to 52 and consists of 5 subscales (panic attacks, agoraphobia, anticipatory anxiety, disability and worries about health). The Spanish version (Saiz et al., Reference Saiz, Martinez-Barrondo, Garcia-Portilla, Corcoran, Morales, Bascaran and Bobes2009) was used.
The Beck Anxiety Inventory (BAI) is a 21-item self-report measure that assesses the severity of anxiety symptoms. The total score ranges from 0 to 63. The Spanish version has shown to have high internal consistency and adequate discriminant validity (Sanz & Navarro, Reference Sanz and Navarro2003).
The Beck Depression Inventory-II (BDI-II) is a 21-item self-report scale designed to measure depressive symptoms (Beck, Steer, & Brown, Reference Beck, Steer and Brown1996). Total score ranges from 0 to 63. The Spanish version has shown excellent internal consistency and convergent validity, and adequate divergent validity (Sanz, García-Vera, Espinosa, Fortún, & Vázquez, Reference Sanz, García-Vera, Espinosa, Fortún and Vázquez2005).
The Panic Disorder Severity Scale Self Report version (PDSS-SR) is a 7-item self-report measure assessing PD severity (Houck et al., Reference Houck, Spiegel, Shear and Rucci2002). The total score ranges from 0–28. The Spanish version has shown excellent internal consistency, adequate convergent validity and good test-retest reliability (Santacana et al., Reference Santacana, Fullana, Bonillo, Morales, Montoro, Rosado and Bulbena2014).
The time frame of the PAS, BAI, BDI and PDSS-SR was changed from one week to one month to fit the evaluated period of the PDSS.
The Clinical Global Impression-Severity scale (CGI) is a 7-point scale measure that assesses global clinical severity (Guy, Reference Guy2000). Total score ranges from 1 to 7.
Valid data for each instrument varied: ASI–3, n = 90, PAS, n = 90, BAI, n = 90, BDI-II, n = 91, CGI, n = 91.
Data analysis
The approach to validating the Spanish PDSS involved four steps. First, three indexes of reliability were established: internal consistency, using Cronbach’s alpha; test-retest reliability, using Pearson’s Correlation; and inter-rater reliability, using intraclass correlation coefficient (ICC). Cronbach’s alpha between .80 and .70, and .90 and .80 are considered to describe acceptable and good internal consistency respectively (Cortina, Reference Cortina1993). Second, structural validity was tested by means of confirmatory (CFA) factor analytic procedures based on the item-level polychoric correlation matrix. Our analyses examined the one-, two- and modified two-factor PDSS models. Third, convergent validity was examined by calculating Pearson correlations between the total and item scores of the PDSS with scores of the BAI, ASI–3, and PAS (self-report measures thought to reflect the item content) and by calculating the ICC between the total scores of the PDSS and the self-report version of the instrument (PDSS-SR). Divergent validity was examined by calculating Pearson correlations between the total and item scores of the PDSS with scores of the BDI-II.
To evaluate the size of correlations, values >.50 will be defined as large, from .30 to .49 as medium, and from .10 to .29 as small following Cohen’s criteria (Cohen, Reference Cohen1997). Fourth, the validity of the PDSS total score as a severity indicator was assessed by calculating the correlation between the PDSS total score and the CGI Severity Scale score.
All analyses were carried out with SPSS (version 18) except factor analyses that were conducted using Mplus version 6 (Muthén & Muthén, Reference Muthén and Muthén2012). Statistical significance was set at p < .05. All statistical tests were two-tailed.
Results
Internal consistency
Cronbach’s alpha for the PDSS was .74. The mean scores of items, the standard deviation, the resulting alpha by omitting items from the scale, and the corrected-item correlation (i.e. the correlation of each item with the sum of the remaining items) are shown in Table 2. Although the corrected item-total correlations were low (specially for Item 5), results indicate an acceptable internal consistency that would not increase with the omission of any item.
Mean, Standard Deviation, Correlation of Each PDSS Item with the Sum of the Other Items, and Internal Consistency (Cronbach’s alpha) if the Item was Deleted (n = 81)
Test-retest reliability
See Table 3 for the Pearson correlations coefficients for each item as well as the total score of the PDSS. Results suggest an overall excellent test-retest reliability except for Item 7.
Test-Retest Coefficients for the PDSS Total Score and Individual Items in Panic Disorder Patients (n=18)
Note: *p < .05; **p < .01.
Inter-rater reliability
Table 4 presents the results for inter-rater reliability for each item as well as the total score of the PDSS. Most ICC values for the total score were > .90, demonstrating excellent inter-rater agreement for overall scores, independently of the expertise of the rater. ICC values of individual items were all above .70 except for Items 3 and 5.
Inter-Rater Reliability of the Spanish PDSS
Note: Expert = Gold standard; Rater 1 = Clinician psychologist with 2-year experience in assessment/treatment of anxiety disorders; Rater 2 = Psychiatrist with general experience in psychopathology but no specific experience in anxiety disorders; CI = Confidence intervals.
Confirmatory factor analysis
On the basis of previous research (Kim, Reference Kim2001; Lim et al., Reference Lim, Yu and Kim2007; Monkul et al., Reference Monkul, Tural, Onur, Fidaner, Alkin and Shear2004; Shear et al., Reference Shear, Brown, Barlow, Money, Sholomskas, Woods and Papp1997; Shear et al., Reference Shear, Rucci, Williams, Frank, Grochocinski, Vander Bilt and Wang2001; Yamamoto et al., Reference Yamamoto, Nakano, Watanabe, Noda, Furukawa, Kanai and Otsubo2004), we conducted a CFA to test a one, a two and a two-modified factor for the scale. In the two-factor model, the first two items –that measure physical symptoms- loaded on one factor and the other five items loaded into another factor. In the two-factor modified model, errors of Items 4 and 7 were correlated. As usual, each item was specified to load on only one symptom factor; factors were allowed to correlate with each other, and error covariances were constrained to zero. The model fit was assessed with “traditional” Maximum-Likelihood chi-square (χ2 ML), Comparative Fit Index (CFI) (Bentler, Reference Bentler1990), Tucker Lewis Index or Non-normed Fit Index (TLI), Standardized Root Mean Square Residual (SRMR), and Root Mean Square of Approximation (RMSEA) (Steiger, Reference Steiger1990). The fit indices for the three stated models are shown in Table 5. The two-modified factor model significantly improved the fit (nested χ2 = 14.01, df = 12, p < .001) and the fit indices reached optimal values fulfilling goodness-of-fit criteria, since CFI and TLI were closer to one (Hu & Bentler, Reference Hu and Bentler1999) and SRMR and RMSEA were lower than .08 (Hu et al., 1999) and .07 (Steiger, Reference Steiger2007) respectively. Factor loadings for the CFA are available from the first author upon request.
Confirmatory Factor Analysis for the PDSS
Note: df = degree of freedom; χ2 = Maximum likelihood Chi-squared; CFI = Comparative Fit Index, TLI = Tucker Lewis Index or Non-normed Fit Index; SRMR = Standardized Root Mean Square Residual; RMSEA = Root Mean Square Error of Approximation.
1 Errors of items 7 and 4 are correlated.
Convergent and divergent validity
Tables 6 and 7 present the results for convergent and divergent validity. The PDSS total score showed a large correlation with the BAI and the PAS total scores (r = .68 and r = .80 respectively) and a medium correlation (r = .48) with the ASI–3 total score indicating overall excellent convergent validity. The correlation between the PDSS and the BDI-II scores was relatively large (r = .52) suggesting some need for improvement in terms of divergent validity. The PDSS showed acceptable convergent validity with the PDSS-SR (ICC = .64).
Convergent Validity of the PDSS (Global and Subscale Scores)
Note: *p <.05; **p <.01.
Large correlations ( ≥ .50) are shown in bold.
PDSS = Panic disorder Severity Scale; BAI = Beck Anxiety inventory; ASI–3 = Anxiety Sensitivity Index; PAS = Panic and Agoraphobia Scale self report; CI = Confidence intervals.
Divergent Validity of the PDSS (Global and Subscale Scores)
Note: *p < .05; **p < .01.
Large correlations (≥ .50) are shown in bold.
PDSS = Panic Disorder Severity Scale; BDI-II = Beck Depression Inventory-II; CI = Confidence Intervals.
Relationship with global clinical severity
The Pearson correlation coefficient between total PDSS and CGI severity was 0.81.
Discussion
The results of the present study indicate that the Spanish version of the PDSS has overall excellent psychometric properties, similar to what has been found with the original English version.
As regards test-retest results, although overall the PDSS performed quite well, item 7 (assessing social impairment) showed a relatively low value. Unfortunately, previous studies did not report on the test-retest reliability of individual items (Monkul et al., Reference Monkul, Tural, Onur, Fidaner, Alkin and Shear2004; Shear et al., Reference Shear, Rucci, Williams, Frank, Grochocinski, Vander Bilt and Wang2001). Therefore, this is a topic which deserves further investigation. The PDSS also performed well in terms of inter-rater reliability, and the results obtained here are similar to previous studies (Monkul et al., Reference Monkul, Tural, Onur, Fidaner, Alkin and Shear2004; Shear et al., Reference Shear, Brown, Barlow, Money, Sholomskas, Woods and Papp1997; Shear et al., Reference Shear, Rucci, Williams, Frank, Grochocinski, Vander Bilt and Wang2001; Yamamoto et al., Reference Yamamoto, Nakano, Watanabe, Noda, Furukawa, Kanai and Otsubo2004). Our study adds to the previous literature in showing that inter-rater reliability does not seem to be affected by different degrees of clinician expertise in anxiety disorders.
Our CFA suggested a two-factor modified model for the scale. Previous discrepancies regarding the structure of the scale have been attributed to sample selection, and the finding of a single-factor structure has been attributed to the presence of a full range of agoraphobic symptoms (Shear et al., Reference Shear, Rucci, Williams, Frank, Grochocinski, Vander Bilt and Wang2001). In our sample, despite a similar full range of agoraphobic symptoms, the data did not fit a one-factor solution, but a two-factor model. The first factor included the “typical” physical symptoms component of PD, and the second factor included behavioral and cognitive components. We allowed errors of Items 4 (agoraphobia) and 7 (social impairment) to correlate. We believe that this makes sense since there is a logic association between agoraphobic symptoms and the degree of impairment in social life. Nevertheless, as previous studies suggest (Lim et al., Reference Lim, Yu and Kim2007), further research is needed to understand the discrepant results on the factor structure of the scale.
The large correlation with the BAI and the PAS, and the medium correlation with the ASI–3 suggest that the Spanish PDSS has excellent convergent validity. Regarding divergent validity, a relatively large correlation was found with the BDI-II, although, this might be reflecting a true overlap between depressive and panic symptoms more than a validity concern. Previous research has shown that there is a high comorbidity between PD and depressive symptoms (Wu & Fang, Reference Wu and Fang2014). The PDSS also showed a relatively large correlation with its self-report counterpart, the PDSS-SR. This is similar to previous studies (Houck et al., Reference Houck, Spiegel, Shear and Rucci2002; Lee, Kim, & Yu, Reference Lee, Kim and Yu2009; Wuyek, Antony, & McCabe, Reference Wuyek, Antony and McCabe2011) and suggests that both versions can be used to assess panic symptoms. Finally, similar to previous studies (Furukawa et al., Reference Furukawa, Shear, Barlow, Gorman, Woods, Money and Leucht2009; Yamamoto et al., Reference Yamamoto, Nakano, Watanabe, Noda, Furukawa, Kanai and Otsubo2004), the large correlation between the PDSS and the CGI indicates that the PDSS correlates highly with overall clinical impression.
This study is not without limitations. First, our sample size was relatively small to conduct a CFA and other factorial models (e.g. hierarchical) could have been tested. Second, we did not look at some psychometric properties of the PDSS such as diagnostic or treatment sensitivity, or discriminant validity with other anxiety or mental disorders. Third, we only included PD patients, limiting the range of the PDSS scores. Future research may address ways of improving the divergent validity of the PDSS with depression/depressive symptoms. Finally, new instruments to asses PD based on dimensional models of psychopathology have been recently developed (Knappe et al., Reference Knappe, Klotsche, Heyde, Hiob, Siegert, Hoyer and Beesdo-Baum2013) and it will be important to compare them with the PDSS.
In conclusion, our data show that the Spanish PDSS has similar psychometric properties as the previous versions of the PDSS, and it can be used to assess panic symptoms in clinical and research settings in Spanish-speaking countries.
