Introduction
With increased rates of hospitalisation in acute psychiatric intensive care units among adults (Blader, Reference Blader2011) and a noticeable decline in the average length of hospitalisations (National Association of Pyschiatric Health Systems, 2002), the clinical effort to help patients engage with their treatment milieu has become a priority in acute psychiatric care. The therapeutic milieu is a key facet of recovery, and is typically comprised of daily activities encouraging skills training, socialisation, self-care, and interaction with clinical staff. Patients who disengage from the treatment milieu have less opportunity to benefit from skills training and support. Additionally, passivity and avoidance are linked with adverse health outcomes, including lowered quality of life (van de Wiel et al. Reference van de Wiel, Geerts and Hoekstra-Weebers2008) and immunity (Stowell et al. Reference Stowell, Kiecolt-Glaser and Glaser2001), and an increase of substance use (Ireland et al. Reference Ireland, McMahon, Malow and Kouzekanani1994), hypertension (Obrist, Reference Obrist1976) and depression featuring suicidal behaviours (DeGenova et al. Reference DeGenova, Patton, Jurich and MacDermid1994; Bruder-Mattson & Hovanitz, Reference Bruder-Mattson and Hovanitz1990). Though measures exist of patients’ avoidant behaviour in an outpatient setting (e.g. Cognitive Behavioral Avoidance Scale; Ottenbreit & Dobson, Reference Ottenbreit and Dobson2004), no standardised measures are available that measure patient participation in, and avoidance of, the therapeutic milieu.
A concise questionnaire that reflects a patient's involvement in the milieu of an acute psychiatric unit offers significant clinical utility. First, a questionnaire may socialise patients to observe clinical recommendations, including taking medications, participating in skills groups, and practicing self-care. Second, daily self-tracking reminds the patient to engage in their treatment goals, highlighting the lowest and highest level of engagement in each treatment activity, and providing instantaneous feedback regarding current treatment engagement. Third, a questionnaire allows staff to cross-check their observations with patient self-ratings of approach and avoidance, permitting the development of appropriate treatment plans and interventions. For example, psychiatrists can track barriers to treatment adherence, nurses can track dimensions of patient interest in treatment modalities, mental health workers can track group attendance, and social workers can track the patient's behaviour regarding planning for aftercare. Together, these potential benefits highlight the need for a novel measure of patient self-rated engagement in the milieu.
Through the use of on-unit observation and consultation with experts in nursing, psychiatry, and psychology, we developed a questionnaire that monitors the extent to which each inpatient was involved in milieu activities in the previous 24 hours of their stay. Relying on a behavioural model (Carver & White, Reference Carver and White1994) linking the Behavioral Activation System (BAS) with goal-directed behaviour and positivity when encountering activities that indicate potential rewards and, conversely, the Behavioral Inhibition System (BIS) with behavioural passivity and the experience of negative affect, items were selected that reflect common behaviours related to self-care, compliance with medical care, interaction with other patients and staff and group involvement. Moreover, the validity of this questionnaire would be supported by positive correlations between behavioural approach and positive affect, as well as between behavioural avoidance and negative affect, converging with previous research finding that high levels of experiential avoidance correlate with increased negative affect (Tanay et al. Reference Tanay, Lotan and Bernstein2012), and increased active coping correlates with positive affect (Schanowitz & Nicassio, Reference Schanowitz and Nicassio2006).
The main objective of this quality improvement project was to evaluate a brief self-report questionnaire of patients’ engagement with the activities offered as part of the milieu. We present psychometric data to support a new self-report questionnaire, the Checklist of Unit Behaviors (CUB), from a group of 132 inpatients hospitalised in an acute psychiatric unit in an urban medical centre who were enrolled in a quality improvement project to validate the CUB.
Method
Patients
Patients were on one of two identical acute psychiatric inpatient units in an urban medical centre. Patients were included if: (a) nurses deemed it safe for the patient to complete the self-reports, (b) the patient resided on the unit for more than 72 hours, and (c) the patient agreed to complete self-reported questionnaires. As part of the hospital's quality improvement protocol, data were not recorded on other demographic variables (e.g. patient race or ethnicity). This quality improvement project was reviewed by the university Institutional Review Board and was then referred to, and approved by, the hospital nurses board and quality and integrity board, which reviews quality improvement projects. All individuals signed a written consent form to permit the use of de-identified data for a quality improvement project, conducted between January 2011 and August 2011.
Assessments
Patients completed the CUB and additional self-reports at least 24 hours after admission. They also completed these questionnaires 24 hours before discharge from the unit. For the factor analysis, we analysed the first completed CUB per patient.
Measures
Behavioral Inhibition System/Behavioral Activation System Scales (BIS/BAS; Carver & White, Reference Carver and White1994). The BIS/BAS is a measure of motivation to approach and avoid and it was used to examine the construct validity of CUB scales. The BIS/BAS contains 20 items: 7 items relevant to the BIS (i.e. aversive motivation) and 13 of which reflect the BAS (i.e. appetitive motivation; Gray, Reference Gray1970, Reference Gray1981, Reference Gray1982). The BIS is measured using a single scale relevant to sensitivity to punishment, and this scale has demonstrated good internal consistency (7 items; α = 0.74; Carver & White, Reference Carver and White1994). The BAS consists of three scales with adequate internal consistencies (Carver & White, Reference Carver and White1994): Drive, measuring pursuit of desired goals (4 items; α = 0.76), Reward Responsiveness, measuring positive responses to the anticipation or occurrence of reward (5 items; α = 0.73), and Fun Seeking, measuring the respondent's willingness to approach a new activity impulsively (4 items; α = 0.66). Scales putatively measure personality qualities and were normalised using an undergraduate sample. Items are scored on a 4-point scale from ‘strong agreement’ to ‘strong disagreement’.
Positive Affect and Negative Affect Scale (PANAS; Watson et al. Reference Watson, Clark and Tellegen1988). Based on Gray's (1970) model (i.e. avoidance associated with negative affect; approach associated with positive affect), the PANAS is a 20-item measure of positive and negative dimensions of affect. Patients rate the extent to which they have experienced specific emotions portrayed by a list of adjectives in the past week. Ratings are made on a five-point scale ranging from ‘not at all’ to ‘extremely’. The Positive Affect scale (10 items) and the Negative Affect scale (10 items) have strong internal consistency (α = 0.88 and α = 0.85, respectively; Watson et al. Reference Watson, Clark and Tellegen1988).
Brief Symptom Inventory (BSI; Derogatis & Spencer, Reference Derogatis and Spencer1982). To characterise psychopathology in the sample, we used the BSI, a 53-item self-report questionnaire that assesses psychological symptom patterns of psychiatric and medical patients. The BSI measures nine specific dimensions of psychopathology and provides a Global Severity Index of general psychiatric distress. BSI scales have shown strong internal consistency for the nine dimensions reflecting symptom sets and the Global Severity Index (Derogatis, Reference Derogatis1993). Test-retest reliability was strong (α = 0.90).
Scale design and development for the Checklist of Unit Behaviors (CUB)
A list of 34 behaviours that reflect treatment engagement and passivity was generated through consultation with inpatient staff members and on-unit observation. Examples included patient's efforts to care for themselves, efforts to follow the structure of the program, participation in recommended treatment modalities and the social milieu, and preparation for discharge. Content validation was checked through discussions with inpatient staff and consultation with an expert (JG) who assisted in revising and condensing the 34-item list into a 22-item questionnaire. Of these items, 11 reflected treatment engaging behaviours (Approach) and 11 reflected treatment disengaged behaviours (Avoidance). Each item was phrased as a statement regarding the degree to which patients engaged in avoidant or proactive behaviours during the preceding 24 hours. Items were rated on a 5-point Likert scale with responses ranging from ‘not at all true’ to ‘very true’. Approach and avoidance items were grouped separately so that patients could visualize their progress over the course of treatment (e.g. higher scores in the first section represented high levels of engagement; lower scores in the second section represented low levels of avoidance). The risk of creating potential response bias due to the Approach-Avoidance split was outweighed by the potential benefit of allowing patients and members of their treatment team a visual metric of their improvement. This questionnaire was disseminated during morning group meetings when two of the authors (BH, DH) asked staff for feedback regarding item clarity. Item reviews and refinements were made based on feedback and consultation with an expert (JG).
Instructions were designed to orient the patient to their behaviours during the past day while receiving treatment on the unit. Instruction clarity was also assessed and modified based on spontaneous feedback from staff and patients. Based on staff observation, completion time was approximately 2–5 minutes. The CUB is shown in Figure 1.
Checklist of Unit Behaviors (CUB)
Analytic plan
Factor analysis was used to estimate the factor structure of the questionnaire. Internal reliability was assessed through the calculation of Cronbach's alpha statistics. Construct validity was estimated through convergent and discriminant bivariate correlations between CUB scale scores and measures of motivation to approach or avoid (BIS/BAS) and affect (PANAS). The effect of treatment on CUB scale scores was estimated through a paired samples t-test determining the magnitude of change for each scale between admission and discharge. Data were analysed with SPSS (Version 20.0.0).
Results
The patient group comprised 60 men and 72 women between the ages of 18 and 80 years (men: M(SD) = 40.9 years (13.1 years); women: M(SD) = 36.2 years; (13.1 years)). Mean length of hospitalisation was 7.2 days with a standard deviation of 6.7 days. Of the 132 patients who completed the surveys, 120 completed all items of the 22-item version of the CUB; these individuals were used for the factor analysis. Some patients provided incomplete responses to the CUB or other measures, therefore validity analyses are calculated from a reduced sample of completers. The sample size used for each calculation is reported. Also, because we used de-identified data, we are unable to provide more specific demographic information. There was no quantified pattern to missing data with items found to be missing completely at random (Little's MCAR: χ2 = 510.4, df = 464; p = 0.07).
Confirmatory factor analysis
Confirmatory factor analysis using principal axis factoring was performed to determine the content validity of the CUB. Technical details of this analysis can be found in the Appendix.
Internal consistency
Results indicated coefficient alphas of 0.93 for the Approach scale and 0.79 for the Avoidance scale. As required to be part of the final solutions, all items had a correlation of at least 0.30 with their factor score (r = 0.50–0.88).
CUB scales across psychiatric symptoms
We examined the two CUB scales across psychiatric symptoms measured by the Brief Symptom Inventory, including the Global Distress Index. Table 1 outlines the means and standard deviations of the two CUB scales for inpatients reporting significant levels of psychopathology across the nine dimensions of the BSI and the BSI's Global Severity Index.
T-scores and CUB scores for BSI scales at admission
Construct validity
Means and standard deviations of the CUB Approach and Avoidance scales and other validating measures are listed in Table 2. Because data were found to be missing at random, if one item was missing for a scale, the scale score was calculated by replacing the missing value with the mean of the other items from that scale (Tabachnick & Fidell, Reference Tabachnick and Fidell2007); however, if more than one item was missing for a scale, the scale was not calculated.
Means and standard deviations for CUB and validating measures
Note. The BIS/BAS scales are coded such that higher scores represent less motivation in a domain (Carver & White, Reference Carver and White1994).
CUB over the course of treatment
To examine change from admission to discharge, we conducted repeated-measures t-tests on the two CUB scales. Results indicated a significant increase in CUB-Approach, (t(76) = −8.62; p < 0.001). Also, results indicated a significant decrease in CUB-Avoidance (t(79) = 5.18; p < 0.001). Effect size estimates (Cohen's d) were large for CUB-Approach (d = 0.9) and moderate for CUB-Avoidance (d = 0.6).
CUB and PANAS
Construct validity was estimated by calculating bivariate correlations at admission and discharge between CUB scales and positive and negative affect (PANAS). CUB-Approach was significantly positively correlated with PANAS-Positive Affect at admission (r = 0.39; p < 0.01) and discharge (r = 0.35; p < 0.01). Correlations between CUB-Approach and PANAS-Negative Affect were non-significant.
CUB-Avoidance was positively correlated with the PANAS-Negative Affect at admission (r = 0.35; p < 0.01) and discharge (r = 0.27; p < 0.01), and CUB-Avoidance also was inversely correlated with PANAS-Positive Affect at admission (r = −0.22; p < 0.05) and discharge (r = −0.39; p < 0.01).
CUB and BIS/BAS
Construct validity was further estimated by calculating bivariate correlations at admission and discharge between CUB scales and sensitivity to reward and sensitivity to punishment (BIS/BAS). At admission, CUB-Approach scale showed a significant and positive correlation with BAS-Reward Responsiveness (r = 0.42; p < 0.01) and BAS-Fun Seeking (r = 0.27; p < 0.01). At discharge, an inverse correlation was observed between CUB-Approach and BAS-Fun Seeking (r = −0.24; p < 0.05).
At admission, the CUB-Avoidance scale was inversely correlated with BAS-Reward Responsiveness (r = −0.23; p < 0.05). At discharge, a positive correlation between CUB-Avoidance and BAS-Drive was observed (r = 0.20; p < 0.05).
Discussion
The Checklist of Unit Behaviors (CUB) is a 16-item questionnaire that quantifies a patient's involvement with milieu activities while hospitalised in an acute psychiatric inpatient unit for general psychiatric disorders. Designed for daily use and taking 2–5 minutes to complete, the scale includes 16 items that are phrased as statements regarding the extent to which patients engaged in avoidant or proactive behaviours in the preceding 24 hours.
Factor analysis suggested that the CUB consists of two scales, specifically, CUB-Approach and CUB-Avoidance, which contain nine and seven items, respectively. Our review of the scales indicates that each demonstrates adequate internal reliability, consistent with the clearly defined behavioural activation and behavioural inhibition systems proposed by Gray (Reference Gray1970, Reference Gray1981, Reference Gray1982) and expanded by a motivational affect model (Carver & White, Reference Carver and White1994). Further, over the course of hospitalisation, significant increases in CUB-Approach and significant decreases in CUB-Avoidance were observed, indicating that the CUB assesses behaviours influenced by treatment in an inpatient care facility. Moreover, CUB-Approach and CUB-Avoidance show evidence of adequate construct validity as estimated with a measure of positive and negative affect, as well as with a measure of appetitive and aversive motivation at admission, suggesting that patient scores on the CUB are associated with theoretically-related constructs.
Many of the significant correlations observed with respect to the CUB-Approach scale align with previous findings. For example, the positive association between CUB-Approach and positive affect was consistent with Schanowitz & Nicassio's (2006) finding that positive affect was associated with active coping. Further evidence for the construct validity of CUB-Approach was indicated by its positive relationship with BAS-Reward Responsiveness, a scale measuring sensitivity to reward (Carver & White, Reference Carver and White1994), at admission.
Our results also indicate expected correlations with the CUB-Avoidance scale and affect. CUB-Avoidance was positively correlated with negative affect, which is similar to results in which experiential avoidance had a significant, positive correlation with negative affect intensity (Kingston et al. Reference Kingston, Clarke and Remington2010). Additionally, CUB-Avoidance and positive affect were inversely correlated, as was reported by Kortte et al. (Reference Kortte, Veiel, Batten and Wegener2009) whose results showed that positive affect was inversely correlated with a measure of experiential avoidance. Taken together, it appears that the behaviours captured in CUB-Approach and CUB-Avoidance represent broader behavioural constructs that are strongly associated with affect and with approach-oriented motivation.
The association between motivation to approach and avoid (BIS/BAS) and actual approach and avoidance behaviours (CUB) at discharge was unexpected. At admission, behaviour is weakly associated with sensitivity to cues of reward or punishment (Reward Responsiveness); however, over the course of treatment, patient engagement in the milieu appeared to become independent of motivation. It is possible that patients displaying the greatest inhibition and least drive may have received the most intensive treatment from unit staff or suffered more severe psychopathology. Also, the highly structured inpatient environment may have altered patient behaviour regardless of motivation. Most importantly, our results suggest that treatment engagement may be changed independently of patient-reported motivation to approach or avoid.
Future research should cross-validate the CUB with clinician-based assessments, and observational measures. Based on the method of review, only patients with the ability to read at a seventh-grade level could fill out the CUB reliably. Finally, test items were divided into unlabeled Approach and Avoidance sections. Creating these groupings could potentially encourage a response set; however, the CUB's strong psychometric properties do not suggest the presence of widespread bias. Also, this divided placement was intentional because the CUB was designed to function as both assessment measure and treatment tool. Future research should also replicate these findings in other psychiatric intensive care units in the United States and internationally.
Conclusions
Developed for, and validated with, inpatients with high levels of psychiatric acuity, the CUB conveys strong psychometric properties and provides clinically useful information about each patient's propensity to engage in, or to avoid, milieu-based activities. On-unit observation and consultation with experts in nursing, psychiatry, and psychology permitted us to formalise these common behaviours into a clinical evaluative tool. The CUB can be incorporated into a variety of inpatient contexts, independent of perspective and clinical modalities, orienting patients and staff towards increased treatment engagement in the milieu.
Acknowledgements
Jackie K. Gollan, Ph.D. has received grant/research support from National Institute of Mental Health, Northwestern Memorial Hospital Women's Board, Davee Foundation, Evergreen Invitational Award, National Alliance for Research on Schizophrenia and Depression (NARSAD), American Foundation of Suicide Prevention and the Brain Research Foundation, and other financial or material support from the American Psychological Association.
Item characteristics were filtered prior to factor analysis, such that items with greater than 75% endorsement of a single response, greater than 5% missing value rate, or less than a 0.75 standard deviation were eliminated. These exclusion criteria have been used previously for the assessment of 5-point Likert scales (Ottenbreit & Dobson, Reference Ottenbreit and Dobson2004). Using these criteria, six items were removed (five items had >75% endorsement of a single response; one item had >5% missing value rate). Based on the analysis of eigenvalues, the two factor solution appeared to fit the data well (Eigenvalue 1 = 7.095; Eigenvalue 2 = 2.079). An analysis of squared multiple correlations and eigenvalues did not suggest the presence of multicollinearity (i.e. no eigenvalues approaching zero; no squared multiple correlations approaching 1.0). Factorability of the correlation matrices appeared likely because the Kaiser-Meyer-Olkin (KMO) Measure of Sampling Adequacy value was greater than 0.60 (KMO = 0.87).
Principal axis factoring with promax rotation (kappa = 4) rotation was used to determine the factor structure of the remaining items. The correlation of −0.54 between factors suggests related, but separate factors; therefore, an oblique rotation was used. After three iterations the analysis confirmed 2 factors out of the 16 remaining items, with these 2 factors accounting for 57.3% of the total variance (Factor 1 = 44.3%; Factor 2 = 13.0%). The 2 factors that emerged were labeled Approach (9 items) and Avoidance (7 items). All items loaded onto hypothesised factors (loading range: 0.28–0.92). One item had similar absolute loading values across both factors (i.e. difference in absolute loadings values less than 0.10); however, these similar absolute correlations are likely accounted for by the moderate inverse correlation between factors. Bartlett's Test of Sphericity was conducted to ensure significance of all correlations within the correlation matrix; this analysis indicated a satisfactory degree of correlation (χ 2 = 1211.0; df = 120; p < 0.001). Table A1 summarises the relationships between items and factors.
Mean (SD), factor loadings and eigenvalues for CUB items (n = 120)
