Introduction
More than 16% of the Portuguese population are now aged over 65 years, with the absolute number of elderly people having doubled in the past 40 years. The corresponding increase in dementia has been associated with an increase in levels of disability, physical illness and social problems. The recently approved Portuguese National Plan of Mental Health (CNRSSM, 2007) emphasizes the need for the decentralization and integration of patients, their families and community services. It also recommends that accurate assessment of patients' social functioning should encompass full needs assessments. However, little is known about the needs of older people with mental health problems in Portugal and this may preclude effective implementation of community-guided policies. In a number of other European countries the Camberwell Assessment of Need for the Elderly (CANE) has been established as an effective needs assessment tool (Orrell and Hancock, Reference Orrell and Hancock2004). Given the crucial importance of needs assessment in elderly people with mental disorders, the aim of this work is to ascertain the reliability and validity of the Portuguese version of CANE, and to determine whether it will be a useful tool for research and clinical practice.
Methods
Description of the instrument
CANE (Reynolds et al., Reference Reynolds2000) is based on the structural model of the Camberwell Assessment of Need (CAN) and had similar criteria in its development (Phelan et al., Reference Phelan, Slade and Thornicroft1995). After a long development process using a modified Delphi process (Pill et al., Reference Pill1971; Rowe et al., Reference Rowe, Wright and Bolger1991) involving a consensus conference and a wider network which included relevant professionals and voluntary groups involved in care (Reynolds et al., Reference Reynolds2000), as well as pilot studies, a final draft (CANE version II) emerged. Versions III and IV have appeared as subsequent refinements, translated into a variety of languages and increasingly used worldwide (Orrell and Hancock, Reference Orrell and Hancock2004).
CANE provides for an evaluation of older people with a wide range of psychiatric diagnoses. It covers 24 areas of need (plus two areas of carer needs) and includes ratings for staff, patient/user and carer's views of needs. (Note: the term “user” refers to patients in contact with mental health services.) Each item is scored as follows: 0 = no need, 1 = met need, and 2 = unmet need. CANE also assesses the informal/formal support that the user receives, whether the rater feels that the user is receiving the right type of help with the problem, and finally, whether the user is satisfied with the received assistance.
Translation process
The Portuguese version, stemming from CANE version IV, was developed in collaboration with the Spanish working group, taking advantage of the common sociocultural background of the regions of Galicia (north-western Spain) and northern Portugal. Following a training workshop (conducted by Orrell and Hancock) held in Santiago de Compostela in 2003, formal procedures were developed in order to ensure validity of the Portuguese version, which are described in detail elsewhere (Gonçalves-Pereira et al., Reference Gonçalves-Pereira2007). The translation was done by the Portuguese CANE workgroup (including Portuguese psychiatrists with expertise in old age mental health and fluency in English). After several group discussions concerning accuracy of this translation, a larger multidisciplinary group from mental health teams of different regions in Portugal (psychiatrists, psychologists, mental health nurses and social workers), provided opinion on face and content validity. A back-translation was performed by a professional English translator with knowledge of the mental health field and discussed with the authors of the instrument. This process resulted in only minor modifications.
Pilot studies
The resulting Portuguese version developed in 2004 was then used in pilot studies in Lisbon and Oporto, prior to the validation study which began in December 2005. Throughout the translation process, some doubts had been raised concerning feasibility of the CANE interviewing procedures in routine clinical settings (e.g. regarding length of interviews and collaboration of interviewees for the whole assessments). Although these may be an issue in some cases, acceptability of the Portuguese CANE was evident from the beginning of fieldwork. Preliminary pilot studies along with multidisciplinary group feedback suggested good ecological, face and content validity. Supporting data are described in greater detail elsewhere (Fernandes et al., Reference Fernandes2007; Gonçalves-Pereira et al., Reference Gonçalves-Pereira2007).
Data collection
A cross-sectional multicenter study was designed, which included teams from hospitals in Oporto (Hospital S. João (HSJ)/Hospital Magalhães Lemos (HML), both public hospitals) and Lisbon (Clínica Psiquiátrica S. José (CPSJ), a semi-public mental health facility collaborating in this research with Universidade Nova de Lisboa). A convenience sample was selected. Data were collected in two phases. First, participants from a variety of settings (urban and rural, outpatient departments, day-hospitals, acute psychiatric wards, day care and nursing homes) attending HSJ or CPSJ facilities were interviewed by a psychiatrist (LF or MGP) and one of six co-workers (all of them psychologists) for the reliability study. An additional sample from HML was added for the validation study. Users, staff and carers were interviewed separately, with interviewer and observer both rating the responses. Seven of the patients could not be interviewed because they had severe dementia. The staff came from a variety of professional backgrounds: psychiatry, psychology, social work and nursing. Duration of all interviews was recorded and for patients the mean time was 25.9 (SD 12.2) minutes, for carers it was 26.2 (SD 20.0) minutes, and for staff 10.5 (SD 7.6) minutes.
Analysis
The analysis was based on the following structure: reliability assessment (inter-rater and test-retest analysis) and validity assessment (face and content validity as well as criterion, construct and ecological validity).
Face and content validity
Face and content validity were assessed by adopting the usual procedures for transcultural validation of assessment tools in psychiatric epidemiology. As described above, translation and back-translation stemmed from the CANE version IV (Orrell and Hancock, Reference Orrell and Hancock2004); this was enhanced by multidisciplinary group discussions, and the refined draft was shown to be acceptable in preliminary pilot surveys (Gonçalves-Pereira et al., Reference Gonçalves-Pereira2007).
Criterion and construct validity
The measures used for comparison in the validation process were the EASYCare (Philp et al., Reference Philp, Armstrong, Coyle, Chadwick and Machado1999), which is a measure of quality of life and dependency; the Barthel Index (Mahoney and Barthel, Reference Mahoney and Barthel1965), a measure of functional ability; the 12-item General Health Questionnaire (GHQ-12, Likert scoring; Goldberg, Reference Goldberg1978) to rate patient and carer stress; the Geriatric Depression Scale (GDS-15; Yesavage et al., Reference Yesavage, Brunk, Rose, Huang, Adey and Leirer1983), a measure of depression for patients and carers; and the Mini-mental State Examination (MMSE; Folstein et al., Reference Folstein, Folstein and McHugh1975) for cognitive skills assessment.
Reliability data
Data were collected on 50 cases for the inter-rater and 46 cases for test-retest reliability analysis. These data were collected by two of the authors (LF and MGP) and two of the co-workers in a variety of settings. Initially, 44 elderly patients, 29 carers and 46 staff members were interviewed in the presence of one of two co-raters (both scoring simultaneously); one week later, 42 patients, 28 carers and 46 staff were re-interviewed.
Statistical analysis
Patient and carer characteristics are presented as raw frequencies and percentages for categorical variables and as mean and standard deviation (SD) for continuous variables, when normal distributions could be assumed, and as median and percentiles 25 and 75, when normality could not be assumed. Needs assessments using CANE were analyzed using number and percentage for each category (no need, met need, unmet need and missing data) and for each item, and calculating the total scores corresponding to the total sum of met needs, total sum of unmet needs and total sum of met or unmet needs. Box plots presenting median, inter-quartile range, minimum and maximum for CANE total scores and assessed by patient carer and staff are presented. Means and 95% confidence intervals were calculated for each total score.
Inter-rater and test-retest reliability were assessed by calculating the percentage of complete agreement and Cohen's κ coefficient for each item (considering the dichotomization of need present/absent), for patient, carer and staff, for the first section of CANE. Reliability for total scores was assessed by calculating the intraclass correlation coefficient (ICC) and 95% confidence intervals (Streiner and Norman, Reference Streiner and Norman1995).
Criterion validity was assessed by calculating Spearman's ρ rank correlation coefficients between CANE total scores and total scores for EASYCare and Barthel Index, given the absence of an adequate gold standard measure. Construct validity was assessed by calculating inter-item and item-total correlations for each item of CANE; by evaluating several previously hypothesized correlations among items of CANE and items of each one of the other measures analyzed; and by evaluating correlations between CANE total scores and total scores for GHQ-12, MMSE and GDS-15 (convergent and divergent validity) (Streiner and Norman, Reference Streiner and Norman1995). Correlations were assessed by calculating Spearman's ρ rank correlation coefficients. Significance tests (testing the null hypothesis of ρ = 0) are presented for every correlation coefficient. The significance level of α = 5% was considered for all hypothesis tests.
The statistical analyses were performed using the software Statistical Package for the Social Sciences Version 15.0 for Windows (SPSS).
Results
The data were collected from a convenience sample of 79 patients who had a mean age of 73.8 (SD 6.6) years (range 62–95 years). The majority were female (76%), 66% lived in urban areas and most had a low socioeconomic status (53%) by the Graffar classification (Graffar, Reference Graffar1956). Most were married (44%) or widowed (39%), and nearly half were living with partner (43%). Overall, 76% lived at home. Only 32% had no carer. Dementia was the most common psychiatric diagnosis (61%). Most cases had somatic comorbidity (85%) and nearly three-quarters were attending outpatient facilities (71%). Table 1 shows the characteristics of the study sample.
Demographic and clinical characteristics of subjects (n = 79)
More than two-thirds (n = 54) of patients had a carer, and 98.2% (n = 53) of these carers were interviewed. They were mainly sons/daughters (46.3%) and spouses (37%). The demographic characteristics of carers showed a predominance of female (78%) and married (80%) persons. Mean age was 57.6 (SD 14.4) years. In this sample, 38 (48.1%) users lived with at least one carer in the same household.
Needs assessment
The number of needs rated by staff for each item of CANE in the total of sample is shown in Table 2. The most common met needs were food, household skills, physical health, memory, money and self-care. The commonest unmet needs were psychological distress, daytime activities, company, memory and carer's psychological distress. No one in the sample was regarded as having a serious unmet need for accommodation, abuse/neglect and behavior or alcohol abuse. Staff were able to rate needs for most items, but approximately 15% did not know whether there were any needs for information. According to the carers (n = 54) only five patients were considered to have a serious unmet need for information (9.4%) and only 14 for psychological distress (26.4%).
Levels of need as rated by staff
Inter-rater and test-retest reliability
ICC = intraclass correlation coefficient; CI 95% = 95% confidence interval.
Box plots presenting median, inter-quartile range, minimum and maximum for CANE total scores and assessed by patient carer and staff are presented in Figure 1. From the 79 cases, the mean total numbers of needs identified were 7.38 (95% CI 6.50–8.25) by patients, 9.02 (95% CI 8.05–9.99) by carers, and 8.46 (95% CI 7.71–9.20) rated by staff. As expected, these needs were expressed differently by patients, carers and staff. While carers and staff rated approximately the same number of expressed needs, patients rated about 20% fewer needs than carers.
Box plots of total scores of needs (median, interquartile range, minimum and maximum)
* Total number of met needs, expressed by patient, carer and staff
** Total number of unmet needs, expressed by patient, carer and staff
*** Total number of met and unmet needs, expressed by patient, carer and staff
Reliability
Inter-rater and test-retest reliability were assessed by ICC between total scores and κ coefficients for each item, considering patient, carer and staff ratings. The κ values for inter-rater reliability ranged from 0.72 to 1.00 (mean values: 0.96 patient, 0.93 carer and 0.90 staff). These results are therefore generally excellent with all but one reaching 0.75 or above.
Test-retest reliability (over one week) was assessed in a subsample of 46 respondents. Overall, κ statistics for test-retest reliability are lower than for inter-rater reliability, but in general they appear adequate, with a mean of 0.80 (range 0.56–1.00) for patient-rated needs, 0.77 (range 0.33–1.00) for carer-rated needs, and 0.81 (range 0.57–1.00) for staff-rated needs.
According to Fleiss criteria (Fleiss, Reference Fleiss and Fleiss1981), staff agreement on retesting was excellent (κ > 0.75) in about 73% of items, while it was good (0.60–0.74) in 19% of items. In the patient domain, on retesting, agreement was excellent (69% of items), good (19% of items) or fair, i.e. 0.40–0.59 (7% of items). In the carer domain, κ statistics showed excellent agreement on retesting in 65% of items, good agreement in 12%, fair agreement in 4% and poor agreement (κ < 0.40) in another 4%.
Intraclass correlations between total scores (met and unmet needs), based on analysis of components of variance, showed a correlation coefficient of 0.98 (95% CI 0.96–1.00), 0.98 (95% CI 0.95–0.99) and 0.95 (0.91–0.97), for inter-rater reliability, for patient, carer and staff ratings, respectively. Calculations for test-retest reliability yielded correlations of 0.82 (95% CI 0.68–0.90), 0.92 (95% CI 0.83–0.96) and 0.92 (95% CI 0.86–1.00), for patient, carer and staff ratings, respectively.
Time taken for each interview was recorded for all reliability interviews, with a mean duration of 17.5 (SD 8.6) minutes for patients, 18.8 (SD 18.5) minutes for carer and 8.3 (SD 8.0) minutes for staff.
Criterion and construct validity
Criterion validity was explored considering CANE and other measures. As there were no other scales specific to the needs of elderly people with mental illness, two scales were used for comparison with CANE: EASYCare to rate the quality of life and dependency, and the Barthel Index as a measure of a functional status.
Additionally, correlations between CANE total scores and total scores for GHQ-12, MMSE and GDS-15 were calculated in order to assess construct validity. Positive correlations between CANE total scores and the other three measures (GHQ-12 for measuring carer and patient stress, GDS-15 to measure depression in patient, and MMSE for cognitive skills) were hypothesized. (The correlation coefficients are shown in Table S1, available as supplementary material online, attached to the electronic version of this paper at www.journals.cambridge.org/jid_IPG.) There were good correlations between CANE total score (sum of met and unmet needs) and GDS total score (rs 0.62; p ≤ 0.01), GHQ (rs 0.58; p ≤ 0.01), and EASYCare (rs 0.46; p ≤ 0.01); and good negative correlations with Barthel Index (rs−0.44; p ≤ 0.01). There were also good direct correlations between the staff-rated items of CANE and the corresponding EASYCare scores in almost all items of this instrument between: household skills/housework (rs 0.557; p < 0.01); food/meals (rs 0.653; p < 0.01); self-care/bath use, self-care, dressing (rs 0.608, rs 0.508, rs 0.518; p < 0.01); mobility/indoor mobility, stairs/mobility, bed to chair (rs 0.591, rs 0.649, rs 0.552; p < 0.01); continence/bladder continence (rs 0.662; p < 0.01). When CANE scores were compared to the Barthel Index, negative correlations were found as expected. Particularly good correlations were found between the following items: self care/bathing, grooming (rs −0.663, rs −0.524; p < 0.01); mobility/bed to chair, level surfaces, stairs (rs −0.570, −0.625, −0.591; p < 0.01); continence/bladder (rs −0.519; p < 0.01). There was a significant negative correlation between the CANE item “Memory” and the MMSE (rs −0.37; p < 0.01). There were also significant correlations between CANE staff-rated psychological distress and GDS-15 total score (patient) (rs 0.60) and between CANE staff-rated psychological distress and GHQ-12 total score for patient (rs 0.72), and also between staff-rated “carer's psychological distress” and GHQ-12 total score for carer (rs 0.51). These results suggest reasonable criterion and construct validity (see Table S1).
Both convergent and divergent construct validity were assessed by creating a correlation matrix for the first section of all 26 items as rated by patients, carer and staff. We subsequently examined whether there were positive or negative correlations between items, where one would intuitively expect to find such correlations (“convergent”), or whether it was obvious that no relationship should exist (“divergent”). Evidence of convergent construct validity is particularly shown in correlation measures between memory and those functions that one would expect to be impaired – in the patients' domain the correlations between memory and households skills were 0.334 (p < 0.01), for memory and self care 0.405 (p < 0.01), for memory and money 0.465 (p < 0.01), for self care and household skills 0.554 (p < 0.01), and for company and daytime activities 0.381 (p < 0.01).
Similarly, there are moderate correlations for these items in the staff domain: between memory and households skills 0.548 (p < 0.01), for memory and self care 0.389 (p < 0.01), for memory and safety 0.287 (p < 0.01), for memory and money 0.450 (p < 0.01) and for self care and household skills 0.455 (p < 0.01).
However, in the carer domain only between memory and money (rs = 0.327; p < 0.01) and self care and household skills (rs = 0.341; p < 0.01) did good correlations exist according to Cohen's criteria (Cohen, Reference Cohen1988). Divergent validity is shown by the lack of any significant correlations between items such as carer's and patient's need for information and all other items (see Table S2 published as supplementary material online with the electronic version of this paper at www.journals.cambridge.org/jid_IPG).
Overall, the correlation coefficients indicate good construct validity (Cohen, Reference Cohen1988), which is reinforced by a more detailed analysis between total score and each item of CANE. In this analysis good correlations were also found consistently for patient, carer and staff domains, especially for the following items: household skills, food, self-care, daytime activities, memory, mobility, continence, safety, company and money.
Discussion
This study clearly indicates that CANE has excellent validity and reliability in a Portuguese population. Although further studies are warranted, the present survey (in conjunction with the first pilot studies) yields very encouraging results, with evidence regarding inter rater and test-retest reliability. There were robust positive data on ecological, face and content validity. Criterion and construct validity seemed adequate and correlations were predicted by initially hypothesized relationships.
An important feature of this instrument is the multiple views of needs by patient, carer and professional. But this advantage may cause methodological problems when the overall assessment of needs is required. This could be more important as CANE comes to be used more widely, especially if some raters have little training (Dech and Machleidt, Reference Dech, Machleidt, Orrell and Hancock2004). In this context it is important to focus on the possible limitations of the instrument, because its application may present some difficulties in clinical settings (Ybarzábal et al., Reference Ybarzábal, Mateos, García, Amboage and Fraguela2002).
The experience gained from this study suggests that using CANE accurately requires some experience and training to provide a true picture of the needs as perceived by the patient, the carer and the professional responsible for care of the patient. This requirement will be reduced with the Portuguese version of the manual (available in the future), with objective situations and clinical vignettes adapted to local cultural and social requirements and based on the experiences reported by the local clinical interviewers. This should lead to greater accuracy as noted in the Spanish and German versions (Ybarzábal et al., Reference Ybarzábal, Mateos, García, Amboage and Fraguela2002; Dech and Machleidt, Reference Dech, Machleidt, Orrell and Hancock2004; Orrell and Hancock, Reference Orrell and Hancock2004).
Furthermore, some difficulties were found for certain groups of patients, including those with significant cognitive decline, or for inpatients with psychiatric comorbidity. For these groups lower concurrent validity was observed. However, CANE allows for the view of the elderly and their relatives, compensating these limitations to some extent (Mateos et al., Reference Mateos, Ybarzábal, Garcia, Amboage, Fraguela, Orrell and Hancock2004).
Regarding validity, good correlations were found between CANE and other scales, particularly with the EASYcare and Barthel Index. Providing carers with the opportunity to give their opinions on the patients' needs was also universally appreciated, particularly by carers of people with dementia for whom the process often proved cathartic. The good correlation between the CANE item “carer's psychological distress” and the GHQ-12 should further enhance the instrument's usefulness (Reynolds et al., Reference Reynolds2000).
Inter-rater reliability as measured by ICC between total scores was very good. This means that raters were rating almost exactly the same numbers of needs. In fact, the κ statistics for the individual items were also generally very high. This finding possibly reflects natural bias in the method of data collection. Raters were not blinded as would be required in ideal conditions (Reynolds et al., Reference Reynolds2000). On the other hand, the test-retest reliability achieved lower κ values when compared with inter-rater reliability. However, test-retest reliability as measured by ICC of total scores is also generally excellent, as it was in the original study (Reynolds et al., Reference Reynolds2000).
Results with CANE assessments seem in accordance with the situation in hospital settings (inpatients/outpatients), nursing homes and day care centers in Portugal, where the proportion of dependent patients has been increasing and care provision has been primarily focused on physical health, while psychosocial needs are sometimes overlooked. This possibly explains the high scores for unmet needs in this sample in the areas of psychosocial domains.
However, the surprising absence of unmet needs regarding abuse/neglect, behavioral problems and alcohol abuse suggests the need for further evidence given the prevalence of these problems in the elderly.
In Portugal, standardized assessment and evaluation of health needs in psychiatry is still a relatively new concept, in contrast to Britain. But this approach becomes more important with the development of new services structured for elderly people and the ongoing reform of the Mental Health Services in Portugal (CNRSSM, 2007). There is a pressing need for coordination of different community services, primary and secondary care and case management, with an emphasis in multilevel integration, as proposed in the recent Portuguese Integrated Continuing Care Network and related geriatric rehabilitation perspectives.
In conclusion, focusing on comprehensive yet feasible assessments of need is a major issue. The Portuguese version of CANE provides an individual, comprehensive and holistic assessment of physical, social, psychological and environmental needs. This assessment must lead to appropriate medical and psychosocial interventions that will improve the quality of mental health services for the elderly. The psychometric properties seem globally encouraging, with robust positive data on ecological, face, content, criterion and construct validity as well as for reliability. The Portuguese version of CANE is therefore a promising tool for research and practical use in old age mental health services.
Conflict of interest
None.
Description of authors' roles
LF led the main phase of the research described in this paper and MGP the initial development of the Portuguese version of CANE. Design and conduct of the study in the field, including data collection and database entry, were accomplished by LF (in Oporto) and MGP (in Lisbon). AL coordinated the Hospital Magalhães Lemos team. Statistical analysis was carried out by LA with the collaboration of CD. SM assisted in the preparation of the manuscript and, with MS, was involved in data collection and database entry. RM and MO provided advice to the project. The initial draft of the manuscript was written by LF, MGP and LA. All authors reviewed and provided valuable contributions to the whole manuscript.
Acknowledgments
We thank all participants (patients, staff and carers). We also thank Horácio Firmino, João Barreto and Duarte Falcão (members of the Portuguese CANE workgroup) for their valuable contribution during the preliminary phase of the validation process, and Richard Rogers for the back-translating procedure. Finally, thanks go to all the other members of the Portuguese CANE Working Group: Rosa Gonçalves, Catarina Janeiro, Jaime Grácio, Mafalda Gomes, Mónica Pereira, Pedro Nunes, Teresa Salgado, Adelson Estrela, Ma José Salgadinho, Luísa Teles, Justina Silva (group discussions and data collection), Marta Coromina, Clara Pinto and Clara Jaria (group discussions).
