No CrossRef data available.
Article contents
Reliability and validity of the Japanese version of the Support Team Assessment Schedule (STAS-J)
Published online by Cambridge University Press: 19 July 2005
Abstract
Objective: The aim of this project was to develop an appropriate and valid instrument for assessment by medical professionals in Japanese palliative care settings.
Methods: We developed a Japanese version of the Support Team Assessment Schedule (STAS-J), using a back translation method, and tested its reliability and validity. In the reliability study, 16 nurses and a physician who work in a palliative care unit evaluated 10 hypothetical cases twice at 3-month intervals. For the validity study, external researchers interviewed 50 patients with matignancy and their families and compared the results with ratings by the nurses in the palliative care unit.
Results: Our results with hypothetical cases were: interrater reliability weighted κ = 0.53–0.77 and intrarater reliability weighted κ = 0.64–0.85. In the validity study comparing nurse evaluations and the results of interviews with patients and families, complete agreement was 36–70%, and close agreement (±1) was 74–100%. As a whole, weighted κ were low: between −0.07 and 0.51. Our results were similar to those in the United Kingdom and Canada.
Significance of results: Although this research was conducted under methodologically limited conditions, we concluded that the STAS-J is a reliable tool and its validity is acceptable. The STAS-J should become a valuable tool, not only for daily clinical use, but also for research.
Information
- Type
- Research Article
- Information
- Copyright
- © 2004 Cambridge University Press
INTRODUCTION
The first hospice in Japan was founded in Seirei Mikatahara Hospital in 1981. Several hospice programs have been developed nationwide since then. Palliative care services provided in hospices and palliative care units were included in the Japanese health insurance system in 1990 (Sakonji et al., 1997). The number of palliative care units has increased in the past decade; 120 units and 2287 beds were available nationwide as of August 2003. As the number of palliative care units increases, issues of quality assurance have arisen (Kawa et al., 1999; Tamura, 2001; Uchinuno, 2001). One approach to quality assurance was a survey of satisfaction of bereaved families that was conducted in 1999 by the Japanese Association of Hospice and Palliative Care Units (Morita et al., 2002a, 2002b).
The need for quality assurance in hospices and palliative care units has been recognized overseas (Higginson & McCarthy, 1989; Higginson, 1994; Ingleton & Faulkner, 1995; Bruera, 1996). As a tool for clinical audits, the Support Team Assessment Schedule (STAS) was developed in the United Kingdom (McCarthy & Higginson, 1991; Higginson, 1993; Higginson & McCarthy, 1993), and the Edmonton Symptom Assessment System (ESAS) was developed by Bruera in Canada (Bruera et al., 1989, 1991). Both have had practical applications (Hearn & Higginson, 1997).
The purpose of hospice and palliative care is to provide comprehensive end-of-life care to patients and their families (World Health Organization, 1990). This includes not only the alleviation of physical pain, but also palliation of psychological, social, and spiritual distress. Therefore, we need instruments that include these four areas in order to assess the quality of hospice and palliative care. In addition, because target patients and families are in crisis, instruments that use assessments by medical professionals would be suitable in some situations. However, no such instrument existed in Japan, although some self-rating symptom assessment scales and quality of life scales have been developed (Kobayashi et al., 1998; Okuyama et al., 2003).
The STAS was developed by Higginson in the United Kingdom. It has nine core items: pain control, symptom control, patient anxiety, family anxiety, patient insight, family insight, communication between patient and family, communication between professionals, and communication from professionals to patient and family. Each item receives a score of 0 to 4 based on the item definition. The STAS was designed to be used by a multidisciplinary team that would determine a group score for each patient (Higginson, 1993). The reliability and validity of the STAS has been reported in the United Kingdom and Canada (Higginson & McCarthy, 1993, 1994; Higginson et al., 1998; Carson et al., 2000), and the STAS had been used worldwide (Edmonds et al., 1998; Lo et al., 1999).
The purposes of this study were (1) to develop a Japanese version of the STAS (STAS-J) and (2) to test its reliability and validity.
METHODS
First, we developed the STAS-J. Then we tested its reliability and validity based on Carson's study (Carson et al., 2000).
Translation
Translation was performed after obtaining permission from Professor Higginson, who developed the STAS. We organized the Japan STAS Working Group (Chief: K. Matoba) to develop a Japanese version. The group consisted of seven health care professionals: one physician and two nurses who had worked in the palliative care and had studied in the United Kingdom, three registered nurses, and a visiting nurse. We adopted a back-translation method. After repeated meetings and revisions we finalized the STAS-J.
Study Subjects
The study subjects were staff (a physician and nurses) and inpatients in a palliative care unit of the National Cancer Center Hospital East and the patients' families. Eligible staff members were full-time employees and had experience working in palliative care units for more than 6 months. The eligibility criterion for patients was diagnosis of a malignancy. Written consent was obtained from both the patients and their families. Patients were excluded if they could not tolerate the study or had severe mental disturbance or cognitive impairment. This study was approved by the institutional review board of the National Cancer Center.
Procedure
Reliability
The reliability study was conducted between January and April of 2000, using hypothetical cases instead of actual patients. Ten hypothetical cases were created by physicians working in a palliative care unit of a different hospital. These cases were based on actual patients. The cases varied in order to highlight different concerns and issues. They were the same length but were a mixture of a variety of issues. Two researchers in palliative care ensured that there were no inconsistencies or simple mistakes in the scenarios.
Because the original STAS had not been used in this unit, we held a training seminar for the participants. We created an original text for this seminar. We also answered questions from the participants. After the seminar, participants simultaneously evaluated the 10 hypothetical cases using the STAS-J. All forms were collected after the evaluation. These data were used to assess interrater reliability. For the intrarater reliability (reproducibility) test, all participants evaluated the same 10 hypothetical cases 3 months later.
Validity
The validity study was conducted from January 2000 to September 2001. Our gold standard was the assessment results of interviews conducted with patients and their families. First, we recruited 50 consecutive eligible patients and families in the unit and obtained written consent. A nurse in charge determined the score on the STAS according to a daily conference. We employed two outside researchers to collect the data from patients and their families. We did not use patients' direct assessments because the STAS-J was not designed to be completed by patients. A researcher visited the unit and collected patient information from the previous week from medical and nursing charts. Then she conducted semistructured interviews with patients and their families separately. The researcher asked questions of patients and families based on items of the STAS-J and made assessments. We did not ask patients the following items: family anxiety, family insight, and communication between professionals. The validity study was conducted after the reliability study so the nurses could become more skillful in the use of the STAS-J.
Analysis
Reliability
For interrater reliability, weighted κ was calculated for all possible combinations of raters for the time 1 data and summarized as mean values. Weighted κ was calculated to obtain intrarater reliability for each item using scores at time 1 and time 2.
Validity
Scores obtained from patients or their families were compared to those from nurses by calculating proportions of identical scores (complete agreement), ±1 (close agreement) and beyond ±1. We also calculated weighted κ, 95% confidence intervals, and Spearman's rank correlation coefficients. In addition, mean and standard deviation (SD) of each item were obtained and verified with Wilcoxon's signed rank test to compare between groups (patients vs. nurses and families vs. nurses).
The statistical package SAS, Version 6.12, was used for all statistical analyses.
RESULTS
Characteristics of Subjects
Reliability
Subjects for the reliability study were 16 nurses and one physician who evaluated hypothetical cases at two points in time. The mean duration of provider work experience was 11.2 (SD 8.0) years and length of time the providers worked in a palliative care unit was 3.7 (SD 3.1) years.
Validity (Table 1)
Fifty patients and their families were enrolled in the study. Twenty-three (46%) subjects were male and mean age was 59.1 (SD 11.3) years. Diagnoses were cancer of the lung, 14 (28%); colon or rectum, 7 (14%); and breast, 7 (14%). Family members who participated were spouse, 27 (54%); children, 13 (26%); and parents, 5 (10%).
Patient characteristics (validity study)
Reliability (Table 2)
For interrater reliability, the weighted κ of each item ranged from 0.53 to 0.77. The weighted κ was higher for the items family insight (κ = 0.77), pain control (κ = 0.75), and family anxiety (κ = 0.72), and lower for symptom control (κ = 0.53) and patient anxiety (κ = 0.59). Spearman's correlation coefficient was between 0.66 and 0.91. Six of nine items had correlation coefficients higher than 0.8.
Result of reliability study
Validity (Tables 3 and 4)
In assessing the validity of the nurses' evaluation compared to patients' scores, complete agreement was 38–70% and close agreement (±1) was 74–100%. The item communication between patient and family had a lower agreement (complete agreement was 38% and close agreement was 74%) and other items were above 88%. The weighted κ varied between 0.08 and 0.51. The only item with a weighted κ of 0.4 or higher was pain control (κ = 0.51), whereas communication between patient and family (κ = 0.08) and communication from professional to patient and family (κ = 0.18) were particularly low. Spearman's correlation coefficients (ρ) ranged between 0.09 and 0.61. The items communication between patient and family (ρ = 0.09) and communication from professional to patient and family (ρ = 0.22) had the lowest coefficients.
Result of validity study (Nurse versus patient ratings)
Result of validity study (Nurse versus family ratings)
The validity assessment of nurses' compared to families' scores revealed that complete agreement was 36–68% and close agreement (±1) was 80–100%: Communication between patient and family was somewhat low (complete agreement was 36%) and close agreement of other items was 80% and higher. The weighted κ were from −0.07 to 0.43: pain control (κ = 0.43) and symptom control (κ = 0.43) had indices of 0.4 and higher and communication from professional to patient and family (κ = −0.07), patient anxiety (κ = 0.06), and communication between patient and family (κ = 0.07) had lower indices. The full range of Spearman's correlation coefficients was from −0.1 to 0.6; communication from professional to patient and family (ρ = −0.1), patient anxiety (ρ = 0.1), and communication between patient and family (ρ = 0.14) were lowest.
DISCUSSION
Japan has lacked a validated instrument for medical professional rating in palliative care settings. We, therefore, developed a STAS-J in order to have a tool, not only for daily clinical use, but also for outcome measurement and explanatory variables for research. Preparation of the validated and reliable instrument is very important for quality assurance and scientific development. STAS-J would contribute to the development of scientific research in this field in Japan.
Landis and Koch (1977) proposed that a κ of 0.21 to 0.40 indicates fair agreement, 0.41 to 0.60 moderate agreement, 0.61 to 0.80 substantial agreement, and 0.81 to 1.00 almost perfect agreement. According to these criteria, all items in the intrarater reliability and most items representing interrater reliability in our study had indices of substantial or higher, which demonstrates a high evaluation score for reliability.
For evaluation of the instrument we used hypothetical cases created from actual patients instead of assessing real patients in the unit. Results of these evaluations probably differ from actual clinical circumstances. However, the limitations of this process are offset by the relative ease of evaluating hypothetical patients. The condition of real terminally ill patients may change daily and the staff may not have access to the same information on each patient. This makes assessment of intrarater reliability almost impossible for real patients.
The use of hypothetical cases as a simulation allowed us to investigate characteristics of the STAS itself. The quality of the hypothetical cases may have had an impact on the results. However, various items in this study showed high reliability. Scores for intrarater reliability tended to be higher than those for interrater reliability. This suggests that reliability can be higher when the same rater evaluates patients. Reliabilities in the items, including symptom control and patient anxiety, had the lowest scores. The former might have resulted from the different symptoms in the participants that were selected for evaluation. The latter is consistent with studies of quality of life assessment by proxies.
Complete agreement in the validity study was 30–70%. However, both complete and close agreement were more than 80% for most items. We believe this is a high figure and error of ±1 is acceptable. On the other hand, weighted κ were not generally high. It has been reported that the weighted κ are much lower when the marginal distribution is imbalanced (Feinstein & Cicchetti, 1990). We excluded patients with serious physical or mental conditions from the study and enrolled patients and their families only if written consent was obtained. Therefore, our subjects may have had relatively few communication problems with families and medical staff. This may have caused lower weighted κ. In our study, 88% of patients and 96% of families had a rating of 0 for the item communication from professionals to patient and family. There were few items ratings that were evenly distributed between 0 and 4. This may have had a strong influence on the weighted κ.
External researchers interviewed subjects in the validity study, based on Carson's study (Carson et al., 2000). Although interviewers checked medical and nursing notes prior to the interview to obtain information for more valid assessments, there is a possibility that this information was not sufficient. Particularly in the STAS, some items, such as anxiety and family relationship, need to be carefully explored. These situations may have resulted in less satisfactory agreement.
Agreement between nurses and patients or families in our study tended to be similar to those in studies conducted in the United Kingdom (Higginson & McCarthy, 1993) and Canada (Carson et al., 2000). We believe that these similar results proved that the procedure of translating the STAS into Japanese did not impair its quality. We assume that in actual clinical settings the same raters will evaluate patients' conditions in a timely manner, which would lead to more accurate assessments. Besides, the more experience practitioners have in the use of the STAS-J, the better the validity and reliability that would be expected.
We held training seminars in the use of the STAS-J for staff before the study. Sufficient preparation will be necessary when the STAS-J is introduced into clinical settings (Higginson, 1993; Lo et al., 1999). As a next step, we need to incorporate the STAS-J into daily practice and explore its actual efficiency in the palliative care unit.
CONCLUSION
We developed a Japanese version of the STAS (STAS-J) and tested its reliability and validity. Our results, using hypothetical cases, were: interrater reliability weighted κ = 0.53–0.77, and intrarater reliability weighted κ = 0.64–0.85. In the validity study comparing evaluation by nurses and interview results on patients and families, complete agreement was 36–70% and close agreement (±1) was 74–100%. As a whole, weighted κ were low, between −0.07 and 0.51. Our results were similar to those in studies in the United Kingdom and Canada. Although this research was conducted under methodologically limited conditions, we concluded that the STAS-J is a reliable tool and its validity is acceptable.
ACKNOWLEDGMENTS
We would like to thank all the patients, families, and staff of the National Cancer Center Hospital East who participated in the study. We also thank Dr. S. Tsuneto, Osaka University, and K. Tamura, Yodogawa Christian Hospital, for translation. We are grateful to Yuka Imamura, Keiko Kazuma, and member of the Department of Adult Nursing/Terminal and Long-Term Care Nursing, The University of Tokyo. This work was supported by Health and Labor Sciences Research Grants, Research on Health Technology Assessment (H12-iryo-014) from the Ministry of Health Labor and Welfare, Japan. STAS-J and the Scoring Manual can be downloaded from: http://plaza.umin.ac.jp/~stas/
References
REFERENCES
Bruera, E.
(1996).
Quality assurance in palliative care—A growing
“must?” [Letter].
Support Care Cancer,
4,
157.Google Scholar
Bruera, E.,
Kuehn, N.,
Miller, M.,
et al.
(1991).
The Edmonton Symptom Assessment System (ESAS): A simple method for
the assessment of palliative care patients.
Journal of Palliative Care,
7,
6–9.Google Scholar
Bruera, E.,
Macmillan, K.,
Hanson, J.,
et al.
(1989).
The Edmonton staging system for cancer pain: Preliminary
report.
Pain,
37,
203–209.Google Scholar
Carson, M.G.,
Fitch, M.I., &
Vachon, M.L.
(2000).
Measuring patient outcomes in palliative care: A reliability and
validity study of the support team assessment schedule.
Palliative Medicine,
14,
25–36.Google Scholar
Edmonds, P.M.,
Stuttaford, J.M.,
Penny, J.,
et al.
(1998).
Do hospital palliative care teams improve symptom control? Use of a
modified STAS as an evaluation tool.
Palliative Medicine,
12,
345–351.Google Scholar
Feinstein, A.R. &
Cicchetti, D.V.
(1990).
High agreement but low kappa: I. The problems of two
paradoxes.
Journal of Clinical Epidemiology,
43,
543–549.Google Scholar
Hearn, J. &
Higginson, I.
(1997).
Outcome measures in palliative care for advanced cancer patients: A
review.
Journal of Public Health Medicine,
19,
193–199.Google Scholar
Higginson, I.
(1993).
Clinical Audit in Palliative Care.
New York:
Radcliffe Medical Press.
Higginson, I.
(1994).
Clinical audit and organizational audit in palliative care.
Cancer Surveys,
21,
233–245.Google Scholar
Higginson, I. &
McCarthy, M.
(1989).
Evaluation of palliative care.
Palliative Medicine,
3,
267–274.Google Scholar
Higginson, I. &
McCarthy, M.
(1993).
Validity of the support team assessment schedule: Do staffs'
rating reflect those made by patients or their families?
Palliative Medicine,
7,
219–228.Google Scholar
Higginson, I. &
McCarthy, M.A.
(1994).
A comparison of two measures of quality of life: Their sensitivity
and validity for patients with advanced cancer.
Palliative Medicine,
8,
282–290.Google Scholar
Higginson, I.,
Wade, A., &
McCarthy, M.
(1998).
A comparison of four outcome measures of terminal care. In
A safer death,
A. Gilmore, S. Gilmore (eds.), pp.
205–211.
New York:
Plenum Press.
Ingleton, C. &
Faulkner, A.
(1995).
Quality assurance in palliative care—A review of the
literature.
Journal of Cancer Care,
4,
49–55.Google Scholar
Kawa, M.,
Miyashita, M., &
Imamura, Y.
(1999).
A preliminary research on introducing a clinical audit in hospice
and palliative care units in Japan. [Hosupisu kea ni kansuru kenkyuu
houkoku syo.] pp.
44–49.
Tokyo,
Japan Sasagawa Health Science Foundation
(in Japanese).
Kobayashi, K.,
Takeda, F.,
Teramukai, S.,
et al.
(1998).
A cross-validation of the European Organization of Research and
Treatment for Cancer (EORTC QLQ-C30) for Japanese with lung cancer.
European Journal of Cancer,
34,
810–815.Google Scholar
Landis, J.R. &
Koch, G.G.
(1977).
The measurement of observer agreement for categorical data.
Biometrics,
33,
159–174.Google Scholar
Lo, R.S.,
Ding, A.,
Chung, T.K.,
et al.
(1999).
Prospective study of symptom control in 133 cases of palliative
care.
Palliative Medicine,
13,
335–340.Google Scholar
McCarthy, M. &
Higginson, I.
(1991).
Clinical audit by a palliative care team.
Palliative Medicine,
5,
215–221.Google Scholar
Morita, T.,
Chihara, S., &
Kashiwagi, T.
(2002a).
Quality Audit Committee of the Japanese Association of Hospice and
Palliative Care Units. A scale to measure satisfaction of bereaved family
receiving inpatient palliative care.
Palliative Medicine,
16,
141–150.Google Scholar
Morita, T.,
Chihara, S., &
Kashiwagi, T.
(2002b).
Quality Audit Committee of the Japanese Association of Hospice and
Palliative Care Units. Family satisfaction with inpatient palliative care
in Japan.
Palliative Medicine,
16,
185–193.Google Scholar
Okuyama, T.,
Wang, X.T.,
Akechi, T.,
et al.
(2003).
Japanese version of the M.D. Anderson Symptom Inventory: A
validation study.
Journal of Pain and Symptom Management,
26,
1093–1103.Google Scholar
Sakonji, M.,
Shimizu, C.,
Shingo, Y.,
et al.
(1997).
Hospice care in Japan. In
Hospice Care on the International Scene,
C.D. Saunders, R. Kastenbaum (eds.),
pp.
219–231.
New York:
Springer.
Tamura, K.
(2001).
Current status and problem about quality of palliative care in
Japan.
The Japanese Journal of Nursing and Hospice Care,
12,
43–52
(in Japanese).Google Scholar
Uchinuno, A.
(2001).
Concept and implementation of quality improvement to terminal
care.
The Japanese Journal of Terminal Care,
11,
341–347
(in Japanese).Google Scholar
World Health Organization.
(1990).
Cancer pain relief and palliative care.
Technical Report Series 804.
Geneva:
World Health Organization.
Patient characteristics (validity study)
Result of reliability study
Result of validity study (Nurse versus patient ratings)
Result of validity study (Nurse versus family ratings)
You have
Access
- 85
- Cited by