Hostname: page-component-74656f9884-j7cmm Total loading time: 0 Render date: 2026-11-07T04:14:31.869Z Has data issue: false hasContentIssue false
  • English
  • Français

Can electronic routine data act as a surrogate for patient-assessed outcome measures?

Published online by Cambridge University Press:  02 March 2005

Hayley A. Hutchings ,
Wai-Yee Cheung ,
John G. Williams ,
David Cohen ,
Mirella F. Longo  and
Ian Russell
Hayley A. Hutchings
Affiliation:
University of Wales Swansea
Wai-Yee Cheung
Affiliation:
University of Wales Swansea
John G. Williams
Affiliation:
University of Wales Swansea
David Cohen
Affiliation:
University of Glamorgan
Mirella F. Longo
Affiliation:
University of Glamorgan
Ian Russell
Affiliation:
University of Wales Bangor and Velindre NHS Trust
Rights & Permissions [Opens in a new window]

Abstract

Objectives: There has been a rapid growth in the use of patient-assessed outcomes (PAOs) that are measured in the assessment of health technologies. The process of collection of such measures can be costly, and there may be problems associated with the ability of the patient to complete them. The use of electronically stored routine data may reduce costs and overcome the problems associated with patient completion. The feasibility of using routine data surrogates for the UK Inflammatory Bowel Disease Questionnaire (UKIBDQ) and the Short Form 36 (SF-36) was examined.

Methods: Clinical terms and codes for the UKIBDQ and SF-36 questions were identified, and data from electronic routine sources were sought on patients participating in a randomized controlled trial. The presence or absence of relevant symptoms was used to generate surrogate scores, which were compared with the original scores.

Results: Most questions in the UKIBDQ and SF-36 were codable but only one third of the terms were recorded routinely in electronic form. The surrogate total IBDQ score had reasonable reliability (Kuder–Richardson coefficient = 0.51), but this reliability could not be determined for the SF-36. Intraclass correlations between routine and designed data were poor to weak.

Conclusions: Although electronic routine data sources had the capacity to develop surrogate measures for patient assessed outcomes, there was evidence of wide underutilization of coding systems leading to an underreporting of symptoms. This finding is consistent with previous literature where only poor correlations were illustrated between patient assessed outcomes and surrogate scoring of symptoms.

Keywords

Patient assessed outcomeInflammatory bowel diseaseRoutine dataFeasibility

Information

Type
RESEARCH REPORTS
Information
International Journal of Technology Assessment in Health Care , Volume 21 , Issue 1 , January 2005 , pp. 138 - 143
Copyright
© 2005 Cambridge University Press

There has been a rapid growth in the number of patient-assessed outcomes (PAOs) (15), and the research community is increasingly recognizing the importance of assessing PAOs in addition to clinical outcomes in health-care evaluation. However, as it forms part of the data collected for primary research, evidence of changes in PAOs is costly to obtain. Moreover, the benefit of these data tends to remain confined to the individual studies that generated them, that is, they are rarely used in routine clinical practice (16).

The amount of electronic stored data of patients' information has escalated over the past 20 years (8). Administrative databases are a relatively inexpensive source and patient outcomes can be tracked longitudinally for years, because these systems allow linking by means of a medical record, social security number, or other identification number for each episode of care (29). Thus, allied with increasing concerns about the quality and costs of health care, researchers have started investigating if patient outcomes resulting from health-care activities could also be systematically retrieved by querying these data (3;4;12;17;18;25). The information requested of routine data is shifting from data related to processes of care and patient outcomes such as mortality and morbidity to data related to more complex measures of health status (e.g., physical, mental, social, emotional, and role functioning) (3;4).

In England and Wales, much administrative, demographic, and clinical data are collected about patients in electronic form in the course of their health care. In the future, the quality and quantity of these data will increase (11;28) with a need to capture patient outcomes routinely along with it (10). Currently, most routine clinical systems in the United Kingdom have the provision to record diagnostic and procedure codes with some more elaborate systems, allowing the recording of patient symptoms and signs. There would thus appear to be considerable potential to examine the capacity of routine data systems to develop surrogate PAO measures and, hence, reduce the need for designed primary research and the costs associated with it.

PAO questionnaires aim to ascertain the well-being of the patient based on patient completed responses to either symptom-specific or generic health questions. A single generated score or several subscales can then give an indication of the “general health” (for generic questionnaires) or the “disease-specific” health (for disease-specific questionnaires) of a patient. This presupposes that the patient is able to complete the questionnaire, but may for various reasons be unable to do so (1). It has been suggested that, in some cases, a health-care professional or caregiver may be used as a proxy to provide useful information on PAOs, especially on observable aspects of PAOs (1). Furthermore, the patient's perception of health and its meaning will vary between individuals and within an individual over time (6).

Given these difficulties, as well as the increasing requirement to capture patient outcome and experience routinely (10), there is a need to explore whether routinely collected clinical data can be used as a surrogate for subjective patient-focused data collection. This potential, however, is dependent on the extent to which routine data are capable of generating valid condition-specific and/or generic PAOs. The assessment of PAOs using routine data should also give careful consideration to the validity of the data (3;4).

To assess the potential of the available electronic routine data in measuring PAOs, we have examined the possibility of developing routine data surrogates for one disease-specific [United Kingdom Inflammatory Bowel Disease Questionnaire, UKIBDQ (7)] and one generic [Short Form 36, SF-36 (5;27)] PAO measure.

METHODS

A previously completed pragmatic randomized controlled trial on patients with inflammatory bowel disease seen in two UK sites (Neath and Morriston) (32) was reanalyzed for this study using electronically held routinely collected data. This approach was in the context of a wider study on the use of routine data for health technology assessment (33). Design and analysis were undertaken by a statistician who was not involved with the original study but proceeded according to the principles set out in the protocol for the original trial.

Electronically held routine data were found in local and national systems (33). One of the study sites uses a generic clinical information system, GeneCIS® (31), on which coded clinical data are collected in detail to support individual patient management. Electronically held routine data were also sought from primary care. Each data source was queried to extract all the data for each of the study patients from the date of recruitment to the study end date.

Appropriate clinical terms and codes for the questions in the PAO instruments used in the original study [UKIBDQ (7) and SF-36 (27)] were identified by a clinical coding specialist using International Classification of Diseases version 10 (ICD-10) (30), Office of Population Censuses and Surveys version 4 (OPCS4) (23), and Clinical Terms version 3 (CTV3) (22) coding systems (see Tables 1 and 2 for examples). These codes representing relevant symptoms, signs, diagnoses, and procedures in the UKIBDQ and SF-36 were then sought in the data obtained from routine sources.

Surrogate subscale and total UKIBDQ scores and subscale SF-36 scores were generated by adding up the presence of relevant symptoms. The calculated baseline surrogate scores were reversed and transformed (20) to make them comparable with the original UKIBDQ and SF-36 scores obtained (32). They were then compared with the scores obtained from the original study using intraclass correlation (9). Kuder–Richardson coefficent (KR-20) or Cronbach's α values (26) were calculated for the surrogate UKIBDQ and SF-36 scores to determine the reliability of the new scales.

RESULTS

Patients studied at the site without the clinical information system GeneCIS® had limited information available, thus making any comparisons meaningless. The results are based on ninety-three patients from the site where data from GeneCIS® and departmental data systems were available.

Table 1 illustrates examples of UKIBDQ comparable coded data from electronic routine data sources. Of the original thirty inflammatory bowel disease questions, twenty-nine had a comparable CTV3 code and twenty had an ICD-10 code. However, the surrogate symptoms were only recorded routinely for ten of them. A standardized item KR-20 of 0.51 was calculated for the surrogate total IBDQ score. KR-20 scores could not be calculated for the UKIBDQ subscales due to the limited number of recorded symptoms in the subscales. The surrogate scores were consistently higher than the original UKIBDQ subscales and total scores. Intraclass correlations between the original UKIBDQ and surrogate scores were poor to weak (see Table 3).

Table 2 illustrates examples of the SF-36 comparable coded data from electronic routine data sources. Of the original thirty-six SF-36 questions, thirty-four had a comparable CTV3 code and thirty had an ICD-10 code. The surrogate symptoms, however, only were recorded routinely for ten of them. Cronbach's α or Kuder–Richardson scores could not be calculated for any of the SF-36 surrogate subscales due to the nonrecording of many of the SF-36 surrogate symptoms. The surrogate SF-36 subscale scores were generally higher than the original SF-36 subscales scores, with the exception of vitality and mental health. Intraclass correlation between the original SF-36 and surrogate scores were poor to weak (see Table 3).

DISCUSSION

This study was a retrospective exploratory study. It explored a unique approach to the measurement of PAOs. This strategy has been possible because of the availability of detailed clinical data that were routinely collected and stored in GeneCIS®. This information system uses a clinically rich terminology for coding (Clinical Terms version 3 (22)), which includes many terms that correspond to the questions in the UKIBDQ and SF-36. More “standard” hospital administration systems record diagnoses only, using ICD-10 (30). Our experience also suggests that data collected on departmental systems under clinical ownership are more reliable than those collected through processes under managerial control (33). This evidence is consistent with the literature (13;14;24). Although many of the questions in the UKIBDQ and SF-36 questionnaire have a corresponding term and code in ICD-10, these are clinically unwieldy and rarely recorded in coded form. Routine systems that do not use a clinically rich terminology, therefore, may miss information that could be useful in measuring PAOs.

Questions that had a corresponding code were generally scored as present if the patient exhibited the code or absent if the code was not present. One flaw with this approach is that it cannot be assumed that a patient did not have a particular symptom just because it was not recorded in the system. Also the presence of a recorded symptom gives no indication of the level of severity of that symptom (29), and the symptoms recorded in records cannot generally substitute for the response to a refined and validated questionnaire (12).

Only ten of the thirty UKIBDQ and ten of the thirty-six SF-36 available codes were ever used in the electronic routine data sources in this study. This finding reflects routine clinical practice and the ad hoc collection of data in the conventional outpatient setting. Routine data, therefore, will consistently underrecord symptoms that might contribute to the assessment of PAOs, even when captured in a clinically rich information system. This finding is reflected in the weak to poor intraclass correlations between the surrogate and original UKIBDQ scores. Our findings are in agreement with published literature that highlights the discrepancy between patient and providers perception of general health status (3;4). Although correlations between the surrogate and original scores were only poor to weak, the surrogate total UKIBDQ scale was shown to be fairly reliable with a Kuder–Richardson value of 0.51. This compares reasonably well with the UK validation study (7). Although symptoms were only recorded as present or absent, it appears that the clinical information system GeneCIS® could be useful in measuring changes in PAOs in disease-specific conditions such as inflammatory bowel disease.

In contrast, although three of the surrogate SF-36 subscales showed a weak correlation with the original SF-36 items, the Cronbach's α or Kuder–Richardson values could not be calculated, as symptoms were absent for the large majority of questions. Because the SF-36 questionnaire is generic, this finding may account for the limited number of questions with symptoms recorded as present. Retrospectively routine data systems appear to have less value in assessing changes in generic PAOs after interventions than condition-specific changes.

Due to the retrospective nature of this study, some codes may have been missed due to extraction difficulties or problems with the systems at the time of the study. There may also be differences in coding procedures and codes used at different study sites that may affect the result of PAO testing with routine data (29). In a prospective trial, it would be possible to identify and record routinely the presence or absence of all symptoms and signs that are normally recorded in disease-specific or generic questionnaires. Context-specific, structured proformas for data collection have been shown to improve performance (19). This approach could eliminate the need for patient-administered questionnaires and the associated difficulties such as patient comprehension and poor response rates (1).

This study used data from a generic clinical information (GeneCIS®), which can be customized for use in different clinical contexts. Symptoms and signs can be routinely recorded in coded form, as well as diagnoses and procedures. It also has the facility for local set up of structured questionnaires, which facilitate the capture of multi-item scores (PAOs, patient preferences, disease severity, disability, activities of daily living, and so on). Such systems are not presently widely available but will be needed to meet the specification of requirements published by the Academy of Medical Royal Colleges (2). If the specification is met, patient-focused outcomes could be measured by proxy across a broad range of clinical conditions and in the course of routine care.

If routinely collected data are used for prospective randomized trials, the data will be recorded by those not blind to the treatment given, who may also have a vested interest in the success of the experimental regimen. Thus, a randomized trial using routine data might fail to comply with the CONSORT guidelines (21). The validity of such adaptations in measuring PAOs needs to be explored further. In addition, for greatest accuracy, outcome data should be captured as close to the source as possible and are best reported by the patients' themselves. Systems, therefore, should allow the patient to report these outcomes directly (35). Validated measures (such as the UKIBDQ or SF-36) could be routinely completed by patients when attending for consultation (18) or admission and held on a clinical system such as GeneCIS®. This strategy may address some of the issues concerning the standardization of use of PAO measures (15) and the process of linking clinical variables with health-related quality of care (34) but may not overcome the difficulties associated with patient completion (1). The feasibility and cost of this process should be explored.

CONCLUSIONS

This exploratory study has shown that, where clinical data are collected in detail in structured form, it may be possible to develop surrogate measures for PAOs. Where only diagnoses and procedures are collected, this development does not appear to be possible. This inability is due, not to the lack of available codes, but due to the underutilization of the terms and codes within the routine data systems. Both the surrogate UKIBDQ and SF-36 scores were consistently higher than the original scores, indicating an underreporting of symptoms. To develop good surrogate measures for PAOs, the consistent, accurate, and complete collection of routine data is required.

CONTACT INFORMATION

Hayley A. Hutchings, PhD, Lecturer (), Wai-Yee Cheung, PhD, Senior Lecturer (), Centre for Health Improvement Research and Evaluation, Swansea Clinical School, University of Wales Swansea, Singleton Park, Swansea SA2 8PP, Wales

John G. Williams, Professor of Health Services Research (), Clinical School, University of Wales Swansea, Singleton Park, Swansea SA2 8PP, Wales

David Cohen, BCom, MPhil, Professor of Health Economics (), Mirella F. Longo, PhD, Research Fellow (), School of Care Sciences, University of Glamorgan, Treforest, Pontypridd CF37 1DL, Wales

Ian Russell, PhD, Director (), Institute of Medical and Social Care Research, University of Wales Bangor; Professor of Public Health, National Public Health Services for Wales, Velindre NHS Trust, Wheldon Building, Bangor Gwynedd LL57 2UW, Wales

We are very grateful to the colleagues who allowed us to re-analyse their original study using routine data and to Jayne Morgan, Hayley Dickinson, Tracey Hughes, Teryl Stoneman, and Julia Demery for their coding, technical, and administrative support.

The study was funded by the National Health Service Health Technology Assessment National Research and Development Programme. The views and opinions expressed do not necessarily reflect those of the National Coordinating Centre for Health Technology Assessment.

References

Addington-Hall J, Kalra L. 2001 Who should measure quality of life? BMJ. 322: 1417– 1420.Google Scholar
Academy of Colleges Information Group. 2001. Specification of core requirements for clinical information systems in support of secondary care. Available at: www.aomrc.org.uk; Accessed:
Bakken Henry S, Lush M, Costantino M, et al. 1997 Health status measurement in computer-based patient record systems. Nurs Admin Q. 21: 50– 60.Google Scholar
Bakken Henry S, Morris J, Holzemer W. 1997 Using structured text and templates to capture health status outcomes in the electronic health record. J Qual Improv. 23: 667– 677.Google Scholar
Brazier J, Harper R, Jones N, et al. 1992 Validating the SF-36 health survey questionnaire: A new outcome measure for primary care. BMJ. 305: 160– 164.Google Scholar
Carr A, Gibson B, Robinson P. 2001 Is quality of life determined by expectations or experience? BMJ. 322: 1240– 1243.Google Scholar
Cheung W-Y, Garratt A, Russell I, Williams J. 2000 The UK IBDQ- A British version of the inflammatory bowel disease questionnaire: Development and validation. J Clin Epidemiol. 53: 297– 306.Google Scholar
Cole L, Houston S. 1997 Integrating information technology with an outcomes management program. Crit Care Nurs Q. 19: 71– 79.Google Scholar
Deyo R, Diehr P, Patrick D. 1991 Reproducibility and responsiveness of health status measures. Statistics and strategies for evaluation. Control Clin Trials. 12: 142S– 158S.Google Scholar
Department Of Health. The NHS Plan—a plan for investment, a plan for reform. London: Stationary Office; July 2000.
Department Of Health. 2001. Building the information core: Implementing the NHS plan. London: Department of Health;
Dolin R. 1997 Outcome analysis: Considerations for an electronic health record. M.D. Computing. 14: 50– 56.Google Scholar
Galland R, Magee T, Berridge D, et al. 1998 Accuracy of centrally recorded OPCS codes for vascular surgery in the United Kingdom. Eur J Vasc Endovasc Surg. 16: 415– 418.Google Scholar
Galland R, Whatling P, Crook T, Magee T. 2000 Regional variation in varicose vein operations in England 1989-1996. Ann R Coll Surg Engl. 82: 275– 279.Google Scholar
Garratt A, Schmidt L, Mackintosh A, Fitzpatrick. 2002 Quality of life measurements: Bibliographic study of patient assessed health outcome measures. BMJ. 234: 1417– 1419.Google Scholar
Higginson I, Carr A. 2001 Using quality of life measures in the clinical setting. BMJ. 322: 1297– 1300.Google Scholar
Johnson R, Hornbrook M, Nichols G. 1994 Replicating the chronic disease score (CDS) from automated pharmacy data. J Clin Epidemiol. 47: 1191– 1199.Google Scholar
Johnston R. 2000 Implementation of a computer-based patient record and an outcomes data-collection system at the Department of Orthopaedic Surgery, University of Iowa. J Bone Joint Surg Am. 82: 1502– 1506.Google Scholar
Mann R, Williams J. 2003 Standards in medical record keeping. Clin Med. 3: 329– 332.Google Scholar
Medical Outcomes Trust. 1994. SF-36 Health Survey Scoring Manual for English Language Adaptations. Australia/New Zealand, Canada, United Kingdom. Boston, MA: Medical Outcomes Trust;
Moher D, Schulz K, Altman D. 2001 The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 357: 1191– 1194.Google Scholar
National Health Service Information Authority. Clinical terminology and classification services. Available at: http://www.coding.nhsia.nhs.uk.
Office of Population Censuses and Surveys. 1990. Tabular list of the classification of surgical operations and procedures. 4th revision. London: HMSO;
Pollock AM, Vickers N. 2000 Reducing DCO registration through electronic matching of cancer registry data and routine hospital data. Br J Cancer. 82: 712– 717.Google Scholar
Rietman L, Hendricks H. 2000 Speaking the same language: Improving evaluation of patient-focused outcomes using standardized terminology. Top Health Inform Manage. 20: 35– 43.Google Scholar
Streiner G, Norman R. 1995. Health measurement scales. A practical guide to their development and use. Oxford: Oxford University Press;
Ware J, Sherbourne C. 1992 The MOS 36-item short form health survey (SF-36). I. Conceptual framework and item selection. Med Care. 30: 473– 483.Google Scholar
Welsh Assembly Government. 2002. Informing healthcare-towards a knowledge-based NHS. Cardiff: Welsh Assembly Government;
Whitman G. 2002 Outcomes research. Getting started, defining outcomes, a framework, and data sources. Crit Care Nurs Clin N Am. 14: 261– 268.Google Scholar
World Health Organization. 1992. International statistical classification of diseases and related health problems. 10th revision. Geneva: World Health Organization;
Williams J, Morgan J, Howlett P, Severs M. 1993 Let there be light. Br J Healthcare Comput. 10: 30– 32.Google Scholar
Williams J, Cheung W, Russell I, et al. 2000 Open access follow up for inflammatory bowel disease: Pragmatic randomised trial and cost effectiveness study. BMJ. 320: 544– 548.Google Scholar
Williams J, Cheung W, Cohen D, et al. 2003 Can randomized trials rely on existing electronic data? A feasibility study to explore the value of routine data in health technology assessment. Health Technol Assess. 7 (26).Google Scholar
Wilson I, Cleary P. 1995 Linking clinical variables with health-related quality of life. A conceptual model of patient outcomes. JAMA. 273: 59– 65.Google Scholar
Zielstorff R. 1995 Capturing and using clinical outcome data: Implications for information systems design. J Am Med Informatics Assoc. 2: 191– 196.Google Scholar
Figure 0

Examples of Some Original UKIBDQ Questions with Comparable Coded Symptoms, Signs, and Diagnoses from All Electronic Routine Data Sources

Figure 1

Examples of Some Original SF-36 Questions with Comparable Coded Symptoms, Signs and Diagnoses from All Electronic Routine Data Sources

Figure 2

Intraclass Correlation between the Original and Surrogate Scores for the UKIBDQ and SF-36