Health technology assessment (HTA) programs typically work by influencing practice on a broad scale, either through reimbursement decisions or national guidelines. However, there are also many important healthcare decisions made by hospitals and integrated healthcare networks. Evidence-based practice techniques can and should be applied to these decisions.
Over the past decade, hospital-based HTA centers have emerged to fill this role (Reference Battista, Côté, Hodge and Husereau1;Reference Cicchetti, Marchetti, Di Bidino and Corio3;Reference Weill and Banta7). They provide evidence necessary to inform clinical decisions at the level of hospital administration. Most often, they adapt general guidelines or evidence reports to their local setting, or create new guidelines or reports. However, unlike payer-based HTA organizations, hospital-based HTA organizations can also integrate local data into their guidelines and reports. Local data not only can fill gaps in the published evidence, but it can also improve the generalizability of that evidence to the local setting.
In this study, we describe two evidence reports from our hospital-based HTA center (details of center history and staffing are presented in Supplementary Table 1, which can be viewed online at www.journals.cambridge.org/thc2010021) which required the integration of local data. Both cases illustrate how local evidence can be used at the institutional level to support the quality, safety, and cost-effectiveness of patient care.
CASE 1: SELECTION OF A CARDIAC CATHETERIZATION LABORATORY
Penn Presbyterian Medical Center (PPMC) is a hospital in the University of Pennsylvania Health System (UPHS) that performs a large number of interventional cardiology procedures such as coronary angioplasty and stenting (annual volume: over 3,600 procedures). The interventional cardiology suite at PPMC includes four catheterization labs, the oldest of which was installed in 1992 and upgraded in 2000. Hospital management sought to purchase a new catheterization lab to replace that aging laboratory, and needed to select from among three competing manufacturers (referred to in this study as manufacturers A, B, and C). UPHS owned systems from all three manufacturers, although no one hospital had systems from all three. UPHS also had a preferred supplier contract with manufacturer C. While the contract did not mandate selection of manufacturer C as the supplier of the new catheterization lab, there were cost advantages to doing so. However, the cardiologists at PPMC were unsatisfied with the equipment from manufacturer C already in use at their hospital, and urged UPHS executives to select a different supplier. At that point, the executives came to our center for an evidence-based analysis of the choices.
We first reviewed the published literature on cardiac imaging equipment, and found no studies comparing the clinical performance of systems from the different manufacturers; however, we did find two published device evaluations of cardiac imaging systems from the UK Medicines and Healthcare Products Regulatory Agency (Reference Lawinski, Emerton and Mackenzie4;Reference Mackenzie, Emerton and Lawinski5). These evaluations covered different models of devices from the ones under consideration at our hospital. They also focused on radiation protection and x-ray function, and did not address device functionality in any context, let alone in the configuration and conditions of our hospital. Therefore, we had to develop our own criteria for evaluating the systems.
The most important element of the methods we used for this analysis was to define system attributes for comparison on an a priori basis. By defining the attributes before attempting to see how each system compared on these attributes, fears that the analysis was set up to favor one supplier or another could be alleviated. After a preliminary list of attributes was drawn up, the project team held a series of meetings with hospital executives, cardiologists, and laboratory managers. The attribute list was revised and expanded based on comments received at the meetings. The meetings also helped us gain a better understanding of the complaints the cardiologists had about manufacturer C's imaging system, and understand some of the issues specific to PPMC that could bear on system selection criteria.
After the meetings, the attribute list was finalized. A total of fifty-three attributes were included. Attributes were grouped into categories. Each category had between three and eight attributes, but the number of attributes did not relate to the relative importance of each category. Categorization did help to keep a balance among different types of attributes, to ensure no one category would dominate the decision-making process. One category (reliability and service) was deleted from the list because data for one manufacturer's system would have been based on a system much different from the current system. That left a total of forty-eight attributes (Table 1), of which twenty-nine were rated of high importance, fifteen medium, and four low.
Categories for system attributes
As much as possible, we tried to frame each attribute in a way that would permit us to evaluate systems in an objective manner rather than on the basis of subjective ratings by users. For example, instead of a subjective rating of ease of controlling the C-arm, we reported the number of different control devices available, and how many C-arm positions could be stored and recalled, among other attributes.
Project team members then proceeded to study the attributes of each system under consideration. The information came from manufacturers’ technical specifications, on-site inspection of systems at UPHS and affiliated hospitals, interviews with laboratory managers at those hospitals, and observations of actual procedures as they were performed by the UPHS cardiologists. Following procedures, we also had the opportunity to observe cardiologists reviewing images and carrying out quantitative analyses, and had hands-on experience at the controls of the angiography system and the image review console.
From the observations and other data, attributes of each system were abstracted into tables comparing the competing systems. Once the tables were complete, a draft report was prepared. We reconvened the group of cardiologists and managers, and asked them to rate the importance of each attribute as “high,” “medium,” or “low.” Participants were blinded to the system-specific results. Only after consensus was reached on the rankings were the comparison results revealed. A sampling of completed attribute tables is shown in Table 2.
Sample Attribute Tables
Note. Data set in boldface type represent attributes where the specified device is less satisfactory than other devices or does not meet the desired standard.
PPMC, Penn Presbyterian Medical Center.
While we sought to define and prioritize objective indicators of each system's suitability for use at PPMC, we did not try to aggregate them into a point system for decision making. Results of such a quantitative system would be heavily dependent on the number of attributes in a category. The ultimate selection decision was in the hands of the hospital administrators; our role was to provide them with objective evidence to be used in conjunction with the subjective comments from the cardiologists, technicians, and other personnel who would be working in the new lab.
The final report included a table for each category of attributes (Supplementary Table 2, which can be viewed online at www.journals.cambridge.org/thc2010021). Each row of the table contained an attribute and its assigned importance level, followed by how each manufacturer's imaging system compared on that attribute. Attributes where one manufacturer's system performed worse than another's were highlighted in the report. Manufacturer A's system had a less satisfactory finding for seven attributes: three rated high-importance, two medium-importance, and two low-importance (Table 3). Manufacturer B's system was found suboptimal on ten attributes, including five of high importance; and manufacturer C's system was found suboptimal on seventeen attributes, including twelve of high importance.
Results of Cardiac Imaging Equipment Assessment
Note. Totals from individual manufacturers do not add up to total of attributes because more than one system may not be fully satisfactory on a particular attribute.
As we completed this project, we found that one of the most important attributes for a cardiac imaging system at PPMC was the ability to image all the way from the patient's groin to the heart without repositioning the C-arm. To have to reposition the C-arm after the catheter is inserted and before it is manipulated toward the heart delays the procedure, adversely affecting productivity. At a center like PPMC with high procedure volume and where there is one physician performing the procedure rather than two, this is an important consideration. It also poses a small safety risk in the event that the patient's condition suddenly deteriorates and the catheter must be quickly changed. This attribute is not talked about in manufacturer's literature: only through the process of consulting with the cardiologists did we identify this and its importance.
Another local consideration we identified was that the picture archiving and communication system (PACS) at PPMC was no longer supported by the manufacturer. Some of the complaints about Manufacturer C's system stemmed from the obsolete PACS. No new imaging system would be able to work to its full potential with the old PACS, so one of our key conclusions was that replacement of the PACS should be prioritized by hospital management: not just for added features, but also for increased reliability and improved productivity.
Our analysis verified some of the cardiologist's complaints about the manufacturer C system in a more objective manner, while ensuring that the strengths of the manufacturer C system were given full consideration in the purchasing decision. Our analysis also provided health system executives with an objective rationale for overriding the preferred status of manufacturer C for this purchase. While they selected the system from manufacturer C for the initial purchase that gave rise to this assessment, they subsequently purchased two systems from manufacturer A.
CASE 2: STRATEGIC DECISION ON ICU TELEMEDICINE
Our second example of the use of local data in hospital-based HTA comes from a review we performed of the critical care telemedicine service in our hospital network. The telemedicine service includes remote intensivist monitoring of critical care patients along with an intelligent system that monitors patients’ vital signs and alerts caregivers to conditions that may indicate a potential problem. Proponents of the technology say it can reduce complications and improve care while paying for itself in reduced length of stay (LOS) and reduced malpractice claims.
Our system's agreement with the telemedicine provider called for an initial implementation covering approximately seventy intensive care unit (ICU) beds, including units at all three UPHS hospitals. As the trial period approached its end, UPHS executives needed to decide whether to extend telemedicine coverage to all ICUs in the network, to discontinue the telemedicine service, or to extend the trial. They sought an evidence-based analysis to ascertain whether or not the service improved patient outcomes, reduced LOS, or reduced malpractice claims.
The first step in our analysis was to systematically review the published literature for evidence on the effectiveness of ICU telemedicine.
The literature search identified only seven published articles reporting on primary studies of the effectiveness of telemedicine in critical care. Just two of them reported data from controlled studies of telemedicine in ICU care (Reference Breslow, Rosenfeld and Doerfler2;Reference Rosenfeld, Dorman and Breslow6). Both of these studies were of pre–post experimental design. The weakness of this study design is that it cannot control for other changes in ICU care which may have been made during the study period. So, even though both studies found reductions in mortality and LOS after telemedicine was introduced to their ICUs, we could not conclude that telemedicine was responsible for the improved outcomes. Beyond their limitations in validity, these studies also had limitations in generalizability to our hospitals. Both studies examined telemedicine in the context of surgical ICUs with no overnight staffing. Yet the majority of our units had house staff and/or fellow coverage 24 hours a day 7 days a week.
Absent good evidence from published trials of ICU telemedicine, we proceeded to analyze our own hospitals’ data on the impact of telemedicine. Because the trial implementation at our hospital covered only some of the critical care units, we had some nontelemedicine units for comparison. By interviewing the medical directors, we were also able to get a listing of major initiatives that were implemented during the study period to improve quality of care in each unit. Such initiatives were taken in both the telemedicine and nontelemedicine ICUs. We also obtained information on staffing of each ICU, and how it might have changed during the study period. While our analyses were done in retrospect and we could not control variables such as changes in staffing or new infection control policies, the users of the report could at least be informed of other possible causes for any observed changes in ICU outcomes.
We queried the UPHS administrative database for information on all patients who had been cared for in a UPHS ICU between July 2004 (the earliest date such data were available) and December 2008. Outcomes of interest included mortality and LOS. Outcomes were assessed for both the ICU component of the stay and the full hospital stay, so any impact of telemedicine arising after patient transfer from the ICU could be identified. Because patients with more severe disease would be expected to have higher mortality and LOS, we adjusted mortality and average LOS in our analyses using two measures of patient acuity: APR DRG case mix index (CMI) and Acute Physiology and Chronic Health Evaluation (APACHE) score. In sensitivity analyses, CMI adjustment and APACHE adjustment gave similar results.
To control for differences in patient populations across different units, we qualitatively and quantitatively compared CMI-adjusted mortality and LOS in comparable units with and without telemedicine before and after the date of telemedicine implementation. To most accurately measure a change in mortality or LOS, it is best to establish trends in these outcomes before and after the intervention of interest. Because most units had only one quarter of data before telemedicine implementation, it was difficult to establish trends during this time period, and thus difficult to perform an accurate analysis. We also compared trends in mortality and LOS after the time of telemedicine implementation in the intervention and control units. These analyses were less robust than the pre–post analyses because they were not able to account for changes in mortality or LOS directly resulting from telemedicine implementation. Instead, they could merely describe whether trends in mortality or LOS were different across units.
Overall, we observed a decreasing trend in mortality in most of the units, with and without telemedicine coverage. This supported our hypothesis that factors besides telemedicine were affecting mortality. LOS decreased in some units, both with and without telemedicine, and remained unchanged in others. We found no significant differences in mortality or LOS between units with and without telemedicine coverage. However, there was still considerable quarterly variation in the data: only large effects could be separated from the noise in the data.
While the analysis of our health system's data could not find a significant effect of telemedicine on mortality and LOS, it was enough to tell us that the effects, if any, would not be large. Because the decision on the future of ICU telemedicine in our hospitals hinged on a balance between the cost of the service and its benefits, placing an upper limit on the benefits could be sufficient for a determination that the service was not worth the added cost.
We took the same approach to an assessment of the impact of ICU telemedicine on malpractice claims. While it is impossible to add up claims that never are made thanks to the telemedicine system, we can put an upper bound on the figure by looking at claims relating to care in nontelemedicine units. We reviewed all claims filed against our network in the period July 2001 to June 2007. This included cases closed with no payment and cases still pending as well as cases that were settled or where UPHS was found liable. The initial screening by a UPHS risk management executive eliminated cases relating to outpatient care, diagnostic testing, and inpatients whose claims clearly did not relate to ICU care. That left fifteen cases from the 6-year period that were possibly related to ICU care. We reviewed the statements of claim and further narrowed the list of potentially preventable claims to four. Because the purpose of the analysis was to set an upper bound on the potential claims savings, we left the definition of “potentially preventable” quite broad. Actual savings resulting from telemedicine would almost certainly be less than the amount claimed in those four cases. The annualized cost of those four claims was substantially less than the cost of the telemedicine service, so we could conclude that the telemedicine service would not pay for itself in reduced claims.
That left staffing and workflow considerations as the remaining argument for continuing to use the telemedicine service. These are highly local considerations, so the best source of evidence on these impacts of telemedicine is interviews with local clinical and executive staff. Unfortunately, most of the evidence they can provide is anecdotal. Opinions of hospital staff were mixed. Nurses appreciated the integration of the telemedicine system with electronic medical records. That permitted them to devote more attention to patient care and less to record-keeping. They also reported that the system helped them monitor unstable patients and patients at risk for falls more closely. However, staff in some units reported that they made relatively little use of the telemedicine system, preferring to continue with the standard recordkeeping systems they were already familiar with. Furthermore, neither the nurses nor the telemedicine staff could document any specific cases where an alert from the telemedicine system prevented a serious adverse event.
The findings of our telemedicine evidence report are summarized in Table 4. We did not find definitive evidence demonstrating that the benefits of the telemedicine system exceeded the costs. Moreover, many of the benefits of the telemedicine program's electronic medical record could be replicated using the noncritical care electronic medical record in service elsewhere in the hospitals. The decision of whether to expand or discontinue the ICU telemedicine system is still pending. Scientifically, the telemedicine case illustrates not only how hospitals can generate data locally to supplement and inform evidence-based decisions, but it also illustrates the point that, even if a precise estimate of the impact of a technology cannot be made, local data may still be sufficient to support an evidence-based decision.
Potential Reasons for Expanding ICU Telemedicine Program
ICU, intensive care unit.
DISCUSSION
Hospital-based HTA centers have demonstrated that the principles of evidence-based medicine can extend beyond individual clinical decisions at the level of the patient and provider to broader institutional decisions, such as those that involve the purchasing of new technologies. Yet, there is often little evidence in the peer-reviewed literature to address these types of questions, so much of the evidence to support these decisions must be developed locally. Sources of local evidence include billing and cost accounting databases, electronic medical records, and equipment service records. Direct observation of care, interviews with clinicians, and hands-on examination of devices can add to the evidence.
Beyond filling gaps in the published evidence, the use of local evidence to support institutional decision making can also reduce problems of external validity. However, the more narrowly one defines the universe of comparable hospitals and the more strictly one defines criteria for applying published evidence to a particular institutional decision, the less evidence there will be upon which to base a decision.
In both of our case studies, we found important differences among the hospitals within our health system. These differences affect the prioritization of different attributes of a technology, and could result in different conclusions being drawn about how the technology should be used at each hospital, even within the same healthcare network.
The second important lesson from these examples is that the experience and expertise of local clinicians should be respected when making decisions at the hospital or health network level. First, it helps decision makers understand possible differences in local patient populations or in processes of care that may affect the cost or effectiveness of the technology. Second, it promotes “buy-in” from the clinicians who must implement the decision. Following evidence-based practice principles such as systematic searching for available evidence and a priori determination of criteria upon which the decision will be based gives stakeholders confidence that the outcome of the decision-making process has not been preordained by health system administrators.
Use of local data to inform evidence-based practice decisions has limitations. In particular, the data have to have been collected and available for analysis. In our analysis of ICU outcomes, we lacked APACHE scores for ICU patients before the introduction of telemedicine coverage, so our ability to control for patient acuity was limited. The claims information made available to us did not include enough detail for us to ascertain whether possible lapses in care happened in the ICU or elsewhere. While there was no such problem with availability for the survey data used in our cardiac imaging decision, gathering that data required considerable fieldwork.
CONCLUSIONS AND POLICY IMPLICATIONS
Evidence-based practice should extend from the bedside to the administrative suite. However, traditional sources of evidence may not provide enough data upon which to base institutional decisions. In that case, additional evidence must be developed locally. Health system administrators and hospital-based HTA centers should be aware of potential sources of locally generated evidence, and should develop policies to effectively integrate this data into institutional decisions. Administrators should also understand the limitations of the evidence, particularly when evidence from one setting is applied in a different setting.
Incorporating local evidence into institutional decision making requires some policy changes. First, infrastructure is needed to obtain, maintain, and interpret local data. As illustrated in the case studies, there is important evidence that does not fit neatly into the standard paradigm of controlled clinical trials.
Second, clinicians can have crucial insight into how a device or technology can increase or decrease the efficiency of care, so they should be consulted before decision-making criteria are finalized. One way to get physicians and nurses involved is to invite those with an interest in evidence-based medicine to serve as liaisons or members of the institution's purchasing or other policy-making committees, providing advice on topics related to their specialty. Clinical liaisons can also communicate the basis for a health system decision to their peers.
Third, more formal approaches to evidence synthesis and analysis might be centered in a hospital-based HTA center: coordinating evidence-gathering for hospital and health system decision makers and then developing plans to implement improved processes of care in the hospital. The hospital-focused evidence-based practice center should include personnel with experience in the acquisition and analysis of clinical evidence: including local utilization and accounting data and local clinical outcomes as well as evidence from the published literature.
Finally, system-wide decisions should be validated through follow-up with local staff. A means should be provided for system decisions to be overridden if local evidence can document that the decision is not leading to improved care. Policies that take local evidence into account when making system decisions will lead to better and more efficient care, while increasing provider satisfaction.
SUPPLEMENTARY MATERIAL
Supplementary Table 1
Supplementary Table 2
CONTACT INFORMATION
Matthew D. Mitchell, PhD (mdmitchell@uphs.upenn.edu), Coordinator, Center for Evidence-based Practice, University of Pennsylvania Health System, 3535 Market Street, Suite 50, Philadelphia, Pennsylvania 19104
Kendal Williams, MD, MPH (kendal.williams@uphs.upenn.edu), Assistant Professor of Clinical Medicine, Department of Medicine, University of Pennsylvania School of Medicine, 39th and Market Streets, Philadelphia, Pennsylvania 19104; Co-director, Center for Evidence-based Practice, University of Pennsylvania Health System, 3535 Market Street, Suite 50, Philadelphia, Pennsylvania 19104
Patrick J. Brennan, MD (p.j.brennan@uphs.upenn.edu), Professor of Medicine, Department of Medicine, University of Pennsylvania School of Medicine and Hospital of the University of Pennsylvania, 3400 Spruce Street, Philadelphia, 19104; Chief Medical Officer and Senior Vice President, Administration/Office of the CMO, University of Pennsylvania Health System, 3400 Civic Center Boulevard 5-A, Philadelphia, Pennsylvania 19104
Craig A. Umscheid, MD, MSCE (craig.umscheid@uphs.upenn.edu), Assistant Professor of Medicine, Department of Medicine, University of Pennsylvania School of Medicine, 3535 Market Street, Suite 50, Philadelphia, Pennsylvania 19104; Director, Center for Evidence-based Practice, University of Pennsylvania Health System, 3535 Market Street, Suite 50, Philadelphia, Pennsylvania 19104
CONFLICT OF INTEREST
All authors report having no potential conflicts of interest.
