Introduction
The use of formal health technology assessment (HTA) is considered common practice by national health systems and insurance companies to inform new technology adoption and reimbursement decisions. HTA is less common within hospitals and health systems (Reference Sampietro-Colom and Martin1). The premise of hospital-based health technology assessment (HB-HTA) is that more informed and structured decisions about the adoption and use of health technologies will improve clinical outcomes and hospital efficiency (Reference Landaas, Baird, Hansen, Flum and Sullivan2;Reference Landaas, Eckel, Wright, Baird, Hansen and Sullivan3). The emergence of HB-HTA activities was derived from the necessity to mitigate the risk of adopting new technologies without sufficient evidence of clinical benefit and potential cost impact (Reference Sampietro-Colom and Martin1). Given that most hospitals make technology decisions at the system level, clinical efficacy, safety, utilization, and cost data should be used to inform these decisions (Reference Landaas, Baird, Hansen, Flum and Sullivan2;Reference Landaas, Eckel, Wright, Baird, Hansen and Sullivan3). However, new technology decision making in hospitals is influenced by a variety of competing interests of various stakeholder groups, including clinicians, hospital financial leadership, patients, and contracted vendors (Reference Landaas, Baird, Hansen, Flum and Sullivan2). As hospitals in the United States of America (USA) have different cultures and processes in place and also differ in several respects with their counterparts in other countries, there is no one-size-fits-all HB-HTA model, and, therefore, varying approaches to implementing programs within a hospital are common (Reference Sampietro-Colom and Martin1).
In 2017, the University of Washington Medical Center (UWMC), a Seattle-based integrated hospital system with four hospital campuses and affiliates, designed, developed, and implemented an HB-HTA program called Smart Innovation. We describe Smart Innovation in more detail in a previously published paper (Reference Landaas, Baird, Hansen, Flum and Sullivan2). In brief, the program provides a process and decision framework to evaluate new technologies objectively, and it works with clinical and executive committees to inform adoption decisions (Reference Landaas, Baird, Hansen, Flum and Sullivan2;Reference Landaas, Eckel, Wright, Baird, Hansen and Sullivan3). The program reviews new technologies that are requested by providers and utilizes nine dimensions of evidence and is considered a comprehensive HB-HTA program (Reference Landaas, Baird, Hansen, Flum and Sullivan2;Reference Landaas, Eckel, Wright, Baird, Hansen and Sullivan3). Smart Innovation brings together stakeholders from hospital administration, clinical leadership, finance, purchasing, and supply chain to optimize new medical technology adoption decisions (Reference Landaas, Baird, Hansen, Flum and Sullivan2;Reference Landaas, Eckel, Wright, Baird, Hansen and Sullivan3). There are very few HB-HTA programs in the USA and a paucity of data evaluating these programs (Reference Gagnon4). This paper reports on the impact of two medical technologies that were adopted at the UWMC.
We compared the UWMC's adoption and diffusion of two medical devices and their associated surgical procedures to matched control hospitals in the USA. We selected one technology that did not undergo formal HB-HTA assessment and was adopted without restriction, the Sentinel™ filter cerebral protection system for transcatheter aortic valve replacement (TAVR), as a control. We then selected a device that underwent a formal HB-HTA review, microwave ablation (MWA) for treating hepatocellular carcinoma. These technology adoption decisions were conducted during the early phase of Smart Innovation, with MWA serving as the institution's first HB-HTA review.
Medical Technologies
TAVR Procedures with the Sentinel™ Cerebral Protection System
The Sentinel™ filter for TAVR procedures was approved by the U.S. Food and Drug Administration (FDA) in 2019 (5). The purpose of the Sentinel™ filter is to prevent aortic wall debris and blood clots from reaching the brain and causing stroke in patients who undergo a TAVR procedure. It is a single-use device and the filters are designed for immediate removal once the TAVR procedure is complete (6). At the time of FDA approval, there was limited evidence available to formally assess efficacy, safety, and budget impact on hospitals. The manufacturer of the Sentinel™ filter claimed that the device reduces strokes; however, the evidence we briefly reviewed was uncertain (6). The new device incurred an additional per procedure cost of $2,850. The UWMC, thus, adopted the Sentinel filter + TAVR procedure without a formal HB-HTA review. We selected this technology as a control given that it did not undergo formal HB-HTA.
Liver Ablation for Treating Hepatocellular Carcinoma
Radiofrequency ablation (RFA) and MWA are therapeutic modalities for treating hepatocellular carcinoma. RFA technology was initially approved by the FDA in 1995 for treating cardiac disorders and was later used off-label for liver tumor ablation (7). The MWA device that we evaluated was approved by the FDA in 2017 (8). The therapeutic effect of both methods relies on thermal injury, but MWA uses an electromagnetic field as opposed to the electrical current used in RFA (Reference Poulou, Botsa, Thanou, Ziakas and Thanos9). Data support the contention that MWA can reduce hospital readmission rates for additional ablation treatments, because MWA can target larger tumors when compared with RFA. However, large-scale, randomized, and prospective clinical trials are needed to understand more completely the clinical and patient outcome difference between RFA and MWA (Reference Poggi, Tosoratti, Montagna and Picchi10).
Clinicians at the UWMC requested permission to use MWA technology, and the Smart Innovation program was, therefore, asked to conduct a technology assessment and make recommendations to the clinical and executive committees regarding adoption. Smart Innovation conducted an HTA including evidence synthesis, budget impact analysis, clinical guideline review, a survey of insurance coverage, and a review of MWA's safety, efficacy, and comparative performance versus RFA technology. The HTA for MWA indicated that MWA had health outcomes similar to the current liver ablation technology (RFA); however, based on our HTA, MWA achieved fewer adverse events (e.g., bleeding, pain) for patients and required less follow-up procedures. Our HTA also indicated that MWA achieved cost savings because the disposable components were less expensive when compared with RFA.
MWA was approved for adoption in July 2017 and implemented at the UWMC in January 2018, and we selected MWA for this study because seven quarters of postadoption data were available for this analysis. The MWA adoption was implemented at the same hospital (UWMC) as the TAVR Sentinel™ filter, which also made it a good comparator.
Hypotheses
We hypothesized that the adoption rate for TAVR procedures with a Sentinel™ cerebral protection system would not differ between the UWMC and the control hospitals over a similar time period. Conversely, we hypothesized that there would be a difference in the adoption of MWA at the UWMC as compared with the control hospitals that was temporally related to the HB-HTA intervention. Furthermore, we hypothesized that a different MWA adoption rate at the UWMC versus control hospitals would be associated with overall costs.
Methods
We conducted a retrospective, observational study from October 2015 to December 2019. The study population included the UWMC and seventeen control hospitals, and we selected controls using specific inclusion and exclusion criteria as well as medical procedure codes to gather utilization data to compare control hospitals with the UWMC. We compared the UWMC with controls between preintervention and postintervention periods, which were separated by the point in time where the UWMC implemented TAVR and MWA. There are two main dimensions in these analyses: Hospitals and Procedures.
Hospital Control Selection Process: Inclusion and Exclusion Criteria
We identified candidate control hospitals based on inclusion criteria gathered from a national claims database (Vizient Clinical Data Base) supplemented with data from the American Hospital Association (AHA) to establish a list of potential controls (Tables 1 and 2) based on hospital characteristics and surgical volume. Vizient is a health-care performance improvement company that serves 95 percent of academic medical centers in the USA (11). The AHA maintains the largest database of hospitals in the USA and is a national organization that represents all hospitals (12).
TAVR procedures for UWMC and control hospitals, 2016 and 2018
UWMC, University of Washington Medical Center; TAVR, transcatheter aortic valve replacement.
a American Hospital Association (AHA).
b Vizient Clinical Data Base
MWA procedures for UWMC and control hospitals, 2016 and 2018
UWMC, University of Washington Medical Center; MWA, microwave ablation.
a American Hospital Association (AHA).
b Vizient Clinical Data Base.
We established unique inclusion criteria for control hospitals by a procedure based on the UWMC's bed capacity, the number of TAVR and MWA procedures conducted at the UWMC, and whether the control was an academic medical center. To identify a sample of potential controls for each procedure, the Vizient Clinical Data Base was queried to find hospitals with the number of hospital beds and procedures similar to those of the UWMC, with the goal of finding a minimum of five control hospitals for each procedure that had similar characteristics to the UWMC.
For TAVR procedures, the control hospitals’ inclusion criteria included: (i) the number of hospital beds must be between 400 and 800, (ii) the control hospitals needed a total of more than 400 TAVR procedures in 2018 and 2019, and (iii) must be an academic medical center. For exclusion criteria, control hospitals for TAVR procedures must not have evidence of using the Sentinel™ filter device based on that device's unique billing code. These criteria provided twelve control hospitals for TAVR.
For MWA procedures, the inclusion criteria included: (i) the number of beds between 300 and 900, (ii) more than 100 total liver ablation procedures in 2018 and 2019, and (iii) must be an academic medical center. These criteria yielded eleven control hospitals. Because RFA and MWA share the same billing codes, claims data could not be used to differentiate between the two technologies, so we contacted each of the eleven potential MWA control hospitals to determine which met exclusion criteria. The exclusion criteria for the MWA control hospitals included the following: (i) hospitals with a formal HB-HTA program and (ii) hospitals using the same technology as the UWMC (MWA). We removed those meeting the exclusion criteria, which left five control hospitals for the MWA analysis.
Analysis: TAVR and MWA Procedures
We utilized surgical procedure billing codes to gather TAVR and MWA surgical data. The level of analysis for both technologies was at the hospital level and patient and hospital data were aggregated into quarters. The main outcome variable of interest was the number of procedures per quarter, and the data were gathered for seventeen quarters between 1 October 2015 and 30 December 2019.
We conducted descriptive analyses to evaluate the UWMC's and control hospitals’ utilization by comparing mean utilization before and after the intervention, and we visualized the trends by plotting the number of TAVR and MWA procedures over time. To aggregate utilization trends for the TAVR and MWA procedures, we calculated arithmetic and weighted means for the number of procedures per quarter. Weights were based on the total number of surgical procedures for the UWMC and each control hospital in the pre- and post periods. We then compared the arithmetic and weighted means to determine if they were different by assessing the percent differences between the pre- and post periods. We concluded that there was no difference in the results based on using either arithmetic or weighted means, and, thus, we used the arithmetic means to aggregate control hospitals in these analyses.
We evaluated individual hospital characteristics and surgical volume in both UWMC and control hospitals for each technology and surgical procedure. We summarized the number of procedures per quarter for each hospital and compared the percent difference between pre- and postperiod trends in utilization for the UWMC and controls. For the TAVR and MWA procedures, we determined if there was an observed difference among utilization trends between the UWMC and the controls.
In addition to descriptive data analyses to measure the effect of Smart Innovation, we employed a difference-in-differences (DID) analytic approach. We first evaluated whether utilization trends were equivalent across the UWMC and controls in the pre period (parallel assumption) by plotting utilization curves for the UWMC and controls. Because visual assessment can be subjective, we then established a multivariable regression model for the number of procedures per quarter for TAVR and MWA in the pre period to test for statistical significance for the interaction between group status (the UWMC vs. control) and time (quarter), while adjusting for hospital characteristics in the pre period. Hospital characteristics assessed for adjustment included the number of acute care beds, as well as hospital-level patient demographics from TAVR and MWA procedures (age, gender, race, payer type, comorbidities).
For the TAVR and MWA procedures, we performed regressions using ordinary least squares (OLS) in the DID models to adjust for hospital-level heterogeneity using demographics that were gathered for each of the seventeen quarters in this study. Analyses were performed using the R statistical software [R v4.0.2, (13) using RStudio v1.3.1073] yielding a p-value significance of differences between the UWMC and the controls for the number of procedures per quarter. Human subjects' review was approved by the UWMC's Institutional Review Board (IRB).
The DID analysis used outcome-specific multivariable regression models, adjusting for hospital-level characteristics for TAVR and MWA. For each DID model, the main outcome measure included the number of procedures per quarter, and the primary covariate of interest was the interaction between time and intervention, where time was pre- and post-HB-HTA decision (0 = pre, 1 = post), and intervention was the implementation of HB-HTA policy decisions at the UWMC (0 = controls, 1 = UWMC). The p-value was used as the measure for statistical significance and the coefficient value would provide the estimated difference in utilization. The DID models were, therefore, used to test whether Smart Innovation policy decisions impacted utilization at the UWMC. Adjustments to the model were made for age, gender, race, and payer type, comorbidities, and the number of acute care beds for TAVR procedures, whereas adjustments for age, gender, race, payer type, and the number of acute care beds were used for MWA procedures. We also adjusted for time (quarter) in our models to control for any temporal differences.
Results
The control hospitals for the TAVR and MWA procedures matched with the UWMC based on teaching status, the number of admissions, and the number of TAVR and MWA procedures (Tables 1 and 2). Among controls for TAVR procedures, all twelve were nonprofits, whereas the UWMC was a state-owned entity. Conversely, as regards the control hospitals used for the MWA procedure analysis, the ownership/state-owned status was similar. For both TAVR and MWA procedure control hospitals, the proportion of these selected procedures to the total surgical procedures was higher at the UWMC than at the control hospitals.
TAVR Procedures
Comparisons of average patient age, percent male, percent white, percent Medicare enrollee, percent with comorbidities, and the number of acute care beds between the UWMC and the controls in the pre- and post periods for TAVR procedures are shown in Supplementary Table 1. There was a standardized mean difference between average age, percent Medicare, and the number of acute care beds between the UWMC and the controls, and all covariates were adjusted in the regression.
We observed a 10 percent reduction in the number of TAVR procedures between the pre- and post periods when comparing the percent change between the UWMC and the controls (Table 3). There were eight control hospitals with a higher percent change (pre–post periods) when compared with the UWMC and four that had a lower percent change compared with the UWMC. Based on our TAVR trend plot, we did not observe a difference in utilization trends between the UWMC and the controls during the time of this study (Supplementary Figure 1).
Mean TAVR procedures per quarter, pre- and post periods
UWMC, University of Washington Medical Center; TAVR, transcatheter aortic valve replacement.
In addition to descriptive data analyses to estimate the effect of Smart Innovation, a multivariate DID analysis was used to test the difference between the UWMC and the controls for the number of procedures per quarter in the pre- and post periods. Potential confounding variables were selected for the regression: age, race (percent white), gender (percent male), payer type (percent Medicare), comorbidities (Elixhauser co-morbidity index score of 1 or more). We also adjusted for time (quarter) and the number of acute care beds. After adjusting for covariates, the DID estimator was −1.5, indicating no statistically significant change in utilization between pre- and post-Sentinel Filter + TAVR implementation (p-value: .87). The DID estimator indicates the change in the number of TAVR procedures per quarter between pre- and post periods, which was 1.5 less per quarter for the UWMC when compared with controls. There were two covariates in the model that were statistically different between the UWMC and the controls (p-value <.05), which included the number of acute care beds and comorbidities.
MWA Procedures
For MWA procedures, we compared patient average age, percent male, percent white, percent Medicare, and the number of acute care beds between the UWMC and the control hospitals between October 2015 and December 2019 (Supplementary Table 2). The only variable with a standardized mean difference above one was the number of acute care beds, which was adjusted for in the model. When comparing the number of MWA procedures at the UWMC with those at the control hospitals, we observed a strong difference in utilization between the pre- and the post periods at the UWMC versus control hospitals. There was an increase in the number of MWA procedures per quarter between the pre- and the post periods at the control hospitals, yet at the UWMC, MWA decreased (Table 4). The UWMC also experienced a downward trend in the number of MWA procedures over the study period, whereas the control hospitals experienced an upward trend during the same time period (Supplemental Figure 2).
Mean MWA procedures per quarter, pre- and post periods
UWMC, University of Washington Medical Center; MWA, microwave ablation.
The UWMC's percent change between the pre- and the post period for the number of MWA procedures per quarter was the most reduced compared with individual control hospitals and the difference in the percent change was 51 percent (Table 4). In addition to descriptive data analyses to estimate the effect of Smart Innovation, a multivariate DID analysis was used to test the difference between the UWMC and the controls for the number of procedures per quarter in the pre- and post periods. Potential confounding variables were selected for the regression: hospital-level age (average age), race (percent white), gender (percent male), and payer type (percent Medicare). We also adjusted for time (quarter) and the number of acute care beds. After adjusting for covariates, the DID estimator was −18.8, indicating a statistically significant change between pre- and postintervention time periods in utilization, relative to the MWA HB-HTA policy decision (p-value: .0007). Only time was found to be a significant covariate between the UWMC and the control hospitals in this analysis (a p-value of <.05). The DID estimator indicates the change in the number of MWA procedures per quarter between the pre- and the post periods, which was 18.8 less per quarter for the UWMC when compared with that for the controls.
We observed a difference in the total costs when comparing the UWMC with the controls between the pre- and the post periods (Supplementary Table 3). The total costs for liver ablation procedures rose modestly at the control hospitals but decreased at the UWMC. Specifically, the UWMC spent $1,702,444 more than the control hospitals prior to the HB-HTA intervention, whereas it spent $647,658 dollars less than the controls after the intervention. The difference in the total costs is consistent with the MWA utilization trends seen in Supplementary Figure 2, the percent change between the UWMC and the controls in Table 4, and the DID results for MWA.
Discussion
This study assessed the early impacts of implementing the Smart Innovation program at the UWMC. We presented the results from a new technology that did not undergo Smart Innovation review and was adopted without restrictions (TAVR). We then evaluated a surgical device adoption decision that underwent a review through the Smart Innovation decision framework. We, thereby, gained insights into surgical trends associated with adopting new technologies in the hospital setting with and without the use of HB-HTA methods.
Based on our analyses, there was a slight difference between the UWMC and the controls for the technology (TAVR) that did not undergo HB-HTA review that did not achieve statistically significance. Technology that was reviewed by Smart Innovation at the UWMC, however, appears to have decreased significantly, with substantial financial savings.
We noticed a difference between the UWMC and the controls based on ownership for TAVR procedures, whereas the controls were all nonprofits and the UWMC was state-owned. We have no reason to believe that this makes the controls systematically different from the UWMC, because there is no significant difference between nonprofit and state-owned hospitals as both of them are not for profit.
The overall increasing trend in the number of TAVR procedures for both the UWMC and the controls during the time of this study (Supplementary Figure 1) likely reflects what is occurring in hospital practice, as TAVR procedures are increasing based on demand and hospitals have responded by increasing the number of procedures over time (Reference Goldsweig, Tak, Chen, Aronow, Shah and Kolte14). Adoption of the TAVR Sentinel™ filter without Smart Innovation review is, therefore, a reasonable target for an evaluation and comparison of the effect of HB-HTA, because it reflects a real-world example of a medical device that occurred organically without high-level scrutiny or intervention. The independent DID test results for each of the two technologies allowed this study to determine whether our HB-HTA approach, Smart Innovation, makes an impact on the real-world data for adopting new medical technologies (Reference Wing, Simon and Bello-Gomez15).
A key assumption using DID models is the parallel trend assumption, that is, the assumption that outcome data will maintain parallel trends in the prior to intervention (Reference Abadie16). Because visual inspection of parallel trends can be subjective, we conducted a multivariable DID regression to empirically test if there was a significant difference in preperiod data using the interaction between time (quarters) and intervention (HB-HTA or not) as the variable of interest. These DID models indicated that the difference between the UWMC and the controls was not statistically significant for the number of TAVR and MWA procedures in the pre period.
Additionally, we plotted each individual control hospital's utilization curves. This allowed us to determine which individual hospitals matched the best with the UWMC and those that did not. We then ran DID models comparing the UWMC with three of the five control hospitals that matched well with the UWMC's utilization trends, as well as one hospital that matched the best with the UWMC. After adjusting for covariates, we had the same results (p-value <.05) when compared with the complete model that included all the control hospitals in the DID model.
Limitations
We acknowledge potential selection bias related to control hospital and technology selection. We selected control hospitals with very narrow inclusion and exclusion criteria. For example, if a potential control hospital had an HB-HTA program or had made a technology decision that was similar to that of the UWMC, they were excluded in this study. There was a small sample size for some of the hospital controls (n = 17) and technology comparisons were limited to two. We selected MWA and TAVR technologies because they were implemented in the pilot phase of Smart Innovation's development and provided a longer time of study for this analysis (post period). Among some controls, the counts of procedures per quarter were small for comparing utilization between the UWMC and the controls. We also recognize that there are many potential confounders that we were unable to measure, such as hospital culture, procurement practices, leadership style, and other characteristics that may have differed across hospitals. Furthermore, the analysis may not be generalizable to U.S. hospitals that are not academic medical centers, have hospital bed size and surgical volume outside of control inclusion criteria, and have different utilization trends among the technologies being reviewed in this study.
We did not identify the number of surgeons who conducted MWA or TAVR procedures among the hospitals in this study. Hospitals could have recruited or lost TAVR or MWA surgeons during the time of this study, which could have had an impact on the number of procedures. Hospitals could have also increased or decreased their marketing efforts for these procedures and this may have impacted the number of TAVR and MWA procedures. Therefore, the number of procedures conducted at the UWMC and controls may be explained by the change in personnel and varying marketing efforts by an individual hospital.
One of the DID model challenges was identifying potential confounding variables either observed or unobserved prior to the study, and we were limited to variables available in Vizient's database. However, DID designs assume that confounders varying across the groups are time invariant, and time-varying confounders are group invariant (Reference Wing, Simon and Bello-Gomez15). One of the potential unknown factors was the time it takes for control hospitals to make adoption decisions and incorporate new technologies into care. The actual uptake of the new technologies was not available as our count data only indicated the number of procedures performed and not which type of technology was used (e.g., old vs. new technology).
Because comorbidities were captured using diagnostic-related group (DRG) data, which are used for inpatient procedures only, we were not able to capture comorbidities for procedures that were conducted in an outpatient setting. TAVR procedures were conducted in an inpatient setting, and liver ablation procedures were conducted in a mix of inpatient and outpatient settings. Therefore, we were not able to adjust for comorbidities in the MWA DID models.
Conclusions
This study reviewed two medical technologies at the hospital level and evaluated utilization based on adoption decisions that were made with and without the support of Smart Innovation at the UWMC. HB-HTA has the potential to improve hospital outcomes; however, it is not common in the USA. This analysis provided insights from a quantitative evaluation of a new HB-HTA program and how the UWMC compared with other similar hospitals regarding new technology adoption and diffusion. When the UWMC used HB-HTA methods for technology adoption, there was a difference in utilization when compared with controls; when the UWMC adopted a new technology without using HB-HTA methods, there was no difference in utilization. When the UWMC used HB-HTA methods, there was a reduction in the total costs when compared with controls following HB-HTA policy decisions. The difference in utilization observed to be correlated to HB-HTA was associated with a large decrease in the total cost.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462321000477.
Acknowledgments
The authors thank Erik Walerius and Dave Flum of UW Medicine and Alyssa Harris, Sam Hohmann, and Rick May of Vizient for their support and assistance with this project.
Funding
The project described was supported by Funding Opportunity Number CMS-331-44-501 from the U.S. Department of Health & Human Services, Centers for Medicare & Medicaid Services. The contents provided are solely the responsibility of the authors and do not necessarily represent the official views of the HHS or any of its agencies.
Conflict of Interest
The authors have no conflicts to report.
