Comparative evaluations of medical devices and other products are uncommon in the literature and rarely conducted. Such evaluations compare the performance of the same product made by different vendors (“head-to-head” comparison of vendor 1 versus vendor 2) or different products that treat the same illness (head-to-head comparison of technology A versus technology B).
In the pharmaceutical arena, new drugs are typically evaluated in terms of absolute efficacy (i.e., relative to a placebo) rather than in terms of relative efficacy (i.e., relative to other drugs in the therapeutic category or older product versions). To the extent that they exist, comparative drug evaluations have typically concentrated on comparing different categories or therapeutic interventions. In general, Food and Drug Administration approval and randomized controlled trials have limited ability for comparing products and procedures routinely used by physicians, which may be more amenable to practical clinical trials and observational studies (3;22;28;32). Only recently have head-to-head evaluations of competing drugs gained prominence. Some examples are the Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) and the Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial (ALLHAT) (1;21). Given the national expenditures on drugs and their potential to improve outcomes, head-to-head comparisons have huge clinical and economic implications (1).
The National Institute for Health and Clinical Excellence (NICE) in the United Kingdom relies on comparative clinical effectiveness information to make coverage decisions regarding drugs and medical procedures. In the United States, by contrast, there is no central agency.
Public sector interest in the United States is growing, however. Section 1013 of the Medicare Prescription Drug, Improvement, and Modernization Act (MMA) authorized the Agency for Healthcare Research and Quality (AHRQ) to evaluate and synthesize available scientific evidence regarding the comparative clinical effectiveness of products and services. AHRQ subsequently launched its Effective Health Care Program in 2005, which included comparative effectiveness reports and observational studies.
Recently, on February 13, 2007, Senator Max Baucus, the new Chair of the Senate Finance Committee, outlined several principles for reforming the U.S. healthcare system. At the center of the third principle, cost controls, Baucus argued the need for comparative effectiveness research to justify the technological imperative fueling rising healthcare spending (20). His arguments mirrored statements in late 2006 by Gail Wilensky, former Administrator of the Health Care Financing Administration, and Carolyn Clancy, Director of the Agency for Healthcare Research and Quality (10;32).
There are several reasons for this increased interest. First, increased government spending on the MMA drug benefit has increased the government's interest in comparative effectiveness of pharmaceuticals. Second, the cost of new technology and its utilization accounts for 20–40 percent of the annual rise in U.S. healthcare spending (6). The share of spending on drugs and devices will soon approach the levels spent on physician and hospital services (6). Third, while U.S. healthcare reform has focused on private sector competition, the agenda has recently focused on the transparency of provider price and quality data to inform consumer choice.
Similar transparency of vendors’ prices and product quality can inform product selection by providers. For hospitals to impact the rising cost of healthcare, they will need to confront the estimated 30 percent of their expenses dedicated to supply acquisition costs, procurement, and handling (6). Such costs have not been systematically controlled in the past (5). To do so, hospitals will need to understand how their clinicians evaluate vendors and their products, and whether or not clinicians view competing vendors’ products as clinically comparable. In the face of persistent vendor marketing efforts, and the close association of vendor sales representatives with some specialists, hospitals need to develop rigorous and reliable product evaluations, independent of any particular vendor (who oftentimes conducts the studies for the provider), and gather data that are credible in the eyes of physicians. Transparency and comparative evidence regarding product effectiveness can have the further benefit of stimulating vendors to bring more cost-effective products to market (23).
RATIONALE
This study documents the comparative effectiveness of one class of medical devices (sutures and endomechanical products) and seeks to improve the evidence base for how to select among these products. Following AHRQ researchers, we define “effectiveness” in terms of how the products work in practice and “comparative effectiveness” as the relative effectiveness of similar products made by alternative vendors (2). Using a surgeon panel's input, we specify relevant dimensions of product ease-of-use and ergonomic comfort, and then evaluate the vendors of these products in head-to-head manner. The study describes an evaluation design that can be replicated across hospitals and extended to other products. It illustrates how to conduct comparative clinical studies that promote cost-effective product selection, vendor competition, and hospital purchasing of medical devices in general.
Lack of Evidence Base in Surgeons’ Product Selection
General surgeons select suture and endomechanical products made by several vendors. Selection is a function of testimonials, anecdotal experiences, guesswork, habit, and tradition (29). There is growing concern that selection of medical device products is based on nonclinical factors such as financial relationships (17;19). The lack of evidence-based practice in product selection is due to a lack of evidence, not surgeon compliance.
The technology assessment firm ECRI uncovered few peer-reviewed studies comparing vendor differences in these product categories (12). One exception is a series of studies of the biomechanical performance of sutures made by leading vendors (14;15;18). The ECRI review also found no systematic differences across vendors in product incident rates drawing on its database of adverse events and problem reports. ECRI researchers concluded there was insufficient evidence to conclude whether competing vendors’ products are of comparable quality.
Economics alone argue for the need for comparative effectiveness studies of these products. Suture and endomechanical products account for a large dollar volume of hospital supply spending. According to 2004 IMS Health data, annual suture sales are roughly $1 billion, and endomechanical sales are roughly $1.5 billion ($1 billion endoscopy, $0.5 billion mechanical).
Determinants of Successful Product Performance
ECRI argues that product performance is a function of product design, product misuse, and patient anatomical differences (12). Other important determinants of product performance are “human factors and usability” (7;24;27). Across eight categories of suture and endomechanical products, safe product use is associated with the surgeon's familiarity, comfort, preference, and ease of product use. An earlier ECRI review of trocar use found that serious adverse consequences were due primarily to surgeon technique and the surgeon–technology fit (13). Similar findings have been reported in studies of cardiac surgeons’ use of minimally invasive instruments and orthopedists’ use of implants and instrumentation (24;25). At the other extreme, many hospitals execute purchasing agreements primarily on the basis of cost, assuming clinical equivalence as the norm, and neglect physician input. Consequently, ECRI recommends that hospitals conduct evaluations to solicit physicians’ preferences and input regarding product quality.
Prior studies and reviews have identified many of the usability issues with these products (12;13;29;31). For sutures, key issues are handling, knot security, tensile strength, and tissue drag. For needles, the issues are strength and sharpness. For trocars, the issues concern ease of use, instrument transfer, seal durability, and sleeve security in tissue. For staplers, the issues are ease of use, staple formation, and clamping firing force. For clip appliers, finally, the issues are ease of use and clip security.
Shortcomings of Prior Industry-Sponsored Evaluations
Evaluations of these products and vendors have typically been conducted by group purchasing organizations (GPOs) and vendors. One GPO cosponsored evaluations with a leading vendor. High-volume surgeons assessed the clinical acceptability of that vendor's products, with no head-to-head comparisons with other vendors, and no consideration of human factors (29–31).
In contrast, vendors commission studies to compare how their products fare against competitors, gathering non–quality-related performance measures (sales and service). Vendors also develop marketing literature to highlight the advantages (absolute and relative) of their products, often with minimal supporting data (16).
Finally, surgeons at hospitals with minimally invasive surgery training centers conduct (unpublished) evaluations of these products. Whereas such evaluations focus on outcomes and human factors, the number of clinicians is small, as are the number of product categories and vendors compared. Because the vendors differ across studies, it is impossible to draw generalizations.
Methods
A large national GPO evaluated five categories of devices (suture and endomechanical products) from eight vendors pursuant to awarding contracts. We describe the elements of the study design.
Product Categories
Surgeons evaluated five categories of products: (i) Clip appliers; (ii) Internal mechanical and endoscopic-mechanical staplers; (iii) Trocars (bladed and nonbladed); (iv) Sutures and needles; and (v) Endoscopic specimen retrieval devices.
Staplers included both internal mechanical and endoscopic mechanical products. Trocars included both bladed and nonbladed 10-mm products. Sutures included both absorbable and nonabsorbable, and both natural and synthetic products.
To ensure product comparability across vendors within a given category, the GPO requested products by matched, cross-referenced catalogue numbers. Vendors shipped only those products and product numbers designated by the GPO to the study sites. Once on site, GPO staff maintained complete control of the products, cataloging the products to be consistent across sites.
Multiple, Participating Vendors
The evaluation must include the market leading vendor(s) that surgeons prefer. It must also include newer entrants and smaller vendors to promote price competition, include innovative products, and mimic the range of vendors and products on the market.
The GPO extended an invitation to all vendors in the five-product categories to participate in the evaluation. Eight vendors submitted products in overlapping categories that could be compared with one another. These firms include the two market leaders and six smaller firms. Among the latter, not all vendors offered or contributed products in each category for evaluation. Some categories (e.g., staplers, trocars, sutures) had more than one type of product. Consequently, the study evaluated a total of fifty-one unique vendor products.
Each vendor was required to adhere to the following “Supplier Rules of Engagement.” Products supplied had to be sterile “patient ready” products shipped at no charge to the evaluation sites. Suppliers must provide a minimum of fifty eaches (unopened containers) of product per site. Suppliers had to provide only the products designated for evaluation with catalog numbers provided by the GPO clinical evaluation coordinator. Supplier sales representatives and clinical specialists could not attend the evaluation or contact any participating surgeons. All products had to arrive at the site at least 3 business days before the evaluation. Suppliers were not allowed to provide alternative evaluation forms to the evaluation facility staff, or to scrutinize evaluation forms used. Finally, suppliers could not provide any form of incentive to surgeons, evaluation coordinators, or evaluation site staff.
Multiple, Participating Surgeons
The evaluation required multiple physicians to evaluate vendors and their products. This strategy necessitates working with local hospitals to recruit their general surgeons and surgical subspecialists. Given the limited supply of such physicians in some hospitals or markets, it may be necessary (and even desirable) to recruit surgeons from a wider geographic area to ensure representativeness.
The GPO invited surgeons from its member hospitals to participate. The GPO intended to exclude any surgeon who received a stipend, grant, retainer, or salary from any of the suppliers. A small number of surgeons on retainer nevertheless participated. Rather than exclude them from the analyses, we statistically controlled for their retainer relationships (which subsequently proved to exert little effect). A total of forty-five surgeons participated in this study from April to September 2005. Comparison of the forty-five surgeons with the American Medical Association Masterfile data revealed no significant differences (all p > .05) in terms of physician age, gender, subspecialty, training (United States versus international medical graduate), hours of hospital practice, or primary type of practice. None of the surgeons was compensated for their participation, other than travel expenses.
Relevant Product Performance Measures
The GPO convened a surgeon panel from member hospitals before the evaluation to review the study design and identify the relevant measures of product quality and performance. A lengthy list of criteria was considered for each product category (available from lead author). In the interests of parsimony, and to streamline the laboratory evaluation, the panel reduced the number of criteria. The list echoes the performance dimensions mentioned in the literature.
The measures assess products in terms of ergonomics, functionality, overall performance, clinical equivalence, and relative rank-order. The panel developed multiple items for each performance dimension to ensure consistency in the evaluations. The dimensions and items are listed in Table 1 (also available at Journal's Web site: http://www.journals.cambridge.org/jid_thc).
Performance Dimensions and Constituent Survey Items, by Product Category
Study physicians answered the items in uniform manner across the five product categories using Likert scales that rate products on a 1–7 scale: (7 = extremely satisfactory, 4 = neutral, 1 = extremely unsatisfactory). Physicians also answered whether they believed the products were clinically acceptable (“Would you use this product routinely?”) in a yes versus no manner. Finally, for clip appliers, trocars, and sutures, surgeons rank-ordered the products against each other: 1 is ranked highest, 2 is ranked second highest, and so on. Again, in the interest of parsimony, we developed summary scales of the items to capture the overall dimension. Table 1 provides the Cronbach's alpha coefficient (which measures the internal consistency and scalability of items) for each scale (4;10;11).
Statistical Controls
Many factors in the physician's background might influence product evaluations. Prior preference for a vendor and tenure with its products might positively influence that vendor's evaluation and negatively influence the evaluation of others. Such factors cannot be avoided but can be statistically controlled. Similarly, a clinician on retainer might evaluate that vendor more positively. Such biases can also be statistically controlled; better yet, they can be eliminated by screening out physicians with such ties from the study.
Before the evaluation, surgeons completed a survey regarding their background (age, gender, and subspecialty), physical characteristics (hand preference, height, glove size), vendor background (past/current retainer relationships, tenure with current vendor), product preference (e.g., bladed versus nonbladed trocars), and vendor preferences (preferred vendor for the product evaluated). The study also included a novel question on the physician's tendency to play video games, theorizing it might be associated with the surgeon's laparoscopic skills and suturing capability (26). This information can then be used to construct measures used in the analyses as statistical controls to isolate the impact of vendor on product performance.
Another possible bias results from differential sampling, that is, over/under sampling physicians on characteristics that impact the performance evaluations. Some likely characteristics might be physician age (correlated with experience and tenure with a vendor) and/or gender (correlated with ease of instrument use). Survey data on participating physicians need to be compared with national (e.g., American Medical Association) data using chi-square analyses. In the event that significant differences in the two distributions are found, the researcher can empirically construct a correction term (inverse mills ratio) for use in subsequent analyses.
Laboratory Settings
This type of evaluation is well-suited to an animal lab, where the surgeon can (i) go to any particular station and operate on multiple animal subjects using the same product made by multiple vendors, and (ii) rotate from station to station to evaluate different products. Animal labs associated with academic medical centers (AMCs) served as the evaluation sites. The labs were recruited to gain national geographic representation of physicians. None of the AMCs were GPO member hospitals. None of the animal labs had any prior relationship with the GPO. Because some labs and AMCs may have had prior ties or ongoing support from specific vendors, the investigators included the lab site as a covariate in all analyses. Review Boards at each lab reviewed and authorized the study protocol. Medical Directors at the AMCs served as advisors on the study's design and implementation and validated the types of products included in the analysis. The labs provided qualified surgical scrub technicians to assist each surgeon.
Three roles/functions were performed in the labs: the clinical evaluation coordinator, the lab coordinator, and the laboratory technician. The clinical evaluation coordinator scheduled the surgeons, staff, evaluation products, supplies, instruments, and animals at the lab; defined and oversaw the duties of the lab coordinator and technicians, and oriented them to the protocol; conducted pre-evaluation interviews with each surgeon to obtain procedure and personal preferences (i.e., suture sizing, needle preference, stapling instrumentation sizing); recorded the catalog and lot numbers of each product used; and ensured the surgeon's completion of the evaluations.
The lab coordinator and surgical scrub technician were responsible to provide all care, handling, and post-care handling of animals; ensure lab availability and readiness for scheduled procedures; ensure availability and readiness of all nonevaluation instrumentation, equipment, and routine surgical supplies; provide adequate anesthesia for designated animals; and perform all cleanup.
Standardization of Surgical Tasks
Four stations were established in each lab, with specific procedures to be conducted with specific types of sutures and endomechanical products. Surgeons rotated through four stations in the lab, but did not start out at the same station for two reasons. First, GPO staff wanted to minimize any possible influence of station and station sequence on the evaluations. Second, because multiple physicians might be scheduled in the lab at the same time, GPO staff sought to physically separate them and their product evaluations.
Each station contained a different set of products to be tested using slightly different procedures. Due to surgeon time constraints, surgeons who rotated through all four stations evaluated roughly forty of the fifty-two products. At each station, a qualified surgical scrub technician assisted the surgeon with the tasks to be completed. Because variation in surgical technique might influence the evaluations, GPO staff sought to standardize the task technique and sequence at each station. The detailed tasks for each station, as well as the pre-lab session activities, are available from the lead author.
Standardization of Data Collection
All data were collected on standardized evaluation forms tailored to each product and procedure track. In the process of completing the tasks, the scrub technician read aloud product evaluation questions and recorded the surgeon's responses immediately after a given product's use. The clinical evaluation coordinator ensured that the surgeon evaluated the appropriate set of products at each station. At the end of the evaluation, the physician's responses were sealed inside an express mail packet and sent to the researchers for analysis.
The physician evaluations are thus based on direct product trial, not on previous recall of product usage and impression. The evaluation design permits head-to-head comparisons of different vendors of the same product across five product categories. To ensure accuracy, the evaluations were also videotaped and sent to the researchers for confirmation of both the procedure and data.
Knowledge of Evaluation Design and Statistical Analysis
This study is obviously not a randomized controlled trial, but rather a quasiexperimental design using statistical controls. In such designs, researchers are necessarily concerned with threats to internal validity (are the reported differences in vendors’ products due to some other influence?) and external validity (can the differences in the sample be generalized to a wider population?) (9). One major set of threats to internal validity are history and maturation among the subjects due to different tenures or closer relationships with vendors. Such factors can bias the surgeon's evaluation of any given vendor. Another major threat to both internal and external validity is sampling bias, which can limit the generalizability of the results. To combat these threats, researchers use statistical controls and methods to assess the representativeness of the physician sample described above.
The evaluation of each vendor product is conducted at the physician level of analysis. The dependent variable is the surgeon's evaluation of each vendor's product on multiple dimensions of performance. The first major research question is whether there are significant differences in the evaluations of different vendors of the same product. The second major research question is whether these differences persist after controlling for the surgeon's preferences, background, and personal characteristics. Researchers commonly compute least-square analyses on the dependent variables that control for all covariates (an analysis of covariance). The specific analytic technique used here is hierarchical random effects regression. Random effects models are required because of repeated observations over the same physician, which can lead to correlated error terms across ratings within a physician. We note that this is not a fully-crossed design, because each vendor does not offer a product in each of the five categories.
Results
The unadjusted means on the performance dimensions for each vendor's product are presented in Table 2 (also available at Journal's Web site: http://www.journals.cambridge.org/jid_thc). To simplify the presentation, we have collapsed product varieties within a given product category (e.g., endocutter and internal cutter staplers, disposable bladed and nonbladed trocars). Analyses of vendor differences by product variety within a product category are analyzed elsewhere (8). The last column shows the results from a test for significant differences among the means.
Surgeon Evaluation of Vendors, by Product Category (Unadjusted Means)
The results reveal several patterns. First, vendors’ products differed significantly on all performance dimensions. In some cases, the variation in performance scores was large. For example, the percentage of physicians rating clip appliers as clinically acceptable ranged from 19.4 percent to 95.1 percent across vendors. Similar ranges were found for staplers and trocars.
Second, Vendor 3 received consistently higher ratings than the other vendors across all product categories. Conversely, Vendors 2 and 8 received consistently low evaluations across several product categories. These findings suggest that physicians perceive distinct differences in the products made by different vendors, which may translate into distinct vendor preferences. The national GPO that commissioned this project used these findings to select the vendor(s) they wished to contract with for suture and endomechanical products.
The second research question is whether differences in product evaluations persisted when statistically controlling for factors and experiences in the physician's background that might influence these evaluations. We estimated hierarchical random effects regression models that controlled for background information and possible nonresponse bias, and then recalculated the appropriate estimates. The results closely mirrored the results presented in Table 2 above, and are thus not replicated here. In the regressions, the specific vendor accounted for more of the explained variation in product evaluations than did physician background characteristics.
Discussion
We acknowledge several limitations of the study. The surgeon sample, while nationally representative, is small and may underrepresent physicians on unmeasured dimensions that might influence product evaluations (e.g., physician anatomical differences). Our analyses also rest on animal laboratory evaluations, not on patient outcomes. Furthermore, the performance data deal with indirect measures of quality (e.g., ergonomics, functionality) rather than clinical results. Finally, our evaluation does not include every vendor or a random sample of vendors.
This study does add to a small body of literature analyzing the performance differences between comparable medical device products made by competing vendors. This literature is important given the fact that, historically, product choice may be based on subjective rather than objective, evidence-based criteria. The study suggests one methodology for conducting technology assessment of these products that solicits the performance evaluations of physicians. Such evaluations can support cost-effectiveness analyses that guide product choice and ultimately improve surgical outcomes.
The evaluation design described above has several advantages for the hospitals that purchase these products. First, it solicits product performance data from the hospital's physicians and, thus, enjoys both credibility and relevance for its patient base. Second, such data can validate or rebut vendors’ marketing claims about their products. Third, as a result, providers can overcome the information asymmetry typical in the device vendor–hospital (buyer–seller) relationship and purchase the best product at a competitive price. Fourth, because the research design has surgeons evaluating multiple products by multiple vendors, there is no need for expensive, long-term, and large-scale studies to gather the data needed for multiple comparisons. Fifth, the comparative studies can be shared across providers to form an independent source of reputable research findings that do not suffer from the potential biases of (and likely criticisms leveled at) studies funded either by the vendors or the insurance companies. In this study, the providers collaborated with university researchers to achieve a rigorous study design and contribute to a small but growing evidence base. As the U.S. Government and healthcare industry debate the value of a center for clinical effectiveness information, the private sector can take the initiative to develop some of the requisite research base.
CONTACT INFORMATION
Lawton R. Burns, PhD, MBA (burnsL@wharton.upenn.edu), The James Joo-Jin Kim Professor, Department of Health Care Systems, The Wharton School – University of Pennsylvania; Director, Wharton Center for Health Management & Economics, The Wharton School, 3641 Locust Walk, Philadelphia, Pennsylvania 19104
Eric T. Bradlow, PhD (ebradlow@wharton.upenn.edu), K. P. Chao Professor, Professor of Marketing, Statistics, and Education, The Wharton School of the University of Pennsylvania, 3730 Walnut Street, 761 Jon M. Huntsman Hall; Academic Director, Wharton Small Business Development Center, The Wharton School of the University of Pennsylvania, 3620 Spruce Street, Vance Hall, Philadelphia, Pennsylvania 19104
J. Andrew Lee, ScM (juholee@wharton.upenn.edu), PhD Candidate, Department of Health Care Systems, The Wharton School, University of Pennsylvania, 3641 Locus Walk, Philadelphia, Pennsylvania 19104
Anthony C. Antonacci, MD, SM, (aantonacci@gmail.com), Clinical Associate Professor of Surgery, Department of Surgery, Weill Medical College of Cornell University, 68th & York Avenue, New York, NY 10021; VPMA & CQO, Christ Hospital, 176 Palisade Avenue, Jersey City, New Jersey 07306
