Economic evaluations are increasingly being conducted and are supposed to follow the standards for such studies (Reference Drummond, Sculpher, Claxton, Stoddart and Torrance1), the underlying principles of which have been adopted by national healthcare organizations, such as the Health Care Knowledge Centre (KCE) in Belgium, Institute for Quality and Efficiency in Health Care (IQWIG) in Germany, the National Institute for Health and Care Excellence (NICE) in the United Kingdom, the Canadian Agency for Drugs and Technologies in Health (CADTH), as well as the French health authorities (Haute Autorité de Santé, HAS). Guidelines for reporting economic evaluations prescribe several criteria that should be adopted to improve the interpretability of such studies, and improve the confidence of decision making (Reference Husereau, Drummond and Petrou2). It has been found that, at least in the field of cost-utility analysis (CUA), the quality of reporting increased in the latter part of the past century (Reference Neumann, Stone, Chapman, Sandberg and Bell3). However, CUAs are only part of the medico-economic evaluations performed, and it is necessary to evaluate the quality of reporting of all types of health economics studies.
Cardiac surgery is one of the medical specialties that has seen substantial scientific progress over the past few decades. As the costs of medical care in this area rise (Reference Cohen and Sukin4), driven largely by the development of innovative medical technologies (Reference Bodenheimer5–Reference McNeil8) that tend to improve medical outcomes and increase costs (Reference Newhouse6;Reference Baker, Birnbaum, Geppert, Mishol and Moyneur9;Reference Bell, Urbach and Ray10), there has been increasing interest in evaluating the cost-effectiveness of medical practices (Reference Cohen and Sukin4). The costs of therapeutic innovations in cardiac surgery, such as heart valve repair or replacement and heart transplant, are quite high (Reference Yaghoubi, Aghayan, Arjmand and Emami-Razavi11–14). Thus, the quality of reporting of economic studies in this field of research is of paramount importance, yet to the best of our knowledge, no published study has evaluated this. The aim of the study was, therefore, to conduct a systematic review of the literature on health economic evaluations in the field of cardiac surgery, and to evaluate the quality of reporting of the selected articles and its change over time using the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) (Reference Husereau, Drummond and Petrou2).
Methods
A systematic search of the literature following the PRISMA statement was conducted from March to August 2017 to identify published articles reporting health economic evaluations of cardiac surgery until December 31, 2016.
Study Selection
Inclusion Criteria
We included in this systematic review full health economic evaluations such as: cost benefit analyses (CBA), cost-effectiveness analyses (CEA), CUA, and cost minimization analyses (CMA) conducted on cardiac surgery without restriction for language. Cardiac surgical procedures included were: open heart surgery, heart transplant, coronary artery bypass grafting (CABG), and minimally invasive surgery.
Exclusion Criteria
Studies that were not a full health economic evaluation or did not investigate cardiac surgery, as well as those that investigated cardiac procedures or postoperative complications were excluded; reports that were not full articles, such as meeting abstracts, letters to the editor, treatment guidelines or recommendations, expert opinion, or narrative reviews, were also excluded, as were duplicated articles.
Literature Search
A systematic search of the literature was conducted using Medline, Web of Science, NIHR CRD Database, Cochrane Library, Scopus, Cost Effectiveness Analysis Registry, Pascal and Francis, and Science Direct. The search terms used were: (Cardiac Surgical Procedures OR Cardiac Surgery) AND (Cost Benefit Analysis OR Cost Effectiveness Analysis OR Cost Utility Analysis OR Cost Minimization Analysis OR Economic Evaluation). The reference lists of the selected articles were examined to identify additional articles, and the grey literature was identified from OpenGrey. The preselection of the studies was carried-out based on titles and abstracts identified by the querying of the databases. Subsequently, the preselected articles were carefully read in full, and according to the inclusion and exclusion criteria, eligible articles were included in the literature review. Articles written in a language other than English or French were translated for analysis.
Two authors (N.L. and P.G.) carried out the literature search, identified articles independently, and assessed the abstracts of the identified studies. They independently appraised the studies considered in this review.
Data Extraction and Quality Assessment
Data were extracted using a standardized extraction table. Study characteristics related to the publications (title, authors, journal name, year of publication, and country, study design comparators, type of health economic evaluation, outcome measure, perspective, time horizon, discount rate) were collected. There are several health economic evaluation tools for evaluating the quality of publications including the study authored by Weinstein et al. (Reference Weinstein, O'Brien and Hornberger15) that reports a tool for the evaluation of modeling studies, as well as other tools (Reference Husereau, Drummond and Petrou2;Reference Williams16). The most widely used is the Drummond and Jefferson checklist (Reference Drummond and Jefferson17). This checklist is intended for wide readership, including both specialists and nonspecialists in health economic evaluations, and is simple to use.
The present study used the CHEERS checklist (Reference Husereau, Drummond and Petrou2). The International Society of Pharmacoeconomics and Outcomes Research (ISPOR) introduced the CHEERS statement in 2013. This checklist attempts to consolidate and update previous efforts into a single useful reporting standard. It is not intended to prescribe how economic evaluations should be conducted; its objective is to ensure that the methodology used is clear and conscientious. It is the most recent checklist, and covers all the main domains of the economic evaluation, such as the Drummond and Jefferson checklist. However, it is more detailed, makes several distinctions and follows the IMRaD structure, which facilitates the evaluation of each part of an article independently.
The CHEERS is a twenty-four–item scale covering six main categories: title and abstract, introduction, methods, results, discussion, and source of funding, and conflicts of interest. To estimate a summary reporting score, a value of 1 was assigned if the study fulfilled the requirement of reporting for that item completely, 0.5 for partially completing the requirement, NA if it is not applicable and otherwise 0. We acknowledge that this method is subjective, but in absence of clear recommendations of the authors of CHEERS checklist, we preferred a simple method of scoring, without attributing ourselves levels of importance to each item. Therefore, the maximum score for a publication that reports completely according to these standards is 24. Ethical approval was not necessary, because the present systematic review did not involve patients.
Statistical Analysis
First, we describe the number and proportion of publications in each period, country, type of review, type of surgical procedures, type of clinical study, and type of health economic evaluation. Second, we report the mean quality score with standard deviation of: all publications per 10-year interval, for each type of surgical procedure, each type of clinical study, each type of health economic evaluation, and each type of publication journal. An analysis of variance (ANOVA) was performed to explain the influence of periods on quality scores. Third, we evaluate items least addressed before and after 2000.
Results
Study Selection
A total of 4,705 records were identified from the initial search. Among these, 4,500 were discarded after screening the titles and abstracts and after removing duplicates, 407 underwent full-text examination; 338 were subsequently excluded (not full economic evaluation, about postoperative complications and about cardiac procedures). In total, sixty-nine studies were included in this systematic review (Figure 1).
PRISMA flow-chart.
Overview of Included Studies
Studies were published between 1981 and 2016. The number of publications increased considerably during this period; six (8.7 percent) studies were published between 1981 and 1990, nine (13 percent) between 1991 and 2000, twenty-four (34.8 percent) between 2001 and 2010, and thirty (43.5 percent) after 2011. Studies were mainly conducted in North America (n = 32; 46.4 percent); twenty-five (36.2 percent) were conducted in Europe, six (8.7 percent) in other countries (Australia, Brazil, Korea, Iran, and Israel), and six (8.7 percent) studies considered several countries in the analysis. Seven (10.1 percent) studies were published in journals aimed at professionals interested in health economics such as the International Journal of Technology Assessment in Health Care, and sixty-two (89.9 percent) were published in clinical journals such as the American Journal of Cardiology.
There were thirty-one (45 percent) studies that investigated CABG, eighteen (26.1 percent) investigated open heart surgery, fourteen (20.2 percent) investigated minimally invasive surgery, and six (8.7 percent) heart transplant. Considering the type of study, twenty-one (30.5 percent) were observational analytical studies, nineteen (27.5 percent) were randomized controlled clinical trials, twenty-five (36.2 percent) were modeling studies, and four (5.8 percent) were nonrandomized controlled clinical trials. Most studies were CUAs (n = 41; 59.4 percent), followed by CEAs (n = 25; 36.2 percent), CBAs (n = 2; 2.9 percent), and one (1.4 percent) was a CMA.
A total of twenty-two (31.9 percent) studies applied a fixed time horizon, such as 1, 3, 5, or 10 years; twenty-five (36.2 percent) studies considered a lifetime horizon; eight (11.6 percent) studies considered another period such as 6 months; fourteen (20.3 percent) studies did not specify the time horizon considered in the study. A societal perspective was used in seven (10.1 percent) studies, third party payer perspective in twenty-eight (40.6 percent) studies, two (2.9 percent) studies used hospital perspective, and thirty-two (46.4 percent) studies did not specify the perspective. Regarding the discount rate, fourteen (20.3 percent) studies applied 3 percent, eight (11.6 percent) studies 3.5 percent, one (1.4 percent) study 4 percent, seven (10.1 percent) studies 5 percent, and thirty-nine (56.4 percent) studies did not apply a discount rate.
Reporting Quality
The results of the assessment of reporting quality according to period are summarized in Table 1. The mean score of all articles was 16.71 (standard deviation [SD] ± 4.71), and ranged from 8.5 to 24. The mean score of articles reporting studies published between 1980 and 1990 was 10.25 (±1.44), for those published between 1991 and 2000, it was 11.21 (±2.46), for those published between 2001 and 2010 it was 15.34 (±4.8), and for those published after 2011 it was 19.97 (±2.92). The mean (±SD) score of articles investigating open heart surgery was 17.1 (±5.12), for articles investigating heart transplant it was 16.33 (±3.1), for articles investigating CABG it was 15.95 (±4.02), and for articles investigating minimally invasive surgery it was 18.04 (±4.14).
Reporting Quality According to Period of Publication
Note. p: comparison with Fisher-ANOVA test
Regarding the type of medico-economic study, the mean score of articles reporting modelling studies was 19.08 (±4.99), for articles reporting randomized controlled clinical trials it was 18 (±6.02), for articles reporting nonrandomized controlled clinical trials it was 13.25 (±6), and for articles reporting observational analytic studies it was 13.12 (±4.85). Concerning the type of health economic evaluation, the mean score of articles reporting a CUA was 19.03 (±3.15), for articles reporting a CEA it was 14.68 (±4.84), for articles reporting a CBA it was 9.75 (±0.56); for the article reporting a CMA the score was 19.5. The mean score of articles published in a health economics journal was 21 (±4.24), and for articles published in a clinical journal the mean score was 16.54 (±5.06).
For the fifteen studies published before 2000, the items that were the least frequently addressed in the CHEERS statement were: a sufficiently detailed abstract (n = 7; 46.6 percent); study perspective (n = 0); time horizon (n = 1; 6.6 percent); discount rate used (n = 2; 13.3 percent); measurement and valuation of preference-based outcomes (n = 7; 46.6 percent); currency, price date, and conversion (n = 0); choice of model (n = 1; 6.6 percent); assumptions (n = 1; 6.6 percent); incremental costs and outcomes (n = 5; 33,3 percent); evaluation of heterogeneity of the results by subgroups analysis (n = 0); study findings, limitations, generalizability, and current knowledge (n = 7; 46.6 percent); the role of funding source in the design, conduct, and reporting of analysis (n = 6; 40 percent); and the conflicts of interest (n = 1; 6.6 percent). Those results are presented in Figure 2.
Percentage of medico-economic evaluation who completed the items before and after 2000.
For the fifty-four studies published after 2000, some items were better reported such as: time horizon (n = 42; 77.7 percent); incremental costs and outcomes (n = 44; 81.4 percent), and study funding, limitations generalizability and current knowledge (n = 44; 81.4 percent). The quality of reporting was still insufficient for several studies after 2000, especially concerning items “characterizing heterogeneity,” “assumptions,” and “choice of model.”
Discussion
The present study found that the number of health economic evaluations in the field of cardiac surgery has increased over time, and that the reporting quality has also improved. These findings are similar to those described elsewhere (Reference Ma, Jian and Xu18).
The assessment criteria used in the present study (CHEERS) have been applied in several systematic reviews reporting the quality of health economic publications (Reference Ma, Jian and Xu18–Reference Aguiar, Lima and Storpirtis21). There were several items that are partially or not reported by a large number of articles. The most frequently not reported was the item “a sufficiently detailed abstract” (i.e., there is no information about perspective, discount rate, time horizon, or uncertainty analysis). This was followed by the item “study perspective”; authors should describe in the main text the perspective (e.g., health care system, societal) in terms of costs included and their associated components (e.g., direct medical costs, direct nonmedical costs, and indirect/productivity costs), and how this fits the needs of the target audience(s) and decision problem (1;2).
Another item that was also rarely reported in the main text was time horizon. This refers to the length of time over which costs and consequences are being evaluated, and should be sufficiently long to reflect all important differences in costs or outcomes between the technologies being compared. Heterogeneity may be important if particular patient subgroups differ with respect to observed or unobserved characteristics, such as age or sex, or differ systematically in ways that affect the results of an economic evaluation. If heterogeneity is important, authors should report differences in costs, outcomes, or cost-effectiveness that can be explained by variations between subgroups of patients. Aguiar et al. (Reference Aguiar, Lima and Storpirtis21) have reported that this item is poorly reported in publications. Sixty-five percent of studies published after 2000 reported information about the source of funding. There was insufficient information about the role of industry in the identification, design, conduct, and reporting of the analysis; given that there is a known association between positive results in health economic studies and industry sponsorship for their own products, it is crucial to ensure absolute transparency in research (Reference Bell, Urbach and Ray10;Reference Aguiar, Lima and Storpirtis21;Reference Garattini, Koleva and Casadei22).
The quality of reporting has increased over time but certain quality aspects still need to be addressed in future evaluations. Items that we believe indispensable for reporting, such as time horizon, study perspective, and measurement of effectiveness, are more frequently reported over time. These items are the essential upstream steps for the interpretation of the study results; for instance, it is not possible to interpret whether estimated costs are pertinent when studies do not state what perspective they were based on. Furthermore, failure to report the perspective limits the usefulness to decision makers. Likewise, it is not possible to evaluate whether the study was conducted over a sufficiently long period to measure impact in terms of costs and effectiveness over time without knowing the time horizon, and the conclusion of a study cannot be understood without knowing how the Measurement of effectiveness was done.
Despite guidelines for conducting health economic evaluations and checklist for evaluating the quality of these studies having been widely available for many years, and previous reviews having already criticized economic evaluations for poor reporting, we observed that the quality of reporting was still insufficient for several studies, especially concerning items “characterizing heterogeneity,” “assumptions,” and “choice of model.” Future studies have to improve reporting on those items. We must also underline that the CHEERS checklist was published in 2013, thus, the increase of the quality until 2013 is not due to its use, but rather to the availability of guidelines at the very end of the last century (Reference Drummond, Sculpher, Claxton, Stoddart and Torrance1).
Although we followed recommendations for conducting a systematic review of quality of health economic evaluations (Reference Jefferson, Demicheli and Vale23), the present study has some limitations. First, despite a systematic search, it is possible that some economic studies where missed because there were unpublished or not indexed in the databases searched. Second, scoring may be underestimated for studies in which some of the items were not applicable such as CMA studies that do not the item “measurement of effectiveness.” However, there are very few CMA studies (n = 1 in the present review). This potential overestimation of the scores also exists for the cost-effectiveness studies, for which the item 12, “measurement and valuation of preference based outcomes,” is scored N/A. In addition, differentiating between partially or fully reported was difficult for some items and we assigned a score of 0.5 for partial reporting, which could be questionable and lead to an increase of the overall score of the studies.
A minor limitation of CHEERS checklist can bring about the possibility of bias (Reference Gonzalez-Perez24): Gerkens et al. (Reference Gerkens, Crott and Cleemput25) reported that the results of the quality assessment of health economic evaluations are more greatly influenced by the assessor than the instrument used. A systematic review of the literature investigating the quality of health economic evaluations provides important information and scientific evidence which will improve the quality of future studies. This review highlights that future economic studies should adhere to the recommendations and guidelines published by several organizations (Reference Drummond, Sculpher, Claxton, Stoddart and Torrance1;Reference Husereau, Drummond and Petrou2), and use CHEERS checklist to be sure to provide good quality reporting that can be used in the decision-making process.
In conclusion, focusing on the field of cardiac surgery, this review found an increasing number of published health economic evaluations and an improvement of quality of reporting of these studies between 1981 and 2016. However, some items are still less commonly reported. Clarity of reporting should be applied in future publications to allow greater interpretability of the results.
Author ORCIDs
Hugo Rabier, 0000-0001-6055-4476; Hassan Serrier, 0000-0002-5920-1830
Acknowledgements
The authors thank the Hospices Civils de Lyon, and especially Dr. Philip Robinson for help in manuscript preparation.
Financial support
This research received no specific grant from any funding agency, commercial or not-for-profit sectors.
Conflicts of interest
The authors declare that there are no conflicts of interest.
