Clinical evidence required for market authorization is mostly obtained from randomized controlled trials (RCTs), which are the gold standard for assessing the efficacy of new drugs (Reference Garrison, Neumann, Erickson, Marshall and Mullins1). However, for many countries, drug approval is not sufficient to allow the integration of innovations in the healthcare system. Indeed, the widespread use of a drug is often closely linked to the reimbursement decisions of public payers. Although therapeutic value is a key component in decision making, health technology assessment agencies also consider the cost-effectiveness of innovations, which is estimated through economic evaluations. Data required to perform a cost–utility analysis comprise utility values and costs, which allow the estimation of the incremental cost-effectiveness ratio. Indirect costs, such as productivity loss, are also considered in economic evaluations performed from a societal perspective. Although they have gained prominence, patient-reported outcomes (PROs) and economic outcomes are not systematically collected in RCTs. Even if they were, RCTs are well-known to have inherent limitations, namely low external validity, highly controlled patient follow-up, and staggering costs. It has led to the use of real-world data (RWD) as complementary evidence in reimbursement decision making (Reference Garrison, Neumann, Erickson, Marshall and Mullins1).
Despite the increasing use of RWD, no formal consensus on its definition has yet been reached. A recent review identified thirty-eight definitions of RWD based on fifty-three published documents and twenty stakeholder interviews (Reference Makady, de Boer, Hillege, Klungel and Goettsch2). The definition developed by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) (i.e., data that are not collected in conventional RCTs) (Reference Garrison, Neumann, Erickson, Marshall and Mullins1) was the most commonly used (53 percent). In 24 percent of the definitions, RWD was restricted to data collected in a setting without interference with treatment assignment, patient follow-up and patient selection (Reference Makady, de Boer, Hillege, Klungel and Goettsch2). Five definitions (13 percent) were limited to data collected in routine clinical practice (no de novo data collection, i.e., use of existing data only) (Reference Makady, de Boer, Hillege, Klungel and Goettsch2). The most cited data sources were registries, electronic health records, claims databases, and administrative databases, followed by PROs and health surveys (Reference Makady, de Boer, Hillege, Klungel and Goettsch2).
Notwithstanding the lack of universal definition of RWD, a commonly voiced purpose is to complement RCT findings for health decision making (Reference Garrison, Neumann, Erickson, Marshall and Mullins1–Reference Wang, Schneeweiss, Berger, Brown, de Vries and Douglas5). Good Clinical Practice guidance provides a quality standard for performing RCTs. As for the conduct of real-world evidence (RWE) studies, a unified standard has yet to be made. Indeed, lack of methodological rigor is a concern about RWE studies (Reference Berger, Sox, Willke, Brixner, Eichler and Goettsch3). The objective of this study was to characterize methodological practices of studies collecting pharmacoeconomic data, namely utility values, resource utilization, and productivity loss, in a real-world setting for the management of type 2 diabetes mellitus (T2DM). T2DM has been chosen to serve as a case study because of its important burden on the healthcare system. Indeed, according to the World Health Organization (WHO), the global prevalence of diabetes (types 1 and 2) has risen from 4.7 percent in 1980 to 8.5 percent in 2014 (6). In 2014, the prevalence of diabetes was 7.3 percent in Europe, 8.3 percent in America, 8.6 percent in South-East Asia, and 13.7 percent in the Eastern Mediterranean region (6). In addition, the worldwide direct cost of diabetes is estimated to be over US$827 billion per year (6).
Methods
Literature Search and Study Selection
A systematic literature review was performed according to a predefined protocol. The study question was established using the PICO framework: population consisted of T2DM patients, intervention and comparator were any intervention for T2DM care or absence of intervention, and outcomes were utility values, resource utilization and productivity loss. A structured literature search was performed using the electronic databases MEDLINE and EMBASE. To capture all potentially relevant studies, the search used pharmacoeconomic terms combined with pharmacoepidemiologic keywords, and was thereafter restricted to diabetes. Full details of the literature search are presented in the Supplementary Material (Supplementary Table 1). Studies were initially selected based on titles and abstracts. Full-text articles were then reviewed using a predefined eligibility form. Inclusion criteria were:
• Studies performed in a real-world setting, herein defined as studies that were not RCTs, and that were performed during routine clinical practice. There was no restriction on interference with treatment assignment or randomization;
• Studies collecting de novo data (no studies based on electronic health records, claims databases, or administrative databases);
• Studies focusing on T2DM (no restriction on age);
• Studies collecting pharmacoeconomic data (utility values, resource utilization, and productivity loss);
• Studies published in English or French;
• Studies published between 1 January 2017 and 27 June 2018;
• Full-text articles (no conference abstracts).
The criterion of de novo data collection was used to exclude retrospective database studies, in which pharmacoeconomic outcomes are usually lacking. Although highly relevant for the assessment of clinical outcomes and direct medical costs, most clinical databases, such as electronic health records, do not provide information on health state utility, productivity loss, and non-medical costs. Even if some databases may provide pharmacoeconomic data, this review did not attempt to analyze pharmacoeconomic results in T2DM but rather focused on research practices. Analyzing the existing databases does not involve the same study conduct and research methodology as generating new evidence in studies using primary data collection. Therefore, this review focused only on studies with de novo data collection. All eligibility criteria were defined a priori and studies needed to meet all of the inclusion criteria to be eligible. Study selection was performed by two independent reviewers, resolving any disagreements through discussion and consulting a third reviewer if necessary.
Data Extraction
For each included study, the following characteristics were extracted using a data collection form: first author, country, design, patient population, intervention, exposure group, measure of exposure, study site, number of study sites, sample size, sample size calculation, sampling method, source for patient selection, recruitment method, recruitment duration, participation rate, ethics approval, patient's consent, duration of follow-up, data monitoring, study public registration, disclosure of conflicts of interest, source of funding, pharmacoeconomic outcome, number of outcome measures, research instrument, source for measure of outcome, recall period, extent of missing data, and external validity. Studies may have collected clinical outcomes in addition to the pharmacoeconomic outcomes. However, the data extraction focused solely on data pertaining to pharmacoeconomic outcomes, and may not reflect the entire purpose of a study. In addition, results of the studies were not presented as this review aimed at characterizing methodological practices only. Data extraction was performed by two independent reviewers, resolving any disagreements through discussion and consulting a third reviewer if necessary. Extracted data were presented as descriptive statistics using frequencies.
Quality Assessment
The risk of bias in the results of comparative studies was assessed using the ROBINS-I tool (Risk Of Bias In Non-randomized Studies of Interventions) (Reference Sterne, Hernan, Reeves, Savović, Berkman and Viswanathan7). As for descriptive studies, the quality was assessed using the Joanna Briggs Institute (JBI) critical appraisal tool for descriptive studies (Reference Lockwood, Sfetcu and Oh8). Critical appraisal was performed for each pharmacoeconomic outcome by two independent reviewers, resolving any disagreements through discussion and consulting a third reviewer if necessary. The ROBINS-I tool covers seven domains: confounding, selection of participants, classification of interventions, deviations from intended interventions, missing data, measurement of outcomes, and selection of the reported result. Domain-level and overall risk of bias judgments were made according to the ROBINS-I guidance. Results were summarized in a weighted summary plot (Reference McGuinness9). The JBI critical appraisal criteria for descriptive studies are: random sample, clear inclusion criteria, identification of confounding factors, outcome assessment validity, sufficient description of groups, sufficient time period, inclusion of patients who withdrew, reliable method for outcome measurement, and appropriate statistical analysis. The quality score was reported in percentage as not all criteria applied to all studies.
Results
The literature search identified 1,158 potentially relevant studies, among which 48 were duplicates. The 1,110 remaining articles were screened by titles and abstracts. Of these, 171 full-text articles were assessed for eligibility. After excluding 111 studies not meeting eligibility criteria, 60 studies were included in the literature review (Supplementary Figure 1).
Table 1 summarizes study characteristics of included articles. Thirty-six studies (60 percent) assessed resource utilization, twenty-two studies (37 percent) collected utility values, and eight studies (13 percent) reported productivity loss. A typical diabetic population was included in 48 percent of studies, whereas 35 percent restricted the study sample to patients receiving a specific intervention, 8 percent to patients with a specific complication/comorbidity, and 8 percent to middle age and/or elderly. Twenty-five studies (42 percent) evaluated a single group whereas thirty-five studies (58 percent) included at least two exposure groups, among which seven were based on medical interventions. The main study design was cross-sectional (68 percent) followed by prospective cohort studies (18 percent). Most studies were performed in hospital outpatient clinics (57 percent), followed by primary care (25 percent) and hospital inpatient departments (15 percent). Sample sizes varied between <50 patients (8 percent), 50–100 patients (8 percent), 101–250 patients (27 percent), 251–500 patients (28 percent), 501–1,000 patients (17 percent), and >1,000 patients (12 percent). Sample size calculation was precision-based for 18 percent of studies and power-based for 15 percent of studies. A non-probability sampling method was used in 68 percent of studies whereas 15 percent used a probability sampling. Ten studies (17 percent) reported participation rates below 80 percent whereas eleven studies (18 percent) reported participation rates of 80 percent and over. Complete data extraction was not possible for many studies. Indeed, exposure measurement was unclear for 8 percent of studies. Also, 20 percent of studies did not describe the source for patient selection, 13 percent did not report the number of study sites, and 38 percent did not report recruitment duration. The sample size calculation and the sampling method were not explained in 67 and 17 percent of studies, respectively. Twenty-two percent of studies did not mention being approved by an ethics committee and 8 percent did not mention obtaining patient's consent. Several studies did not disclose conflicts of interest (13 percent) or funding (20 percent). Moreover, 22 percent of studies did not demonstrate any evidence of a formal protocol (i.e., no study public registration and no institutional approval). In addition, external validity was not discussed in 25 percent of studies.
Characteristics of studies (n = 60)
a Studies may be classified in more than one category, that is, exceed 100%.
Characteristics of pharmacoeconomic outcomes are presented in Table 2. Among the thirty-six studies assessing resource utilization, 39 percent were comparative and 61 percent were descriptive only. Eleven percent did not mention the source for outcome collection and 3 percent did not report the mode of administration. Only 11 percent of the research instruments were validated forms, namely the International Diabetic Federation Diabetes Impact questionnaire (3 percent), the Rapid Assessment of Avoidable Blindness-Diabetic Retinopathy survey (3 percent), the Behavioral Risk Factor Surveillance System survey (3 percent), and the Health and Social Services Utilization Inventory (3 percent). Twenty-two studies reported utility values, among which 68 percent were comparative. Only validated questionnaires were used to collect utility, namely EQ-5D index (43 percent), EQ-5D index and visual analog scale (VAS) (26 percent), VAS only (13 percent), SF-6D (9 percent), standard gamble (4 percent), and time trade-off (4 percent). The mode of administration was unclear in 23 percent of the studies. Six (75 percent) of the eight studies evaluating productivity loss were descriptive only. Research instruments were researcher developed forms for 75 percent of studies collecting productivity loss, whereas 13 percent used the Endicott Work Productivity scale, and 13 percent used the International Diabetic Federation Diabetes Impact questionnaire. The extent of missing data was not reported for 22 percent of the resource utilization outcomes, 23 percent of the utility outcomes, and 38 percent of the productivity loss outcomes.
Characteristics of pharmacoeconomic outcomes
a Studies may be classified in more than one category, that is, exceed 100%.
None of the comparative studies were classified at low risk of bias. Among the thirty-one analytical outcomes, 26 percent were judged at moderate risk of bias, 45 percent at serious risk, and 29 percent at critical risk. The domains the most at risk of bias were confounding, outcome measurement and reporting of results. Half of the studies did not report sufficient information on which to base a judgment about the risk of selection bias. No information was reported about exposure measurement, outcome assessment, and missing data in 6, 6, and 26 percent of the studies, respectively (Figure 1). Risk of bias for each comparative outcome is presented in the Supplementary Material (Supplementary Table 2). None of the descriptive studies met all the JBI critical appraisal criteria. Among the thirty-five descriptive outcomes, 6 percent had a quality score below 25 percent, 17 percent a score between 25 and 49 percent, 37 percent a score between 50 and 74 percent, and 40 percent a score between 75 and 90 percent. The criteria that were less likely to be met were random/pseudo-random selection of patients and outcome assessment validity (Figure 2). Quality assessment for each descriptive outcome is presented in the Supplementary Material (Supplementary Table 3).
Risk of bias in comparative studies.
Quality assessment of descriptive studies.
Discussion
The aim of this study was to characterize methodological practices of studies collecting pharmacoeconomic outcomes in a real-world setting for the management of T2DM. This objective broadened our knowledge on how this type of study is performed and documented whether concerns about RWE procedural practices are true or not. Results of this review showed that most studies were cross-sectional, performed in hospital outpatient clinics, and measured self-reported pharmacoeconomic outcomes. Another noticeable observation was that complete data extraction was not possible for many studies, showing a lack of transparency. Indeed, several studies did not clearly explain the methods of outcome and exposure assessment. A high proportion of studies did not describe sample selection, recruitment duration, number of sites, sample size calculation, and sampling method. Several studies did not demonstrate evidence of a formal protocol, and did not disclose conflict of interest and funding. Moreover, not all studies reported being approved by an ethics committee and obtaining patient's consent. Participation rates and missing data were also often lacking.
Quality assessment demonstrated that a significant proportion of the included studies do not provide robust evidence. Caution should therefore be exercised when interpreting results. Poor quality scores and high risk of bias question the usefulness of the results to complement RCTs and to provide credible economic evaluations. Unfortunately, this review corroborates concerns about RWE practices. Previously voiced concerns include uncertainty about internal validity, inaccurate recording of health events, missing data, opaque reporting of conduct and results, unmeasured confounding, measurement error, model misspecification, selection bias, and even fraud (3;10;11). As demonstrated by the current results, many of those threats were found to be true. It is nonetheless unfortunate that many studies lack credibility because RWE is a promising complement to RCT findings. Therefore, it is worth pursuing efforts to increase confidence in RWE studies.
The International Society for Pharmacoepidemiology (ISPE) and ISPOR have paved the way to good practices for RWE studies, yet their recommendations are not widely adopted (3;5). In addition, the Patient-Centered Outcomes Research Institute (PCORI) (12;13) and the Agency for Healthcare Research and Quality (AHRQ) (14) published important guidelines for prospective observational studies. Nevertheless, they focus on clinical outcomes. The ISPOR Task Force (Reference Berger, Dreyer, Anderson, Towse, Sedrakyan and Normand15) and the AHRQ (16) also provided guidance on the conduct of prospective observational studies, but specifically for comparative treatment effectiveness. The National Institute for Health and Care Excellence (NICE) also made recommendations on the use of RWD for comparative treatment effectiveness (Reference Bell, Wailoo, Hernandez, Grieve, Faria and Gibson17). The other guidelines for prospective observational research comprise those developed for pharmacovigilance and pharmacoepidemiology methods (18). As for pharmacoeconomic outcomes, the ISPOR Task Force published six reports on good practices for measuring drug costs in cost-effectiveness analyses (18). However, their recommendations mainly address the use and calculation of costs, and not primary data collection. There are other ISPOR Task Force reports adapted to pharmacoeconomic outcomes, but they focus on budget impact analyses, conjoint analyses, and transferability of economic information (18). The other guidelines that have been identified are used in synthesizing evidence rather than in generating evidence. Mostly, they provide guidance on evidence synthesis research methods, analysis methods, and the use of evidence in healthcare decisions. There are also several guidelines on reporting of evidence (18). An important publication from the ISPOR Task Force consists of a questionnaire designed to assess the relevance and credibility of observational studies (Reference Berger, Martin, Husereau, Worley, Allen and Yang19). The questionnaire helps identify strengths, weaknesses and fatal flaws, and was developed to assist decision makers in evaluating studies. Although highly relevant to assess the quality of published studies, it was not exactly designed for investigators.
To our knowledge, there are no guidelines focusing on methodologies to collect pharmacoeconomic outcomes in a real-world setting specifically. The results of this review clearly highlighted that RWE is commonly generated using poor quality methodology, at least in T2DM. Guidelines specifically adapted to pharmacoeconomic outcomes may help future investigators in performing credible RWE studies for reimbursement decision making. Adapting procedural practices to T2DM, or prevalent chronic diseases, would increase relevance and usefulness for researchers.
This systematic review was conducted according to a predefined protocol and was performed by two independent reviewers. Because a transparent methodology was used, this review could be replicated, allowing the assessment of RWE practices in other clinical settings. As part of the limitations, gray literature was not screened. Moreover, a short timeframe was used for study selection. A longer observation period might have retrieved different methodologies than those of this review. Nevertheless, extending the study period would only have allowed for the retrieval of even more heterogeneous methodologies, thus worsening the lack of credibility and uniformity already demonstrated in the current review. Because this research field has greatly evolved through the years, limiting the study selection to the most recent years has permitted to provide a current overview. In addition, only data and methodologies related to pharmacoeconomic outcomes were extracted, even if clinical outcomes were reported in the study. As per eligibility criteria, findings of this review only reflect practices in primary data collection. This review was limited to T2DM for pointing out the shortcomings in collecting RWD. Any other common chronic diseases might have also been chosen and the same conclusions would have been expected. Indeed, methodological practices used for collecting pharmacoeconomic outcomes were not specific to T2DM, and would probably not differ in studies assessing similar conditions. Still, it would be interesting to analyze RWE studies in other therapeutic fields to confirm this hypothesis.
Conclusion
RWD are complementary evidence in reimbursement decision making, yet the conduct of RWE studies has not been standardized. Findings of this literature review corroborate the lack of credibility and transparency in RWE methodologies. There is a need for good procedural practices that can increase confidence in RWE studies. Guidelines adapted to RWE studies collecting pharmacoeconomic data in T2DM could help future investigators in performing good quality research.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462320000392.
Acknowledgments
This research received no specific funding from any agency, commercial, or not-for-profit sectors.
Conflict of Interest
All authors have nothing to disclose.
