Canada has a long history of the use of clinical evidence to support healthcare decision making, both at an individual patient level with the promotion of evidence-based medicine (Reference Sackett1) (EBM) as well as its counterpart in population-based recommendations and decisions. The latter includes long-standing National programs providing evidence-based recommendations for immunization (Reference Desai, lsmail, Lerch, Warshawsky and Gemmill2) and preventive services (3), and more recently, provincially coordinated evidence-based recommendations for community-based and oncology medicines through the Canadian Agency for Drugs and Technologies in Health (CADTH) (Reference Battista, Côté, Hodge and Husereau4). The introduction of drugs has also been informed by a patchwork of provincial and institutional health technology assessment (HTA) and pharmacy and therapeutics committees (Reference Martin, Polisena, Dendukuri, Rhainds and Sampietro-Colom5).
These evidence-based programs relied heavily on an approach inspired by EBM, ascertaining potential clinical benefit through the “conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients” and applying clinical judgment to the “best available external clinical evidence from systematic research” (Reference Sackett, Rosenberg, Gray, Haynes and Richardson6). With an emphasis on perceived clinical benefits (Reference Rocchi, Miller, Hopkins and Goeree7), rather than economic impact in it drug reimbursement frameworks, Canadian drug policymaking could be argued to more closely align with French and German HTA processes (Reference Panteli, Arickx and Cleemput8), as opposed to the approach taken in the United Kingdom. It may also be a key explanatory factor in Canada's high production rate of systematic reviews (first overall when adjusted for population size) (9;10) and can be observed in studies of the impact of economic evaluation in Canadian reimbursement policy (7;11).
Along with these developments, approaches for obtaining reliable information about costs and service use have also evolved as a means of evaluating the quality and economic impact of new services. To accomplish this, and like other jurisdictions, inventories of health services and associated standard costs were developed alongside the increasing use of administrative data in health services research in the 1990s (12;13). At the same time, the potential for routinely collected data to be used for HTA was also proposed (Reference Fisher, Malenka, Wennberg and Roos14).The 1990s also saw a first-in-kind risk-sharing agreement for the use of finasteride in patients with benign prostatic hypertrophy, where one Canadian province (Saskatchewan) monitored performance through capturing administrative data and was refunded the drug cost where the drug failed to provide a benefit (Reference Klemp, Frønsdal and Facey15). Given these early experiences and the availability of substantive province-wide data holdings in relatively few sites, it may be surprising to some that the use of Canadian healthcare data to manage the entry of new drugs by payers did not become commonplace by the early 2000s (Reference Lucyk, Lu, Sajobi and Quan16).
Given improvements in data holdings and analytic capacity in Canada (Reference Morin and Flegel17) and a renewed interest by the global pharmaceutical industry in the development and use of data to inform decision making outside of clinical trials, that is, “real-world” data, predominantly healthcare administrative data and registries (although how real-world data is defined is a matter of debate and an issue discussed below), the purpose of this study is to reflect on what opportunities may be available to produce and use real-world evidence and how it might be received by reimbursement authorities and funding agencies. This includes perceptions of the value of real-world evidence in reimbursement and decision making, barriers to its optimal use in pricing and reimbursement, current initiatives that may lead to its increased use, and what role the pharmaceutical industry may play in this. To better understand this, four multi-stakeholder discussions were held between 2014 and 2017. This study highlights and synthesizes the findings of these discussions. Its intent is to inform a broader international discussion on optimizing the use of real-world evidence for pricing and reimbursement decisions.
Methods
Multi-stakeholder meetings were conducted under the Chatham House Rule (18) to discuss areas of common concern. To capture perceptions of opportunities and challenges, current approaches and future developments in the use of real-world evidence in Canada, a convenience sample of participants were identified according to background roles (i.e., academic/public/private sector) and geography (i.e., across Canada) and invited to participate in roundtable discussions.
Four separate roundtable discussions dedicated to the use of real-world data were held (Apr/14; Nov/14; Dec/16; Jun/17) with ninety-one participants representing academics/care providers (n = 23), pharmaceutical companies (n = 17), public sector including data custodians (n = 12), health technology assessment bodies and evidence experts (n = 11), payers (n = 10), private sector consultants (n = 10), patient representatives (n = 1), public research funders and coordinators (n = 6), and medical device (n = 1) companies. Three of these four meetings were conducted face face-to to-face for 6 to 8 hours. Another half-day meeting was held by Web conference, with participants communicating verbally but unable to see each other. Meetings generally followed a format of presentation and moderated discussion. Further details regarding the approach and findings of these meetings can be found on the internet (19–21).
At each meeting, two researchers (D.H. and E.N.) took detailed notes of the discussions in addition to discussions that were also audiotaped and transcribed. Those notes were compiled and sent to roundtable participants for validation (member checking). They were then analyzed qualitatively by informal identification of concepts by two researchers (D.H. and E.N.), which were categorized into themes. A “STEP” (Social, Technological, Economic, Political) analysis approach was used to classify themes into policy domains. Draft reports were then made available to roundtable members to ensure accuracy.
Findings
Key themes identified through discussion are identified in Table 1.
Emerging Themes from Discussions of Optimizing RWD in Canada
Understanding the Concept of “Real-World” Data
A common theme emerging in discussions about the value of real-world data for healthcare decision making was the need to clarify what exactly is meant by “real-world” data. Payers and HTA representatives often equated evidence from real-world data with “observational studies”; in contrast, data scientists were surprised to see distinctions made between evidence from randomized controlled trials (RCTs) versus observational studies or other sources of evidence for policymaking. Participants noted that other Canadian real-world data initiatives have used the terms secondary use data (Reference Scott, Rigby and Ammenwerth22), research data, and health and health-related data (23). Throughout these conversations, international definitions from IMS and the International Society for Pharmaceutical Outcomes Research (ISPOR) (Reference Garrison, Neumann, Erickson, Marshall and Mullins24) were provided to inform discussions. For most, the data source was less relevant than how data was interpreted and analyzed. Definitions of real-world data were generally not challenged by participants, only the issue of the lack of a standard definition was identified.
Different Perspectives on What Real-World Data Can Offer
While there was general consensus that better use of real-world data in Canada was desirable, there were different perspectives on how this could be best achieved. While industry stakeholders were interested in its use in adopting new technologies through pricing and reimbursement mechanisms (e.g., coverage or access with evidence development) (Reference McCabe, Stafinski, Edlin and Menon25), health system leaders and payers saw its real value in gauging the overall quality and appropriateness of health care, which may or may not involve medicines. Academic stakeholders saw real-world evidence as an efficient means to reduce the uncertainty of decisions by demonstrating the effectiveness of new medicines, especially through the power of database linkage (Reference Morin and Flegel17).They also viewed real-world data as a means to provide important information to payers at a reduced research cost (Reference Wodchis, Bushmeneva, Nikitovic and McKillop26). Administrators, on the other hand, viewed it as a means of monitoring and improving the appropriate prescribing of proven interventions (e.g., drugs with minimally uncertain health effects) and validating original claims of effectiveness. It was also seen by administrators as a promising means of supporting disinvestment decisions for low-value care (Reference Elshaug, Moss and Littlejohns27).
In this context of pharmaceutical policy, it seems the perceived primary purpose of real-world data in Canada is as a means to assess and improve appropriate prescribing of existing agents and reducing waste or improving patient care. This was referred to as a “harvestable offset” by healthcare leaders, who suggested real-world data is most useful when its results help manage budgets or improve health system efficiency. The promise of spillover effects outside of the drug budget programs were considered harder to justify by drug program leaders. In some cases, as with the Alberta government Retina Anti-Vascular Endothelial Growth Factor Program for Intraocular Disease (RAPID) program, registration and ongoing monitoring of patients can additionally serve to constrain excessive expenditure growth while ensuring quality of care (28). While there were perceptions of additional value from real-world data, there were also very few cases in Canada provided by participants where they believed the value of real-world data had been definitively shown. Canadian real-world data platforms discussed at these roundtables and that may be relevant to pricing and reimbursement and their overarching objectives are highlighted in Table 2.
Public Sector-Led Canadian Real-World Evidence Platforms, as of July 2018
CIHR, Canadian Institutes of Health Research; PHAC, Public Health Agency of Canada; BC, British Columbia.
There was also broad agreement that real-world data may be more useful when there is a lack of trial data, such as in studies of patients with rare conditions, or when important data, such as health-related quality of life or long-term outcomes affected by disease progression, are not captured in clinical trials. Having populations excluded from RCTs, such as the very young or very old, or those with comorbidity, poor health performance status measures, or poor renal function, and clinical trial comparators that are not locally relevant was also perceived as creating a need.
Perceived Barriers to Increasing the Use of Real-World Data
Patchwork and Varying Access to Data Holdings
It is clear from these discussions that there are real disparities across Canadian provinces in terms of access to relevant data. Access is directly related to data stewardship and governance models (see below) that range from poorly developed in some provinces, where publicly funded researchers have difficulty conducting studies with real-world data, to well-developed systems, where analysis of de-identified data is available to private sector researchers. An example of the latter is the Institute for Clinical Evaluative Sciences (ICES), a government-mandated research institute that has progressed from single center data holdings limited to a few ICES-designated scientists, to a model involving multiple centers and broader access to data. In 2014, ICES introduced the Data and Analytic Services program (29), intended to provide analyses to non-ICES scientists including private sector researchers.
Other provinces in Canada with well-developed information governance, such as Alberta and Newfoundland, appear to be following in this direction, and have loosened restrictions to access to public sector researchers while considering how to liberate data for private sector research. They have also invited collaborations with the pharmaceutical industry to support research and innovation in strategic clinical areas (30).
A related theme emerging from these discussions was timely access to data. Provinces with more established data governance models have recognized the need for more timely data and have shortened timelines for the availability of routinely linked and de-identified data from 1 year to several weeks. Timing of data was still seen as a significant challenge related to its relevance and large-scale pan-Canadian initiatives (such as the Canadian Network for Observational Drug Effect Studies, CNODES, described below) were seen as facilitators for shortening the cycle length of analyzable data. One researcher commented that, although approval times for data, including research ethics review, had been considerably shortened in his province (i.e., to weeks instead of months), the resources and time dedicated to cleaning and de-identifying data so that it could be shared was still quite long (i.e., months).
Need for Improved Data Governance
Another emerging theme was the need for appropriate governance to be in place to ensure appropriate consent, privacy and data protection. An interesting aspect of access to data in Canada is the roles of the research ethics boards (REB). While data custodians must approve requests for data, their use in research must also pass a separate process of research ethics board approval. Provinces with more developed processes have moved to separate REB processes for data access requests, and may also recognize REB approvals from other provinces. Academics and private sector researchers noted, however, that approval of request for data was highly dependent on the wording of requests and whether viewed as a research, health system management or quality improvement initiative. Turnaround time for data from hospitals and other private institutions used strictly for quality improvement or performance evaluation was perceived as much faster. Yet many believed these distinctions to be somewhat artificial, in an era of applied research and implementation science.
Similarly, the wording and timing of issuing consent for patients, especially in the context of pragmatic trials, was viewed as a means of more efficiently producing real-world data. This has in turn raised questions about changing models of informed consent (Reference Kalkman, van Thiel, Zuidgeest and Goetz31), such as the use of an integrated consent model (Reference Kim and Miller32), as a means to more efficiently enroll patients in the context of care. Nonetheless, the long-held concept (Reference Campbell33) of a learning health system and integrating research into care is a common vision among policy makers and evidence experts in Canada.
Finally, privacy legislation related to data sharing is enacted at provincial levels and can create meaningful differences across provinces. Discussion from participants across provinces revealed a lack of understanding of these cross-provincial differences, which in some cases translated to meaningful differences in the capacity to carry out large-scale research.
Need for Interoperability Standards of Real-World Data
The quality of existing data along with opportunities to improve quality and comparability of data sources was a significant theme. Canada has a small population spread over a wide area and governed by numerous individual systems of health care, making standards an important factor for the meaningful pooling and comparison of data. Notable was that not only did healthcare systems investments in governance mechanisms vary, but some Canadian provinces lack data custodians that collect and pool information suitable for research and are limited to analyses within individual health service programs by administrators.
There are also no common national frameworks for capturing information in primary care. For example, only two-thirds of primary care physicians report using electronic medical record (EMR) systems in Canada (41), and there are no obligations for physicians to collect these data or make them available for research. Further to this, there are no enforced standards or common data elements for electronic medical records leading to challenges in pooling or comparison. There are similarly no enforced standards for registry development. Participants noted that improved data standards including standardized EMR information would not only improve the use of real-world data for policy making, but also facilitate its use in patient care.
Hospitals in Canada are similarly private, not-for-profit institutions and may keep additional data holdings not related to provincial financing that they are not obligated to share. Even healthcare administrative data among very similar publicly owned programs of care, such as public drug insurance for seniors, may harbor subtle differences. This has required retrospective harmonization and the use of federated data analysis or pooled analysis using mapping algorithms to allow for larger-scale analyses. An example of this is the Canadian Network for Observational Drug Effect Studies (CNODES), a pan-Canadian initiative that relies on academic researchers from health jurisdictions to develop algorithms and protocols tailored to their individual settings to provide answers related to drug safety (Reference Roos, Roos and Freemantle52).
Need for Standards for Interpreting Real-World Study Findings
A consistent theme by evidence experts and payers was a lack of confidence in the results of real-world studies, due to concerns about the effect of study designs and differences in how data are captured and analyzed. A rigorous and transparent approach to data capture and analysis was identified as a necessary factor to increase the confidence of findings and increase the use of real-world data by decision makers. It also highlighted the need for pan-Canadian interoperability standards (e.g., common data elements), or guidance for those using patient registries and electronic medical and health records as a data source for real-world data. One initiative identified was work toward adopting common data elements in the area of cancer, similar to the IS National Cancer Institute, a collaboration between the Canadian Partnership Against Cancer and the Canadian Institute of Health Information, as a means of improving the use of real-world data (34). Increasing the use of real-world data has been identified as a strategic priority by provincial cancer care programs.
Some participants observed that even well-conducted and reported observational studies can lead to more questions than answers; pragmatic clinical trials were often identified and perceived as more useful, with the acknowledgement that these studies may require more significant resources and capacity to carry out. One particularly promising development was the emergence of pragmatic trials linked to existing administrative data, as a means of identifying patients or analyzing outcomes. The Ontario Strategy for Patient Oriented Research Support Unit (OSSU) has been particularly instrumental in this regard, providing resources and training to researchers seeking to conduct pragmatic trials, supporting studies to test the linkage of trial and administrative data, moving the provincial healthcare system toward a learning healthcare system and “implementation laboratory” to better study complex interventions of care that may involve medicines (Reference Ivers and Grimshaw35).
Variation of Types of Available Data in Canada's Federated Health System
Another barrier to use of real-world data identified is the type and variation of data holdings available across the country. Because health care in Canada is delivered provincially rather than nationally, data holdings occur in individual provinces and according to province-specific healthcare structures. The Canada Health Act, legislation that promotes consistency in delivery of health care across provinces, mandates that provinces provide public insurance for acute care and physician encounters, and this has led to more abundant and consistent administrative data sources for these types of patient encounters.
For other types of health data, such as laboratory test results or outpatient medicines, the delivery of care and data collected are more heterogeneous. For example, participants observed that data on laboratory services is only centrally collected in over half (n = 6/10) of provinces; similarly, universal pharmacare is available to seniors in all provinces but available to children in only two of ten provinces making cross-Canadian comparisons that require these data difficult. Public insurance formularies differ as well. An added challenge is that non-Nationally mandated services are also widely available through employer-sponsored private insurance programs, making analysis related to some demographics (i.e., working-age) difficult.
Need for Specialized Human Resources
The need for resources to both supply and analyze real-world data was also identified. One researcher noted that approvals for access to data occurred quickly through a well-defined process of data stewardship but receiving these data took much longer as few people were dedicated to this activity. Similarly, Canadian drug plan managers have no obligations to conduct research with their existing data holdings and may lack additional funding to hire analysts to create robust analyses. Initiatives such as Ontario's Drug Policy Research Network, which uses publicly-held data to inform provincial drug decisions were born out of separately allocated funds to increase analytic capacity rather than the drug program budget. Similarly, the aforementioned ICES Data and Analytic Services program was born out of National research program intended to improve provincial health research capacity in comparative effectiveness research (36). There was general agreement that pan-Canadian programs such as CNODES, part of the Drug Safety Effectiveness Network, have had a positive impact on increasing intellectual capital in the area of real-world data.
Current and Future Developments
An examination of efforts to promote the use of real-world data by the public sector in Canada reveals a natural evolution from province-based systems to the more recent development of networks, partnerships and coalitions. Willingness to cross provincial boundaries is likely due to more evolved standards for data governance, a comfort level with regional systems, combined with the need to collectively monitor and assess health system performance. One example is the announcement of real-world effectiveness studies of bevacizumab and ibrutinib through collaboration of multiple provincial cancer agencies. These pilot studies were announced after better use of real-world data was identified as a priority for cancer drug reimbursement; they will gauge the feasibility of assessing new therapies using existing provincial cancer data sets (Reference Chan37).
The CADTH has also received increased funding to examine options for the use of real-world data in the management of health technology (38). Although no plans have been announced yet, CADTH has also revised its guidance on the acceptability of real-world evidence in the submission of clinical data, recognizing that RCTs are not always required to address issues of clinical importance. It is anticipated that CADTH will do more in this space in coming years.
A final important development is the development of a plan to provide a distributed data network across Canadian provinces. Originally proposed as the Pan-Canadian Real-world Health Data Network (PRHDN), the SPOR National Data Platform is seen as a portal intended to “act as a single place to submit data requests” using linked data holdings (39). Although the SPOR National Data Platform is still in a developmental phase, it has already created significant partnerships and interest with custodians of publicly held data and research funders.
What Role for Industry?
The various public initiatives, perceptions and priorities for the use of real-world data have not fully described a role for private sector innovators or those interested in private sector development of data holdings. Clear from our discussions is that the public sector in Canada is deeply interested in the collection and analysis of data for policymaking using publicly funded platforms. While there have been burgeoning partnerships to work with industry, they have been largely focused on discovery initiatives rather than applied research or the development of real-world data oriented to policy making, with some exceptions.
Given the current environment, participants were clear that the pharmaceutical industry could have a role if real-world data were used to meet healthcare system priorities and that proposals for the development of real-world evidence platforms or research are targeted at strategic priorities related to the use of real-world data that include: (i) reducing unnecessary or low-value care; (ii) improving access to proven, high-value care; (iii) reducing healthcare system costs; (iv) developing pan-Canadian infrastructure and data platforms; (v) improving data consistency within and across jurisdictions; (vi) improved governance of patient support programs and other privately data holdings; (vii) supporting stakeholder engagement and collaboration; (viii) improved understanding of the value of real-world evidence.
Participants also observed that whether these priorities can align with the business objectives of innovators remains an open question and challenge. Certainly, those paying for new drugs see real-world evidence as a means to improve care and reduce costs.
Discussion
Our discussions on perceptions of the use of real-world evidence in Canada have led to several insights. This includes different perceptions among key stakeholders about what real-world evidence is, what it can offer. Many stakeholders expressed concerns about the quality of data available for gauging drug effectiveness. Certainly, much of the emphasis to date has been on healthcare planning, quality improvement, and monitoring safety. The use of healthcare data as means to measure drug performance is still at a nascent stage. More recent activity in Canada is largely focused on one-off conceptual and pilot projects gauging the effectiveness cancer drugs along with other select therapeutic areas (e.g., mental health and dermatology).
A dominant theme emerging from the discussion was the complexity in accessing, integrating, and analyzing data across Canada's federated health systems. Often a surprise to those outside of Canada is the lack of universal coverage for drugs coupled with provincial authority for delivering health care. This patchwork of 10 provincial health systems with differences in healthcare administration, data structures, and privacy legislation creates challenges in sharing data. It has also led to various network initiatives as well as discussions about common data standards and how provinces can work together. Differences in the size and resources available to these jurisdictions also means more populated provinces, like Ontario, have more advanced data stewardship models and capacity to conduct this research, while other provinces much less so.
While payer, industry and academic stakeholders saw the value of improving the use of real-world data for informing policy decisions, there appeared to be clear differences in what the key priorities should be, and how stakeholders might be involved. Some areas of commonality include using real-world for quality improvement initiatives, as well as healthcare planning. For example, industry currently has no access to provincial insurance claims data for developing population and budget impact analyses submitted to provincial payers. This has led to estimates based on available epidemiologic information or using proxies from private data sets, and has likely led to more uncertainty than if sharing mechanisms for these data were in place.
Another theme emerging from these discussions was a need to use existing healthcare data, ideally coupled with pragmatic trials rather than an emphasis on developing new patient registries. This lack of emphasis on better use of registries appears to be in stark contrast with European approaches, where registries have played a leading role in pricing and reimbursement policy in countries such as Germany and Italy and where more discussion has been observed on how to improve the usefulness of these (40). Canada may also benefit from a better understanding of the utility of registries from work done in Europe on mapping and improving the use of patient registries across jurisdictional boundaries and to inform HTA (41).
We acknowledge the use of real-world evidence may be more applicable, and possibly, more valuable for other aspects of innovation in health care beyond commercialized products. However, these discussions were intended to focus on drug policy and had pharmaceutical industry representation, given the increased resources dedicated to real-world data from that sector. Therefore, some of these themes may not be generalizable to other valuable aspects of innovation beyond commercialized pharmaceutical products, including other commercialized innovation (medical devices and vaccines), changes in clinical procedures, organization of care and information technology and finance structures. However, we believe these insights may be of use to those in other jurisdictions who are at a less-developed stage.
In conclusion, the use of real-world data in Canada to inform pricing and reimbursement decisions is far from routine but nascent and slowly increasing. The federated structure of Canada's health system and the lack of universal public insurance for drugs have led to initiatives that have been focused on the networking of healthcare administrative data across provincial jurisdictional boundaries. There also appears to be a desire to see better use of pragmatic trials linked to these administrative data sets.
There has been little emphasis on gauging the real-world effectiveness of drugs to date, although emerging initiatives, particularly pan-Canadian initiatives in cancer, may make the use of real-world data more commonplace. Health technology assessment agencies (CADTH and INESSS) have not explicitly defined how real-world data should be used, although CADTH has recently received funding to examine and improve its use.
Much of the activity in Canada identified has been publicly funded, often with funds outside of public drug insurance budgets, and with minimal involvement from commercial drug producers. Despite perceived differences in how real-world data might best be used across stakeholders, there still appears to be many areas of common interest including better health system planning and improving the overall quality of healthcare services.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462319000291
Conflicts of interest
Dr. Husereau reports personal fees from Institute of Health Economics, personal fees from Eli Lilly, during the conduct of the study.
