Pubic funders are increasingly basing their decisions about whether to fund new health technologies on reviews of the available evidence (Reference Morgan, McMahon and Mitton16). One question that quickly arises in attempting to base coverage decisions on clinical and economic evidence is how to handle situations in which there is insufficient evidence. For many promising technologies, adequate study simply has not been done for funders to confidently say whether they are a good investment of healthcare resources or for which specific patients these interventions should be targeted. Rather than simply granting or denying access in these cases, funders in several countries have been exploring a range of alternatives that allow for coverage or limited access, but only as part of a further evaluation of the technology so that a more definitive coverage decision can be made. NICE calls its ruling for this type of restricted coverage only-in-research (OIR) (Reference Chalkidou, Hoy and Littlejohns3). Others variants on this idea include coverage with evidence development (Reference Hutton, Trueman and Henshall10) and field evaluations (8).
There seems to be growing acceptance of these types of arrangements among those involved in making coverage and investment decisions around new health technologies (Reference Dhalla, Garner, Chalkidou and Littlejohns7). It is less clear what the general public think about delaying access to potentially effective technologies to generate sufficient evidence to justify a decision to invest or not invest. The public's perspective on these arrangements is critically important due to the fact that they involve several value choices, have a direct impact on patients’ access to care and affect how public tax dollars are spent (Reference Miller and Pearson15). Public support is also essential if the continued and expanded use of OIR is to remain politically viable. Because policy makers and healthcare professionals cannot legitimately represent the views of lay citizens, direct public engagement is required (Reference Martin, Abelson and Singer12).
Public engagement on this topic to date has focused on the guidelines agencies should use for selecting technologies for these types of arrangements. In 2007, the Citizens Council of NICE debated the question of when should NICE recommend that an intervention be used OIR (17). Similarly, the Centers for Medicare and Medicaid Services (US) received public input on its guidelines for selecting which technology should be covered only under the proviso that further evidence will be collected (Reference Tunis and Pearson20). The lay public has rarely been asked, however, for its opinion on a real case in which patients are being given only limited access to a new technology until the completion of further research. A concrete example of OIR likely provides a more stringent and realistic test of the public's view of these arrangements. A specific case gives the public details which may influence their support, including about the technology involved, the disease it is used to treat, the restrictions being placed on access, the costs involved, the nature of the further research projects being proposed, and the level of evidence which the funders want before providing wider access. A real example also makes more apparent many of the practical difficulties in conducting rigorous evaluative research. The public may support the idea of OIR in general, but does it still support the idea of restricting access until more evidence is available when it is clearer which specific patients are being affected, what technology is being restricted, and how long it may take to get the relevant evidence?
THE CASE: THE EVALUATION OF PET SCANNING IN ONTARIO, CANADA
The case we presented to members of the public was the province of Ontario's decision to restrict access to positron emission tomography (PET) scans until better quality evidence about the effectiveness and cost-effectiveness of the technology becomes available. The issue of access to PET scans in the province began in 1999 when the Ontario Association of Nuclear Medicine urged the provincial government to introduce the technology for oncology, cardiology, and neurology patients (5). In response, the government commissioned a health technology assessment, which concluded, in 2001, that there was only poor quality evidence indicating that PET scanning has a positive impact upon patient outcomes (Reference Laupacis, Alter and Mamdani11). The provincial government decided that it would limit access to PET scanners until better evidence of benefit (i.e., methodologically rigorous clinical trials showing improvement in clinically important outcomes) was available. In addition, to support the creation of the relevant evidence, the government committed to funding five research trials. These trails were designed to evaluate the clinical utility of PET scanners for patients with (i) head and neck cancer, (ii) breast cancer, (iii) early stage lung cancer (Reference Maziak, Darling and Inculet13), (iv) stage III (locally advanced) lung cancer, and (v) colorectal cancer with metastases to the liver. The comprehensive scale of this research program was partly based on the hope that the information garnered through these trials could contribute to the global evidence base for PET technologies. The research focused on cancer, because it is the disease for which the technology showed the most promise. Since 2001, most cancer patients in Ontario who wanted to undergo a PET scan could only do so by enrolling in the appropriate clinical trial.
There are several issues that make this case a particularly challenging test of the public's support for OIR. The Ontario approach to PET scanning has been controversial, with critics and stakeholder groups asserting that waiting for the completion of more clinical trials unreasonably restricts access for patients who would benefit from the technology. The fact that many Western countries and other Canadian provinces already widely fund PET scans for their populations raises issues relating to inequitable access. Within the Canadian healthcare system, these interprovincial variations in access to care, while common, are very politically sensitive (Reference Chafe2;Reference Gregoire, MacNeil and Skilton9). The fact that the restricted access related to the treatment of cancer, a life-threatening disease the public usually have a strong emotional reaction toward (Reference Sontag18), may raise other difficult issues for the lay public. There were several issues which slowed patient recruitment in, and ultimately the completion of, the clinical trials the government funded. Although PET technology costs significantly more than other modalities in both its capital and operating costs, other diagnostic technologies had been adopted by the province without requiring the same level of review. Finally, because this technology is still evolving, it is uncertain that when the trials are finished, they will still be relevant to the investment decision the government will face. All of these issues were raised with the members of lay public during their consideration of the case.
METHODS
To gauge the lay public's response, we devoted a session of the Toronto Health Policy Citizens' Council to examining this specific example of OIR. The Citizens' Council, which has been established as part of a wider research project examining public engagement in health policy, consists of twenty-six individuals who were randomly selected by an independent third-party (19). We did not make an explicit attempt to ensure that the Council was “representative” of Ontarians. Potential members, however, were screened and selected to ensure that there was a wide diversity of ages, backgrounds and viewpoints among Council members. The purpose of the council is to have ordinary citizens deliberate and comment on important, value-sensitive issues in health care. Our goal in our selection of participants was to ensure robust deliberation reflecting numerous lay perspectives. None of the council members are employed in health-related occupations. Council members are remunerated for participating. Research ethics approval for the entire research project has been granted by the Research Ethics Board (REB) of St. Michael's Hospital (Toronto).
On January 26 and 27, 2008, the council members were asked to deliberate on the following two questions: (i) In your opinion, is the Ontario Ministry of Health and Long-Term Care's (MOHLTC) approach to funding PET scanning a reasonable one and in the best interests of the citizens of Ontario? (ii) What suggestions for modifications do you have, and why?
As with NICE's Citizens Council (Reference Davies, Wetherell and Barnett6), the Toronto Health Policy Citizens' Council is set up as a deliberative process, in which the council members interact with experts and with each other to arrive at a set of recommendations. The advantage of this type of process is that it allows citizens to become educated about a topic, to deliberate on it for a period of time, and to come to a more informed and considered position (4). In addition to the presentations by experts, the research team ensured that content experts, with opposing view on the Ontario government's approach, were available throughout the 2-day council meeting to answer any specific technical questions council members had about the technology or about the government's approach.
The council meeting began with presentations from four experts, who explained the technology, its relation to other diagnostic imaging modalities, the possible benefits it may have for cancer patients, the possible use of PET scans for diseases other than cancer, and the costs associated with the technology. The experts reviewed the history of the Ontario government's position on funding PET scans, in particular the amount and quality of evidence available about the clinical impact of PET scanning, and provided an update on the status of five clinical trials currently under way in the province. None of the trials were completed at the time of the council meeting. One expert presented arguments for and another provided arguments against the approach Ontario has taken regarding PET scanning. At the start of the second day, another expert who was directly involved in the government's decision to initiate the research trials presented and answered questions from the council. The council members were then divided into three small groups to discuss the two questions presented to them. Each group was moderated by a member of the research team, whose role was to encourage every member's voice to be heard and to answer any factual questions which arose over the course of the deliberations, while abstaining from injecting personal opinions about the topic. After 2 hours of small group discussion, the entire council met to hear the conclusions reached by the three small groups and further discuss the topic. Based on this large group discussion, the council members then agreed on a joint conclusion about the topic, including identifying points about which consensus was not reached.
To allow another opportunity for the citizens to express their views on this topic, a questionnaire was sent out several days after the council meeting. Once the results of the survey were available, a draft of the report on the council meeting and the survey results were then sent to all council members to review and to provide suggested edits. During the next council meeting, the council, as a group, approved the revised final report as being an accurate reflection of the findings of the previous council meeting (Reference Merali, Laupacis and Levinson14). The entire process is outlined in Table 1.
Process Overview
RESULTS
Most council members had no prior knowledge of Ontario's approach to funding PET scans or of the technology itself. Many of the preliminary questions following the expert presentations focused on practical issues relating to PET scanning, such as the cost, the lifespan of the machines, likely levels of patient usage and expected clinical impact of the technology. Given Ontario's geography size (approximately equal to the combined size of France and Germany), there were questions concerning the location of scanning facilities across the province both during the clinical trial stage and their likely distribution if funding is ultimately approved. There were also several questions about the government-funded clinical trials and other possible sources of evidence which could clarify the effectiveness of the technology. As the council session progressed, participants clearly appeared to have gained a sufficient understanding of the technology and the rationale for the government's approach to comfortably debate the merits of limiting access until further evidence was available in this case. This understanding was demonstrated by both the sophistication of the questions the council members asked the experts and the comfort with which council members debated the topic, especially by the second day of the council meeting.
Council members were quite concerned with the delays in completing the clinical trials, which one expert reported had been a 5- to 6-year process. They believed that the experiences of other jurisdictions, which use the technology more widely, should be drawn upon to provide evidence of whether the technology is effective and for which patient populations. Council members also debated whether the government might be using criteria that are too restrictive for assessing the technology. For example, one council member believed that “it is better to err on the side of caution than to wait a bit too long, and it is too late [for some patients to benefit].” Other council members believed, however, that both the timelines and the rigor of the trials were appropriate given that the impact of PET technology on patient outcomes is still uncertain and that the resources needed to fund it as a routine service would come at the expense of providing other health services. As one council member said, “I'm okay with the slowness because I don't want them to just throw money at it [PET technology] if it's not proven.”
In each of the small group sessions, views on the government's approach initially ranged from strong support to indignation that access to PET scans was being restricted. The citizens debated whether PET should be the “line in the sand” from which the government starts asking harder questions about new health technologies and whether this technology was “chasing a disease” rather then truly needed to improve the lives of patients. Recognizing that complete certainty about a technology's effectiveness is an unreachable ideal, the council members spent a good deal of time debating the question of when is there enough evidence so that the government should adopt a technology. Council members who were supportive of the government's approach often referred to the uncertainty around the technology, while those who were more critical of the government's position seemed to accept that the technology offered benefits to patients, a view expressed by some of the expert presenters.
Two of the small groups ultimately came to a consensus that they generally agreed with the government's approach to funding PET scanning, but that progress of the government-funded trials was too slow. Some council members, however, continued to disagree with the government's overall approach. These citizens believed that these machines are used in life-saving situations and should be made available. As one council member said, “Why can't Ontario value human life a bit more rather than worrying about the funding formula? The bottom line is human lives are being saved.” During the expert presentations, the council members were told that several hospitals in India had PET scans and that patients in the province of Quebec (Canada) currently received thirty times the number of PET scans as patients in Ontario. PET scans are also available in some of the other Canadian provinces. In response, some of the dissenting council members pointed out that PET scans are already widely available in many developing countries and that a province as affluent as Ontario should also be able to provide PET scans for its citizens. Others countered that it was good to see that government was taking a more long-term view relating to healthcare spending, even though it may not serve politicians to take this approach. Following the large group discussion, the majority of the council agreed with the position that the government's approach to funding PET scanning was reasonable and acceptable, but that the process of getting further evidence was too slow. A minority of council members disagreed that the government's approach was in the best interests of patients and the public, but rather held that the government was unnecessarily and unfairly limiting access to care that may help people in need.
Most council members (twenty-one of the twenty-six) responded to the follow-up survey. The survey used a rating scale for each recommendation (1 – Strongly disagree; 2 – Disagree; 3 – Agree; 4 – Strongly Agree). Results closely mirrored the opinions voiced by the group during the council meeting. The recommendation that “the usefulness of PET through research should be assessed before widespread use is implemented” received an average rating of 3.1, with a standard deviation of ± 0.9. The council members also expressed the view that the overall council process worked well as a means of gauging the public's perspective on the Ontario government's approach to funding access to PET scanning.
DISCUSSION
OIR and similar arrangements have been portrayed as balancing funders’ needs for “rigorous evaluation” of new medical technologies against “the public's demand for rapid access to new services” (Reference Carino, Sheingold and Tunis1). Our engagement with the public about a specific case of OIR shows that the lay public can be generally supportive of thoroughly evaluating new technologies, even when it is presented with a challenging example of OIR. In fact, none of the citizens rejected the basic idea that the overall benefits of new health technologies should be reasonably assessed before their widespread use, even if this further evaluation delays access to technologies that may later be shown to be effective (Reference Tunis and Pearson20). Concerns were expressed, however, about several aspects of the case presented to the Citizens' Council, including whether the restrictions unfairly limit access to beneficial care and the length of time it is taking the government to obtain further evidence.
This Citizens Council is one of the first in-depth engagements of the lay public to measure its reaction to a specific case of OIR. It is important that when gauging the public's response to OIR that it is presented with actual detailed examples. The members of Toronto Health Policy Citizens' Council generally agreed with the notion underlying OIR, but were divided in what was the most reasonable way for the government to act regarding funding PET scanning, reflecting differences in the values and judgments people applied when confronted with the details of the case. The council's discussion illustrates that contextual factors affect the public's support for these arrangements. For example, many of the people who believed that the government's approach was unfair were troubled by the fact that the technology was widely available in other jurisdictions. Those opposed to the government position also appealed to the fact that the technology is used to support the early diagnosis of cancer, a life-threatening disease. Agencies responsible for selecting technologies for further evaluation need to consider whether there are factors related to the technology which may adversely influence the public's support or conflict with widely held the public values. The expanded use of deliberative public engagement has been shown in this study to be an effective means of garnering the lay public's opinions of these arrangements, even for technologies members of the lay public are initially unfamiliar with. Deliberative public dialogue can be used to improve these arrangements and help identify cases the general public is most likely to support.
The study has several limitations. The research team only asked the public about one example of OIR and with one group of citizens. This meeting was also the first session of the Toronto Health Policy Citizens' Council. Further public engagement is needed both regarding the idea of OIR generally and on specific cases of OIR. Areas to which public dialogue can likely make a positive contribution is in selecting which technologies should be candidates for OIR, reviewing the OIR choices of funders, making recommendations about how prospective cases are structured, deciding whether public funds should be used to support recommended further research, or recommending what amount of further evidence should be required before a coverage decision is made. Given the ethical issues and judgments involved in restricting access to potentially innovative health technologies, the public have a useful and legitimately role to play in structuring these arrangements. Agencies responsible for making evidence-based coverage decisions are thus advised to establish strategies for better engaging the lay public to advise them about the reasonableness of prospective cases of OIR.
CONTACT INFORMATION
Roger Chafe, PhD (roger.chafe@cancercare.on.ca), Postdoctoral Fellow, Department of Health Policy, Management, and Evaluation, University of Toronto, 155 College Street, Toronto, Ontario M5T 3M6, Canada; Postdoctoral Fellow, Cancer Services and Policy Research Unit, Cancer Care Ontario, 620 University Avenue, Toronto, Ontario M5G 2L7, Canada
Farhan Merali, BS (farhan_merali@hms.harvard.edu), MD/MBA Student, Department of Health Care Policy, Harvard Medical School, 25 Shattuck Street, Boston, Massachusetts 02115
Andreas Laupacis, MD, MSc, FRCPC (laupacisa@smh.toronto.on.ca), Professor, Medicine and Health Policy Management and Evaluation, University of Toronto, 155 College Street, Toronto, Ontario M5T 3M6, Canada; Executive Director, Keenan Research Centre, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 30 Bond Street, Toronto, Ontario M5B 1W8, Canada
Wendy Levinson, MD (wendy.levinson@utoronto.ca), Chair, Medicine, University of Toronto, 190 Elizabeth Street, 8–305 RFE, Toronto, Ontario M5G 2C4, Canada
Douglas K. Martin, PhD (douglas.martin@utoronto.ca), Associate Professor, Health Policy, management and Evaluation and the Joint Centre for Bioethics, University of Toronto, 88 College Street, Toronto, Ontario M5G 1L4, Canada
