Health technology assessment (HTA) is grounded in rigorous scientific process, focused on quantitative methods to determine clinical, and sometimes cost, effectiveness. As it is intended to inform policy, patients have a democratic right to be involved (Reference Facey, Bertelsen, Wale, Facey, Hansen and Single1). However, misunderstandings about the robustness of research into patient aspects and concerns about the biases that patients may bring, have led to slow uptake of patient involvement processes. Meanwhile HTA bodies that have implemented patient involvement processes are keen to learn from others and understand what works best and why, so that they can develop processes that suit their own context. However, few HTA bodies have published evaluations of patient involvement and been explicit about the outcome or impact of involvement (Reference Gagnon, Dipankui, DeJean, Facey, Hansen and Single2–Reference Abelson J and Gauvin F-P4).
The multidisciplinary and deliberative nature of HTA, in which different sources of evidence and expert opinion must work together to address a problem, means that identifying a discrete part of an HTA process, or cause for an impact, may not be simple. Along with other expert input, patient involvement is intended to inform all the elements of an HTA from shaping research questions and informing cost-effectiveness models to communicating the advice or recommendations. As a result, the integration of patient views in recommendations is considered as a measure of impact (5), while an increase in recommendations to reimburse or fund the cost of medicines is not (Reference Berglas, Jutai and MacKean6).
This study aims to promote further discussion about the ways in which patient involvement can impact HTAs, studying particular cases in-depth, using stories told by people who lead the practice in HTA bodies. It uses the HTA international (HTAi) definition of patient involvement in HTA and reflects on (i) patient participation in HTAs, usually taking the form of patient input, and (ii) research into patient aspects, which can be described as patient-based evidence (Reference Facey, Bertelsen, Wale, Facey, Hansen and Single1). Specifically, these stories were told in response to the question of: ‘What difference is patient involvement making to your HTAs?’ and their intent was to share learnings with other HTA bodies and all HTA stakeholders.
Methods
At HTAi's 2017 Annual Meeting, a 1-day interdisciplinary workshop entitled Patient Involvement in HTA—Why, When, and How? was held to present and discuss the book Patient Involvement in HTA (7). An important element of this was to hear case studies or stories from HTA bodies about the value of patient involvement in their own jurisdiction.
Eight people experienced in patient involvement in HTA bodies were invited to present in one of two panels (Table 1). Four were asked to take part in a panel addressing the difference that research in patient aspects had made to an HTA and four were asked to form a panel focusing on the difference made by patient participation. Presenters were chosen because they represented different HTA bodies included as case studies in the book, although they were not necessarily an author of the case study. Rather than repeating the content of the book, which provided detailed descriptions of organizational patient involvement processes, they were asked to reflect on their personal experiences and observations of patient involvement and were given the additional prompt of “How or when is it adding valuing?”. Subsequently, the presenters provided personal stories or narratives of impact from the perspective of someone employed by an HTA body to make sense of the experience (Reference Greenhalgh, Russell and Swinglehurst8) rather than reports of formal evaluations of impact. In doing so, the stories offered their understanding of the impact of patient involvement on an HTA at the time of telling the story.
Table 1.HTA Bodies and Speakers Who Took Part in the Workshop Panels
Following the workshop, each of the presenters were invited to document these observations in a study. Four of the eight presenters, that is, from the Scottish Health Technologies Group (SHTG), Canadian Agency for Drugs and Technologies in Health (CADTH), England's National Institute for Health and Care Excellence (NICE), and Brazil's National Committee for Health Technology Incorporation (CONITEC), agreed to take part and four either declined or did not respond to the invitation. A draft paper based on notes from the presentations was prepared by one of the authors (A.S.). The four presenters then used this draft as an aid to further reflection on the topic, including discussions with colleagues in their organizations, and redrafted the paper accordingly.
Results
The HTA bodies included in this study use a variety of approaches to patient involvement, which are summarized in Table 2.
Table 2.Summary of Organizations and Their Processes
Patient-Based Evidence for Antimicrobial Wound Dressings (SHTG)
SHTG, part of Healthcare Improvement Scotland, assesses nonmedicine health technologies, such as devices, diagnostics and changes to treatment pathways, predominantly in rapid HTAs of clinical and cost-effectiveness. It also undertakes some full HTAs that have time scales and resources that allow it to follow the Danish model of HTA (Reference Kristensen and Sigmund9) that includes thorough evaluation of patient aspects and organizational issues alongside clinical and cost-effectiveness.
Patient involvement in HTAs of nonmedicine technologies can be challenging. Often there is no clear patient group to participate in these HTAs and patients may be unaware of the specific intervention used. However, patients’ needs, preferences, and experiences can be essential to ensure nonmedicine health technologies are administered optimally.
SHTG's HTA of antimicrobial wound dressings in patients with chronic leg ulcers (10) demonstrates the way in which patient-based evidence can provide essential evidence to formulate conclusions and develop relevant advice. SHTG found wide variability in the type of antimicrobial wound dressings used in each health board in Scotland, and uncertainty in the clinical and cost-effectiveness of these dressings. To identify sufficient literature relating to patient perspectives and values, SHTG iteratively developed a literature search (Reference Hielkema and Wessels11), adding terminology to capture people living with chronic wounds and any type of wound dressings.
Additionally, SHTG conducted one focus group (n = 8) and six telephone interviews with people in Scotland. Thirteen of the participants had experienced chronic wound treatment with antimicrobial wound dressings and one participant was a relative of a person who had experienced the treatment. Participants were identified and recruited by a practicing tissue viability nurse in a National Health Service board following ethical approval for the research. Qualitative evidence synthesis (QES) was then used to develop a comprehensive patient aspects section for the HTA report (10). This section identified patient communication and education issues, as well as outcomes of importance to patients. For example, while wound healing was the most important outcome to patients, controlling the signs and symptoms, such as odor, exudate, and pain, and preventing infection and wound deterioration were also important outcomes.
This robust evidence base enabled patients’ perspectives and experiences to guide recommendations with three of the recommendations incorporating the needs and values identified in the QES. This additional evidence source also enabled HTA researchers to have sufficient evidence to develop consensus guidance and make recommendations (10).
The impact of patient involvement continued beyond publication of the recommendations as the patient-based evidence guided a short life implementation working group, the creation of a patient version of the HTA report (12), and guidance for Scotland's National Health Service (NHS Scotland) and staff training materials. While such an outcome is persuasive of the impact and potential value of patient involvement in HTA, it also highlights the urgent need to develop this methodology so that it can be used in more rapid HTAs which account for most HTAs in Scotland.
Patient-Based Evidence for Obstructive Sleep Apnea (CADTH)
Over the past few years, CADTH has made a concerted effort to enhance how it incorporates the patient voice within projects related to medical devices and clinical interventions (13). Direct engagement has been challenging as there are not always obvious patient groups to involve when conducting HTAs, for example when assessing diagnostic or screening interventions. As a result, CADTH has explored using QES of patient-based evidence in assessments such as the Optimal Use assessment of interventions for the treatment of obstructive sleep apnea (OSA) in adults (14).
An Optimal Use assessment aims to encourage the appropriate use of a health technology by considering the clinical and cost-effectiveness, safety, and patients’ experiences of the intervention (13). Additionally, it addresses ethical issues, implementation considerations, and any potential environmental impacts. Optimal Use assessments are reviewed by CADTH's Health Technology Expert Review Panel for recommendation
The QES considered the perspectives and experiences of patients, their family members, and their nonclinical caregivers about interventions for the treatment of OSA. This included positive airway pressure therapies such as continuous positive airway pressure (CPAP), oral appliances, surgical interventions and lifestyle modifications, such as diet and exercise. From the resulting thematic synthesis, two primary themes emerged. The first theme identified a range of factors that influence whether people seek and initiate treatment for OSA. For example, the review highlighted factors that might prevent people from seeking a diagnosis or starting treatment such as risk awareness and fear. It also stressed the importance of family, spouses or partners in encouraging diagnosis in the first place. The second theme reflected that all the interventions for OSA are inconvenient, uncomfortable, and require considerable adaptation to daily routines. Some people living with OSA can adapt and incorporate interventions into their lives, although others cannot. This leads to low compliance despite great effort. Support of family members and caregivers again emerged as important in this process.
The QES contributed to the HTA in three major ways. First, the findings allowed for a better understanding of the clinical findings, in particular, how the interventions were used and how this could affect the assessment of the effectiveness of the intervention. In particular, the QES challenged assumptions in the context of compliance as it remains unclear whether some people do not observe a positive effect because they cannot comply with their treatment, or whether they do not comply because they do not observe a positive effect. Second, the synthesis informed the recommendation generated by the expert committee, that is, people with moderate or severe OSA should first try CPAP and then if they cannot tolerate it, consider a mandibular appliance. Third, the review helped to identify important implementation considerations. For example, at times there is insufficient instruction provided on how to use and care for CPAP machines or oral appliances which can affect compliance and effectiveness.
Recognition of the added value of QES of patient aspects has resulted in its wider use in CADTH and an understanding that the influence of patient-based evidence varies across HTA topics. Currently, the syntheses are conducted in parallel with other evidence gathering which is partly due to aggressive timelines. However, CADTH is exploring whether this type of review should be done earlier, perhaps during the scoping phase to better inform the entire assessment. Using QES to explore relevant patient aspects is a rapidly evolving aspect of CADTH's work and has been very well accepted by expert committees.
Patient Input in Tumor Necrosis Factor-Alpha Inhibitors (NICE)
It is likely that HTA bodies and the patients and patient groups that take part in HTAs, may at times have different perspectives about how and when patient involvement added value to an HTA. An example is drawn from a case study prepared by a patient group, the National Ankylosing Spondylitis Society (NASS), following its participation in a National Institute for Health and Care Excellence (NICE) HTA of tumor necrosis factor (TNF)-alpha inhibitors for ankylosing spondylitis and axial spondyloarthritis (15). NASS recorded the case study to enable other patient groups to learn from their experience.
The case study suggests that to have impact, a patient group needs to undertake considerable work before, during, and after the initial assessment. NASS identified the need to keep up to date with therapy developments in its areas, including monitoring the medical advisory board and the latest clinical research papers and attending conferences. NASS also gained a solid understanding of the needs of people living with ankylosing spondylitis in the United Kingdom by conducting surveys every 3 years with their members, taking 5,000 calls per year on their helpline, speaking to members at an annual members’ day and awareness events around the United Kingdom, and interacting through social media. This preparation enabled NASS to provide an informed response within the short timelines when NICE announced the appraisal.
NASS drew on its knowledge base when completing the NICE template for patient group submissions to provide information about the impact on daily life of ankylosing spondylitis, but also ran an online survey with their members during the submission period to identify patient views on specific issues, such as intravenous infusion and sequential treatment. For example, the online survey used closed questions to gather statistics such as the number of patients currently on anti TNF therapy and open questions that could be analyzed to gain a greater understanding of the advantages and disadvantages of treatment options. The survey was open for 4 weeks and attracted 608 responses. When the data were analyzed, it provided NASS with useful statistics, themes, and illustrative quotes.
Following a review of the evidence and submissions, NICE published its draft recommendations in its Appraisal Consultation Document (16). At this point, NASS had 20 days to comment on the draft recommendations, which included two important areas for people with ankylosing spondylitis that were negative. First, infliximab was not recommended for the treatment of ankylosing spondylitis. Second, treatment with another anti TNF therapy was not recommended for people whose disease had not responded to treatment with the first anti TNF therapy, or those who had an initial response which was then lost.
In response to the draft recommendations, NASS conducted another survey with members on these two issues, particularly seeking the views of people who had switched anti TNF therapies, reasons as to why they had switched, and how beneficial this switch had been. The survey was open for only 8 days due to the consultation deadline, but attracted 858 responses which provided statistics and quotes to clarify patients’ experiences and needs.
From these data, NASS was able to provide a slide presentation (including graphs and quotes from patients) for the NICE expert committee on the sequential use of anti TNF therapies and the specific people/populations who might benefit from infliximab, because it was not a self-administered treatment. These included people with memory loss, learning disabilities, dexterity problems, or a fear of needles.
These real-life data resulted in the negative recommendations being changed as follows:
Extracts from the final guidance
Patient evidence paragraphs (guidance 4.22, 4.23, 4.24)
• There is also anecdotal evidence suggesting that a second or third TNF‑alpha inhibitor can be clinically effective if the first has failed.
• [Infliximab] might benefit people with memory problems, learning disabilities, dexterity problems, or a fear of needles. (15)
Patient Input – Submissions, Surveys and Representatives (CONITEC)
In Brazil, community participation is stipulated as a right in the Brazilian Federal Constitution and other laws, including those related to the Brazilian Public Health System (SUS). The Law 12,401/2011 created the National Committee for Health Technology Incorporation (CONITEC) and formalized public and patient involvement (PPI) in the SUS HTA process through: public consultations for all recommendations about the inclusion, exclusion, or modification of health technologies; public hearings, depending on the assessment (not yet undertaken); and the participation of the National Health Council, which represents citizens and users of SUS, as a member of CONITEC (17).
Usually, CONITEC recommendations are submitted for public consultation for 20 days, but consultation can be prolonged. Anyone can provide comments on the recommendations if they identify themselves.
In 2014, a consultation form specifically for the public was created to capture the patient and caregiver perspectives of the technologies being assessed. Currently, there are two forms for public consultation comments available online. The first, seeking technical and scientific information, is usually completed by health professionals, industry, and the general public. The second aims to capture opinions or experiences and is completed by patients, caregivers, and health professionals (Reference Silva, Petramale, Rabelo, Santos, Facey, Hansen and Single18).
The number of submissions received from patients and the public is one measure of patient involvement impact (Reference Silva, Petramale, Rabelo, Santos, Facey, Hansen and Single18). In 2014, CONITEC began to disseminate a list of upcoming public consultations on its website and e-mail lists and partner's social media. This resulted in a more than 400 percent increase in submissions annually; from 2,584 submissions in 2014 to 13,619 submissions in 2015. This figure was influenced by two topics, that is, beta interferon 1-a in multiple sclerosis (19) and cesarean section (20), which attracted 4,846 submissions and 3,706 submissions, respectively. However, an exponential increase can still be noticed if these two topics are excluded.
Between January 2012 and June 2017, CONITEC performed 219 public consultations, receiving more than 30,000 submissions with more than half of these from SUS users. Since 2015, CONITEC has produced summary versions of its technical reports for the public and patients in plain language to aid understanding of the reports and enable patients to contribute more easily. This tool may also have contributed to the increase in community participation.
To investigate patient needs and preferences, CONITEC began conducting patient surveys in 2015 to inform its Clinical Protocols and Therapeutic Guidelines (PCDT). By June 2017, 13 surveys had been completed. Patients provided information about their diseases and made suggestions, such as healthcare improvements, appeals for new technologies and aspects that in their perspective critically needed addressing in the PCDT.
In 2014 and 2015, patient representatives took part in CONITEC's plenary sessions aimed at solving issues related to the use of technology including “Budesonide and Formoterol in aerosol for the treatment of asthma” (Reference Silva, Petramale, Rabelo, Santos, Facey, Hansen and Single18). One of the patient representatives taking part in a plenary session, a guest patient representative of the Brazilian Association of Asthmatics (Associação Brasileira de Asmáticos, ABRA), reported the difficulties faced by patients to control the disease, the high cost of treatment and, most importantly, the educational process needed for safe and correct use of medicines. The patient representative ratified the similar efficacy of the analyzed technologies and identified education as the most important issue as patients were often unaware of how to correctly use medical devices in asthma.
Conclusions
Reflections on the Case Studies
These observations from people experienced in patient involvement in HTA bodies provide specific case studies that use different patient involvement approaches and reflect on their impact in HTA. The story from CONITEC describes increasing the number of written submissions from patients as a measure of impact and highlights the role patients can play in identifying issues that matter most to patients, such as education about the correct use of a health technology. The story teller makes a link between the implementation of communication tools, such as disseminating information about upcoming consultations and preparing plain language guides to technical reports, and achieving the impact of increased patient participation.
In the CADTH story, patient involvement was described as an important source of evidence to inform an assessment with patient-based evidence revealing patient behaviors when seeking treatment and using the health technologies, as well as educational needs. The story teller makes a link between conducting QES on patient needs and preferences and achieving the impact of adding information that can test assumptions and address gaps in the clinical and cost-effectiveness evidence. This is consistent with findings that patient involvement can provide useful information in an HTA when there are uncertainties and gaps in the literature (Reference Menon, Stafinski and Dunn21).
The SHTG story, demonstrates the value of using primary and secondary research into patients’ needs, preferences, and experiences to determine the optimal use of a health technology. It highlights that secondary research may only identify condition specific information and that primary research may be needed to answer specific research questions relating to patients’ perspectives and experiences of using the health technology in the local health system. It suggests patient involvement can be especially valuable when there is a paucity of clinical and economic evidence and has an important role for HTAs of nonmedicines and medicines. The storyteller makes the link between patient involvement and impact of the inclusion of recommendations in the HTA report, which reflect their words, needs, and preferences. Finally, this story teller links patient involvement with a wider impact on the HTA process with patient involvement requiring the development and incorporation of appropriate methodologies, influencing questions that can be asked and the evidence assessed in future HTAs.
The experience of NASS and NICE also points to the potential for patient involvement to enable HTAs to make more informed recommendations by providing data that were not available in the published literature. This story teller emphasized a link between the patient organizations preparation, responsiveness, and ongoing involvement in the HTA and the impact of changing the recommendation to take account of the experiences of patients who would use the treatment, and the differing needs of patients with the same condition.
It is noteworthy, that, although the NICE case study resulted in changing a negative recommendation for a medicine to a positive recommendation, the common theme to each example provided is that of greater information about patients’ lives leading to improved decisions about how and when a health technology is used. Additionally, while patient involvement is sometimes considered an additional cost and burden lacking methodology, these examples suggest that HTA bodies may make better informed decisions using documented practices such as patient submissions and robust methodologies such as QES.
Without patient involvement could SHTG have developed consensus guidance on the optimal use of antimicrobial wound dressings? Without patient involvement could CADTH and CONITEC have identified the most important issues for sleep apnea and asthmatic patients respectively? And without the data collected by NASS, how could NICE have identified sub-groups not identified in the clinical trials who could benefit from a treatment? As work continues to develop the tools to evaluate the impact of patient involvement in HTA, it may also be worth considering the impact of omitting patient involvement.
Other Evaluations of Impact
The term “impact” lacks definition and is used interchangeably with influence in the literature. It can be linked with achieving a goal (5) or fulfilling a purpose which is evidenced by a range of outcomes such as the decisions made, behavior changed and knowledge gained (22). While this study focuses on a positive impact, unintended consequences and negative impacts are also an important part of measuring impact. Identifying the impact of patient involvement may depend on locating outcomes or actions “that can be credibly linked to” (22) the goal or intent of patient involvement. Furthermore, the stories presented in this study are just one example of impact evaluation. Each HTA body uses other processes.
To understand the impact of patient involvement, HTA bodies need to document its use and influence (Reference Abelson J and Gauvin F-P4). An evaluation of Danish HTAs points to how this information should be documented, such as including a section on patient aspects in the HTA report to enable others to determine how patient aspects were addressed, the kind of data included and how they were generated, and if it was integrated in the conclusion (Reference Hansen, Lee and van Randwijk23). Similarly, all National Institute for Health Research (NIHR) projects require a section in their annual reports detailing how patients were involved and the difference it made (24) and the HTAi Values and Quality Standards for Patient Involvement in HTA requiring HTA bodies to report the influence of patient contributions on conclusions and decisions in each HTA forms part of the CADTH framework for patient engagement in HTA (13).
As SHTG rarely undertakes full HTAs it is not possible to fully evaluate the impact of patient involvement across these reports. However, the HTA report documents the methods for primary and secondary research into patient aspects, identifying key findings. The final recommendations then document whether they have arisen from the patient aspects section. Furthermore, each full HTA is evaluated according to a logic model that has been developed to guide all SHTG work. A new process for targeted patient involvement in more rapid HTAs has been developed and an evaluation framework process is being developed.
At CADTH, a recent independent evaluation of patient involvement in medicine HTAs, specifically the Pan-Canadian Oncology Drug Review, included interviewing expert committee members, agency staff, and patient groups to explore different expectations for patient involvement and if these had been met (Reference Rozmovitz, Mai and Chambers25). It also measured impact against set goals defined by the participants, rather than external criteria, to explore variable achievement of democratic and scientific goals.
At NICE, the impact of most patient involvement has not been routinely documented or evaluated; rather its contribution has been noted when it has provided new insight, such as changing a comparator, or real-life data to answer a question in the absence of published (or existing) evidence. However, since February 2016, NICE has systematically captured the impact of patient input in two types of HTAs: the interventional procedures and ultra-orphan programs. This is to identify the most effective methods of patient involvement, provide examples of what works well, and to inform feedback letters to patient organizations of their involvement and subsequently help strengthen any future involvement.
Meanwhile, in addition to other processes described in this study, CONITEC has an ongoing partnership with the Oswaldo Cruz Foundation (Fiocruz) School of Governance in Health based in Brasilia to enable the systematic measurement of impact of the PPI at all (local, regional, and national) levels.
Future Use of Impact Stories
Personal stories have a role to play in reflecting on the impact of patient involvement in HTA, enabling evidence to be collected on activities individuals link to patient involvement, potentially enabling a wider understanding of the construct of impact in patient involvement. Future research should explore stories of the impact of patient involvement in HTA that come from all stakeholders involved in HTA, particularly patients and their representative groups and HTA decision makers. This could consider short- and long-term impacts and the way in which patient involvement changes HTA bodies and patient organizations, HTA processes and industry.
You have
Access
Health technology assessment (HTA) is grounded in rigorous scientific process, focused on quantitative methods to determine clinical, and sometimes cost, effectiveness. As it is intended to inform policy, patients have a democratic right to be involved (Reference Facey, Bertelsen, Wale, Facey, Hansen and Single1). However, misunderstandings about the robustness of research into patient aspects and concerns about the biases that patients may bring, have led to slow uptake of patient involvement processes. Meanwhile HTA bodies that have implemented patient involvement processes are keen to learn from others and understand what works best and why, so that they can develop processes that suit their own context. However, few HTA bodies have published evaluations of patient involvement and been explicit about the outcome or impact of involvement (Reference Gagnon, Dipankui, DeJean, Facey, Hansen and Single2–Reference Abelson J and Gauvin F-P4).
The multidisciplinary and deliberative nature of HTA, in which different sources of evidence and expert opinion must work together to address a problem, means that identifying a discrete part of an HTA process, or cause for an impact, may not be simple. Along with other expert input, patient involvement is intended to inform all the elements of an HTA from shaping research questions and informing cost-effectiveness models to communicating the advice or recommendations. As a result, the integration of patient views in recommendations is considered as a measure of impact (5), while an increase in recommendations to reimburse or fund the cost of medicines is not (Reference Berglas, Jutai and MacKean6).
This study aims to promote further discussion about the ways in which patient involvement can impact HTAs, studying particular cases in-depth, using stories told by people who lead the practice in HTA bodies. It uses the HTA international (HTAi) definition of patient involvement in HTA and reflects on (i) patient participation in HTAs, usually taking the form of patient input, and (ii) research into patient aspects, which can be described as patient-based evidence (Reference Facey, Bertelsen, Wale, Facey, Hansen and Single1). Specifically, these stories were told in response to the question of: ‘What difference is patient involvement making to your HTAs?’ and their intent was to share learnings with other HTA bodies and all HTA stakeholders.
Methods
At HTAi's 2017 Annual Meeting, a 1-day interdisciplinary workshop entitled Patient Involvement in HTA—Why, When, and How? was held to present and discuss the book Patient Involvement in HTA (7). An important element of this was to hear case studies or stories from HTA bodies about the value of patient involvement in their own jurisdiction.
Eight people experienced in patient involvement in HTA bodies were invited to present in one of two panels (Table 1). Four were asked to take part in a panel addressing the difference that research in patient aspects had made to an HTA and four were asked to form a panel focusing on the difference made by patient participation. Presenters were chosen because they represented different HTA bodies included as case studies in the book, although they were not necessarily an author of the case study. Rather than repeating the content of the book, which provided detailed descriptions of organizational patient involvement processes, they were asked to reflect on their personal experiences and observations of patient involvement and were given the additional prompt of “How or when is it adding valuing?”. Subsequently, the presenters provided personal stories or narratives of impact from the perspective of someone employed by an HTA body to make sense of the experience (Reference Greenhalgh, Russell and Swinglehurst8) rather than reports of formal evaluations of impact. In doing so, the stories offered their understanding of the impact of patient involvement on an HTA at the time of telling the story.
HTA Bodies and Speakers Who Took Part in the Workshop Panels
HTA, health technology assessment.
Following the workshop, each of the presenters were invited to document these observations in a study. Four of the eight presenters, that is, from the Scottish Health Technologies Group (SHTG), Canadian Agency for Drugs and Technologies in Health (CADTH), England's National Institute for Health and Care Excellence (NICE), and Brazil's National Committee for Health Technology Incorporation (CONITEC), agreed to take part and four either declined or did not respond to the invitation. A draft paper based on notes from the presentations was prepared by one of the authors (A.S.). The four presenters then used this draft as an aid to further reflection on the topic, including discussions with colleagues in their organizations, and redrafted the paper accordingly.
Results
The HTA bodies included in this study use a variety of approaches to patient involvement, which are summarized in Table 2.
Summary of Organizations and Their Processes
a Canadian Agency for Drugs and Technologies in Health (CADTH) (2019) CADTH Framework for patient engagement in Health Technology Assessment. https://www.cadth.ca/cadth-framework-patient-engagement-health-technology-assessment (accessed June 6, 2019).
b Silva AS, Petramale CA, Rabelo RB, Santos VCC (2017) Brazil. In: Facey KM, Hansen HP, Single ANV, eds. Patient involvement in health technology assessment. Singapore: Springer Nature, 243-246.
c National Institute for Health and Care Excellence (NICE) (2019) What we do. https://www.nice.org.uk/about/what-we-do (accessed June 6, 2019).
d Health Improvement Scotland (n.d) Scottish Health Technologies Group. http://www.healthcareimprovementscotland.org/our_work/technologies_and_medicines/shtg.aspx (accessed June 6, 2019).
e Weeks L, MacPhail E, Berglas S, et al. (2017) Canada. In: Facey KM, Hansen HP, Single ANV, eds. Patient involvement in health technology assessment. Singapore: Springer Nature, 247-264.
f Thomas V, Livingstone H, Norburn L, et al. (2017) England. In: Facey KM, Hansen HP, Single ANV, eds. Patient involvement in health technology assessment. Singapore: Springer Nature, 275-288.
g Single ANV, Macpherson K, Fearns N, et al. (2017) Scotland. In: Facey KM, Hansen HP, Single ANV, eds. Patient involvement in health technology assessment. Singapore: Springer Nature, 321-332.
Patient-Based Evidence for Antimicrobial Wound Dressings (SHTG)
SHTG, part of Healthcare Improvement Scotland, assesses nonmedicine health technologies, such as devices, diagnostics and changes to treatment pathways, predominantly in rapid HTAs of clinical and cost-effectiveness. It also undertakes some full HTAs that have time scales and resources that allow it to follow the Danish model of HTA (Reference Kristensen and Sigmund9) that includes thorough evaluation of patient aspects and organizational issues alongside clinical and cost-effectiveness.
Patient involvement in HTAs of nonmedicine technologies can be challenging. Often there is no clear patient group to participate in these HTAs and patients may be unaware of the specific intervention used. However, patients’ needs, preferences, and experiences can be essential to ensure nonmedicine health technologies are administered optimally.
SHTG's HTA of antimicrobial wound dressings in patients with chronic leg ulcers (10) demonstrates the way in which patient-based evidence can provide essential evidence to formulate conclusions and develop relevant advice. SHTG found wide variability in the type of antimicrobial wound dressings used in each health board in Scotland, and uncertainty in the clinical and cost-effectiveness of these dressings. To identify sufficient literature relating to patient perspectives and values, SHTG iteratively developed a literature search (Reference Hielkema and Wessels11), adding terminology to capture people living with chronic wounds and any type of wound dressings.
Additionally, SHTG conducted one focus group (n = 8) and six telephone interviews with people in Scotland. Thirteen of the participants had experienced chronic wound treatment with antimicrobial wound dressings and one participant was a relative of a person who had experienced the treatment. Participants were identified and recruited by a practicing tissue viability nurse in a National Health Service board following ethical approval for the research. Qualitative evidence synthesis (QES) was then used to develop a comprehensive patient aspects section for the HTA report (10). This section identified patient communication and education issues, as well as outcomes of importance to patients. For example, while wound healing was the most important outcome to patients, controlling the signs and symptoms, such as odor, exudate, and pain, and preventing infection and wound deterioration were also important outcomes.
This robust evidence base enabled patients’ perspectives and experiences to guide recommendations with three of the recommendations incorporating the needs and values identified in the QES. This additional evidence source also enabled HTA researchers to have sufficient evidence to develop consensus guidance and make recommendations (10).
The impact of patient involvement continued beyond publication of the recommendations as the patient-based evidence guided a short life implementation working group, the creation of a patient version of the HTA report (12), and guidance for Scotland's National Health Service (NHS Scotland) and staff training materials. While such an outcome is persuasive of the impact and potential value of patient involvement in HTA, it also highlights the urgent need to develop this methodology so that it can be used in more rapid HTAs which account for most HTAs in Scotland.
Patient-Based Evidence for Obstructive Sleep Apnea (CADTH)
Over the past few years, CADTH has made a concerted effort to enhance how it incorporates the patient voice within projects related to medical devices and clinical interventions (13). Direct engagement has been challenging as there are not always obvious patient groups to involve when conducting HTAs, for example when assessing diagnostic or screening interventions. As a result, CADTH has explored using QES of patient-based evidence in assessments such as the Optimal Use assessment of interventions for the treatment of obstructive sleep apnea (OSA) in adults (14).
An Optimal Use assessment aims to encourage the appropriate use of a health technology by considering the clinical and cost-effectiveness, safety, and patients’ experiences of the intervention (13). Additionally, it addresses ethical issues, implementation considerations, and any potential environmental impacts. Optimal Use assessments are reviewed by CADTH's Health Technology Expert Review Panel for recommendation
The QES considered the perspectives and experiences of patients, their family members, and their nonclinical caregivers about interventions for the treatment of OSA. This included positive airway pressure therapies such as continuous positive airway pressure (CPAP), oral appliances, surgical interventions and lifestyle modifications, such as diet and exercise. From the resulting thematic synthesis, two primary themes emerged. The first theme identified a range of factors that influence whether people seek and initiate treatment for OSA. For example, the review highlighted factors that might prevent people from seeking a diagnosis or starting treatment such as risk awareness and fear. It also stressed the importance of family, spouses or partners in encouraging diagnosis in the first place. The second theme reflected that all the interventions for OSA are inconvenient, uncomfortable, and require considerable adaptation to daily routines. Some people living with OSA can adapt and incorporate interventions into their lives, although others cannot. This leads to low compliance despite great effort. Support of family members and caregivers again emerged as important in this process.
The QES contributed to the HTA in three major ways. First, the findings allowed for a better understanding of the clinical findings, in particular, how the interventions were used and how this could affect the assessment of the effectiveness of the intervention. In particular, the QES challenged assumptions in the context of compliance as it remains unclear whether some people do not observe a positive effect because they cannot comply with their treatment, or whether they do not comply because they do not observe a positive effect. Second, the synthesis informed the recommendation generated by the expert committee, that is, people with moderate or severe OSA should first try CPAP and then if they cannot tolerate it, consider a mandibular appliance. Third, the review helped to identify important implementation considerations. For example, at times there is insufficient instruction provided on how to use and care for CPAP machines or oral appliances which can affect compliance and effectiveness.
Recognition of the added value of QES of patient aspects has resulted in its wider use in CADTH and an understanding that the influence of patient-based evidence varies across HTA topics. Currently, the syntheses are conducted in parallel with other evidence gathering which is partly due to aggressive timelines. However, CADTH is exploring whether this type of review should be done earlier, perhaps during the scoping phase to better inform the entire assessment. Using QES to explore relevant patient aspects is a rapidly evolving aspect of CADTH's work and has been very well accepted by expert committees.
Patient Input in Tumor Necrosis Factor-Alpha Inhibitors (NICE)
It is likely that HTA bodies and the patients and patient groups that take part in HTAs, may at times have different perspectives about how and when patient involvement added value to an HTA. An example is drawn from a case study prepared by a patient group, the National Ankylosing Spondylitis Society (NASS), following its participation in a National Institute for Health and Care Excellence (NICE) HTA of tumor necrosis factor (TNF)-alpha inhibitors for ankylosing spondylitis and axial spondyloarthritis (15). NASS recorded the case study to enable other patient groups to learn from their experience.
The case study suggests that to have impact, a patient group needs to undertake considerable work before, during, and after the initial assessment. NASS identified the need to keep up to date with therapy developments in its areas, including monitoring the medical advisory board and the latest clinical research papers and attending conferences. NASS also gained a solid understanding of the needs of people living with ankylosing spondylitis in the United Kingdom by conducting surveys every 3 years with their members, taking 5,000 calls per year on their helpline, speaking to members at an annual members’ day and awareness events around the United Kingdom, and interacting through social media. This preparation enabled NASS to provide an informed response within the short timelines when NICE announced the appraisal.
NASS drew on its knowledge base when completing the NICE template for patient group submissions to provide information about the impact on daily life of ankylosing spondylitis, but also ran an online survey with their members during the submission period to identify patient views on specific issues, such as intravenous infusion and sequential treatment. For example, the online survey used closed questions to gather statistics such as the number of patients currently on anti TNF therapy and open questions that could be analyzed to gain a greater understanding of the advantages and disadvantages of treatment options. The survey was open for 4 weeks and attracted 608 responses. When the data were analyzed, it provided NASS with useful statistics, themes, and illustrative quotes.
Following a review of the evidence and submissions, NICE published its draft recommendations in its Appraisal Consultation Document (16). At this point, NASS had 20 days to comment on the draft recommendations, which included two important areas for people with ankylosing spondylitis that were negative. First, infliximab was not recommended for the treatment of ankylosing spondylitis. Second, treatment with another anti TNF therapy was not recommended for people whose disease had not responded to treatment with the first anti TNF therapy, or those who had an initial response which was then lost.
In response to the draft recommendations, NASS conducted another survey with members on these two issues, particularly seeking the views of people who had switched anti TNF therapies, reasons as to why they had switched, and how beneficial this switch had been. The survey was open for only 8 days due to the consultation deadline, but attracted 858 responses which provided statistics and quotes to clarify patients’ experiences and needs.
From these data, NASS was able to provide a slide presentation (including graphs and quotes from patients) for the NICE expert committee on the sequential use of anti TNF therapies and the specific people/populations who might benefit from infliximab, because it was not a self-administered treatment. These included people with memory loss, learning disabilities, dexterity problems, or a fear of needles.
These real-life data resulted in the negative recommendations being changed as follows:
Extracts from the final guidance
Patient evidence paragraphs (guidance 4.22, 4.23, 4.24)
• There is also anecdotal evidence suggesting that a second or third TNF‑alpha inhibitor can be clinically effective if the first has failed.
• [Infliximab] might benefit people with memory problems, learning disabilities, dexterity problems, or a fear of needles. (15)
Patient Input – Submissions, Surveys and Representatives (CONITEC)
In Brazil, community participation is stipulated as a right in the Brazilian Federal Constitution and other laws, including those related to the Brazilian Public Health System (SUS). The Law 12,401/2011 created the National Committee for Health Technology Incorporation (CONITEC) and formalized public and patient involvement (PPI) in the SUS HTA process through: public consultations for all recommendations about the inclusion, exclusion, or modification of health technologies; public hearings, depending on the assessment (not yet undertaken); and the participation of the National Health Council, which represents citizens and users of SUS, as a member of CONITEC (17).
Usually, CONITEC recommendations are submitted for public consultation for 20 days, but consultation can be prolonged. Anyone can provide comments on the recommendations if they identify themselves.
In 2014, a consultation form specifically for the public was created to capture the patient and caregiver perspectives of the technologies being assessed. Currently, there are two forms for public consultation comments available online. The first, seeking technical and scientific information, is usually completed by health professionals, industry, and the general public. The second aims to capture opinions or experiences and is completed by patients, caregivers, and health professionals (Reference Silva, Petramale, Rabelo, Santos, Facey, Hansen and Single18).
The number of submissions received from patients and the public is one measure of patient involvement impact (Reference Silva, Petramale, Rabelo, Santos, Facey, Hansen and Single18). In 2014, CONITEC began to disseminate a list of upcoming public consultations on its website and e-mail lists and partner's social media. This resulted in a more than 400 percent increase in submissions annually; from 2,584 submissions in 2014 to 13,619 submissions in 2015. This figure was influenced by two topics, that is, beta interferon 1-a in multiple sclerosis (19) and cesarean section (20), which attracted 4,846 submissions and 3,706 submissions, respectively. However, an exponential increase can still be noticed if these two topics are excluded.
Between January 2012 and June 2017, CONITEC performed 219 public consultations, receiving more than 30,000 submissions with more than half of these from SUS users. Since 2015, CONITEC has produced summary versions of its technical reports for the public and patients in plain language to aid understanding of the reports and enable patients to contribute more easily. This tool may also have contributed to the increase in community participation.
To investigate patient needs and preferences, CONITEC began conducting patient surveys in 2015 to inform its Clinical Protocols and Therapeutic Guidelines (PCDT). By June 2017, 13 surveys had been completed. Patients provided information about their diseases and made suggestions, such as healthcare improvements, appeals for new technologies and aspects that in their perspective critically needed addressing in the PCDT.
In 2014 and 2015, patient representatives took part in CONITEC's plenary sessions aimed at solving issues related to the use of technology including “Budesonide and Formoterol in aerosol for the treatment of asthma” (Reference Silva, Petramale, Rabelo, Santos, Facey, Hansen and Single18). One of the patient representatives taking part in a plenary session, a guest patient representative of the Brazilian Association of Asthmatics (Associação Brasileira de Asmáticos, ABRA), reported the difficulties faced by patients to control the disease, the high cost of treatment and, most importantly, the educational process needed for safe and correct use of medicines. The patient representative ratified the similar efficacy of the analyzed technologies and identified education as the most important issue as patients were often unaware of how to correctly use medical devices in asthma.
Conclusions
Reflections on the Case Studies
These observations from people experienced in patient involvement in HTA bodies provide specific case studies that use different patient involvement approaches and reflect on their impact in HTA. The story from CONITEC describes increasing the number of written submissions from patients as a measure of impact and highlights the role patients can play in identifying issues that matter most to patients, such as education about the correct use of a health technology. The story teller makes a link between the implementation of communication tools, such as disseminating information about upcoming consultations and preparing plain language guides to technical reports, and achieving the impact of increased patient participation.
In the CADTH story, patient involvement was described as an important source of evidence to inform an assessment with patient-based evidence revealing patient behaviors when seeking treatment and using the health technologies, as well as educational needs. The story teller makes a link between conducting QES on patient needs and preferences and achieving the impact of adding information that can test assumptions and address gaps in the clinical and cost-effectiveness evidence. This is consistent with findings that patient involvement can provide useful information in an HTA when there are uncertainties and gaps in the literature (Reference Menon, Stafinski and Dunn21).
The SHTG story, demonstrates the value of using primary and secondary research into patients’ needs, preferences, and experiences to determine the optimal use of a health technology. It highlights that secondary research may only identify condition specific information and that primary research may be needed to answer specific research questions relating to patients’ perspectives and experiences of using the health technology in the local health system. It suggests patient involvement can be especially valuable when there is a paucity of clinical and economic evidence and has an important role for HTAs of nonmedicines and medicines. The storyteller makes the link between patient involvement and impact of the inclusion of recommendations in the HTA report, which reflect their words, needs, and preferences. Finally, this story teller links patient involvement with a wider impact on the HTA process with patient involvement requiring the development and incorporation of appropriate methodologies, influencing questions that can be asked and the evidence assessed in future HTAs.
The experience of NASS and NICE also points to the potential for patient involvement to enable HTAs to make more informed recommendations by providing data that were not available in the published literature. This story teller emphasized a link between the patient organizations preparation, responsiveness, and ongoing involvement in the HTA and the impact of changing the recommendation to take account of the experiences of patients who would use the treatment, and the differing needs of patients with the same condition.
It is noteworthy, that, although the NICE case study resulted in changing a negative recommendation for a medicine to a positive recommendation, the common theme to each example provided is that of greater information about patients’ lives leading to improved decisions about how and when a health technology is used. Additionally, while patient involvement is sometimes considered an additional cost and burden lacking methodology, these examples suggest that HTA bodies may make better informed decisions using documented practices such as patient submissions and robust methodologies such as QES.
Without patient involvement could SHTG have developed consensus guidance on the optimal use of antimicrobial wound dressings? Without patient involvement could CADTH and CONITEC have identified the most important issues for sleep apnea and asthmatic patients respectively? And without the data collected by NASS, how could NICE have identified sub-groups not identified in the clinical trials who could benefit from a treatment? As work continues to develop the tools to evaluate the impact of patient involvement in HTA, it may also be worth considering the impact of omitting patient involvement.
Other Evaluations of Impact
The term “impact” lacks definition and is used interchangeably with influence in the literature. It can be linked with achieving a goal (5) or fulfilling a purpose which is evidenced by a range of outcomes such as the decisions made, behavior changed and knowledge gained (22). While this study focuses on a positive impact, unintended consequences and negative impacts are also an important part of measuring impact. Identifying the impact of patient involvement may depend on locating outcomes or actions “that can be credibly linked to” (22) the goal or intent of patient involvement. Furthermore, the stories presented in this study are just one example of impact evaluation. Each HTA body uses other processes.
To understand the impact of patient involvement, HTA bodies need to document its use and influence (Reference Abelson J and Gauvin F-P4). An evaluation of Danish HTAs points to how this information should be documented, such as including a section on patient aspects in the HTA report to enable others to determine how patient aspects were addressed, the kind of data included and how they were generated, and if it was integrated in the conclusion (Reference Hansen, Lee and van Randwijk23). Similarly, all National Institute for Health Research (NIHR) projects require a section in their annual reports detailing how patients were involved and the difference it made (24) and the HTAi Values and Quality Standards for Patient Involvement in HTA requiring HTA bodies to report the influence of patient contributions on conclusions and decisions in each HTA forms part of the CADTH framework for patient engagement in HTA (13).
As SHTG rarely undertakes full HTAs it is not possible to fully evaluate the impact of patient involvement across these reports. However, the HTA report documents the methods for primary and secondary research into patient aspects, identifying key findings. The final recommendations then document whether they have arisen from the patient aspects section. Furthermore, each full HTA is evaluated according to a logic model that has been developed to guide all SHTG work. A new process for targeted patient involvement in more rapid HTAs has been developed and an evaluation framework process is being developed.
At CADTH, a recent independent evaluation of patient involvement in medicine HTAs, specifically the Pan-Canadian Oncology Drug Review, included interviewing expert committee members, agency staff, and patient groups to explore different expectations for patient involvement and if these had been met (Reference Rozmovitz, Mai and Chambers25). It also measured impact against set goals defined by the participants, rather than external criteria, to explore variable achievement of democratic and scientific goals.
At NICE, the impact of most patient involvement has not been routinely documented or evaluated; rather its contribution has been noted when it has provided new insight, such as changing a comparator, or real-life data to answer a question in the absence of published (or existing) evidence. However, since February 2016, NICE has systematically captured the impact of patient input in two types of HTAs: the interventional procedures and ultra-orphan programs. This is to identify the most effective methods of patient involvement, provide examples of what works well, and to inform feedback letters to patient organizations of their involvement and subsequently help strengthen any future involvement.
Meanwhile, in addition to other processes described in this study, CONITEC has an ongoing partnership with the Oswaldo Cruz Foundation (Fiocruz) School of Governance in Health based in Brasilia to enable the systematic measurement of impact of the PPI at all (local, regional, and national) levels.
Future Use of Impact Stories
Personal stories have a role to play in reflecting on the impact of patient involvement in HTA, enabling evidence to be collected on activities individuals link to patient involvement, potentially enabling a wider understanding of the construct of impact in patient involvement. Future research should explore stories of the impact of patient involvement in HTA that come from all stakeholders involved in HTA, particularly patients and their representative groups and HTA decision makers. This could consider short- and long-term impacts and the way in which patient involvement changes HTA bodies and patient organizations, HTA processes and industry.
Conflicts of interest
Ms. Single reports other from Springer Nature, personal fees from Patient Voice Initiative, outside the submitted work; and is an active member of HTAi's Patient and Citizen Involvement Interest Group which supported (nonfinancial) the workshop where these case studies were discussed. Dr. Facey reports personal fees from Sanofi, personal fees from Novartis, personal fees from Health Technology Wales, NHS HTA body, personal fees from Allergan, personal fees from Celgene, personal fees from Janssen, personal fees from Roche, personal fees from UCB, personal fees from Takeda, personal fees from International Plasma Protein (industry umbrella body), personal fees from Scottish Health Technologies Group (NHS HTA body), personal fees from FIPRA, personal fees from Eli Lilly and Company, personal fees from AIFP (Industry umbrella organisation in Czech Republic), other from HTAi, other from ISPOR, other from EUPATI, grants from EC H2020 grant for university research for 2 days per week outside the submitted work.