Background and Objectives
Health technology assessment (HTA) bodies using accounts of patient experiences in their decision making, often called patient input (Reference Facey, Single and Hansen1), are often asked by their stakeholders, other HTA bodies, and patient advocacy groups (PAGs) what impact it has on the committee deliberation and final recommendations. However, the concept of making a difference or having an impact means different things to different people; an impact for an HTA agency may be different from an impact for a PAG. Is there an impact (Reference Mason, Searle, Bombard, Rahmadian, Chambers and Mai2) only if there is a change to the final recommendations or is it about how the decision-making process is affected? The need for examples of impact of patient input in HTAs was also a recurring theme at a workshop on supporting patient group input of patient experiences into HTAs at HTAi 2015 in Oslo (3).
We have, therefore, studied the impact of patient input on the interventional procedures (IPs) (4) and highly specialised technologies (HSTs) (5) guidance at the National Institute for Health and Care Excellence (NICE), the English HTA body. These two programs were chosen for their different remits and approaches to patient input, their interest in evaluating that input, and their relatively small support teams compared with the NICE's technology appraisal program, which allowed more flexibility for the study. Both programs follow the same overall guidance development stages: scoping (setting the question), presentation of the evidence to the committee and draft recommendations, public consultation, response to comments received and writing of the final recommendations by the committee, resolution period, and guidance publication.
The HST process at NICE was specifically set up to evaluate ultraorphan treatments in England. Typically, these treatments have a very small patient population, higher technology costs, and a narrow evidence base with consequent uncertainties. The committee takes account of the clinical effectiveness and cost-effectiveness of each treatment. There are opportunities for patient input when scoping and submitting evidence, and also during committee discussion, consultation, and appeal (see Supplementary File 1). For this study, we considered patient input at the evidence stage, which is threefold: a written “evidence” submission from PAGs, written statements from individual patient experts before the committee meeting, and patient expert participation in committee discussion.
IP guidance considers new procedures that involve making a cut or a hole to gain access to the inside of a patient's body, gaining access without cutting, or using electromagnetic radiation. The committee assesses a procedure's safety and efficacy for use in the NHS. For this study, IP guidance was considered as an HTA, although it is more akin to regulatory approval as it does not consider cost-effectiveness.
Patient input into IP guidance consists of submissions of evidence from patient organizations, individual patient experiences collected through a process called “patient commentary,” and patient organization and individual patient responses to consultation. However, the primary patient input into IP is through “patient commentary” whereby surveys are created to collect patient experiences of a new procedure and presented at the evidence stage. The clinicians performing the procedure or the PAGs distribute surveys to patients, which are then completed and returned to NICE.
The study's aim was to establish whether patient input adds impact to the HTA process. The HTA international (HTAi) Patient and Citizen Involvement Interest Group has set international values and standards for patient involvement in HTA (6); the fifth standard for individual HTAs is “Feedback is given to PAGs who have contributed to an HTA, to share what contributions were most helpful and provide suggestions to assist their future involvement.”
Feedback has the potential to provide several levers for action: providing case studies, examples of impact, and information to produce guides and tools to support patient stakeholders. It can also highlight the value of including input from patient stakeholders to HTA agencies, PAGs, and other stakeholders in the HTA process.
To establish answers to the issues above, we set up a process at NICE to capture and evaluate this information.
Methods
Study Design
A mixed method, semi-inductive and exploratory study was conducted between January 2015 and November 2019. The study was split into three parts:
(1) HST phase one: a retrospective study conducted in April 2017 of four HSTs published between January 2015 and February 2017;
(2) HST phase two: capturing data while the guidance was being developed for a further six HSTs published between February 2018 and November 2019;
(3) IP guidance, capturing data during guidance development for twenty-six IPs scoped between February 2016 and October 2018.
Consent for the study and method of delivery was obtained from the respective committee chairs and associate directors for the two guidance types on behalf of their committees.
This study looked at two factors, first the amount, and second the type of impact by topic for an HTA process. We then aggregated this information across topics to evaluate the impact by HTA process. Depending on the findings, we would then recommend stopping input with no added impact and further investigate input that had impact. If the study established a magnitude of impact and a type of impact, we could use this information for several purposes, a key one being to provide feedback and direction to PAGs for future engagement.
Questionnaire Development
A short self-completion questionnaire (Supplementary Files 2–5) was developed to capture quantitative and qualitative data on the amount and type of impact of patient input into NICE HTAs. A survey questionnaire was chosen for speed of data collection, maximized response rate, timeliness and convenience (it could be completed immediately after the committee had discussed the patient input), and uniformity of responses with the potential to provide a quantitative account of the phenomenon.
The aim was to have enough overall consistency between questionnaires for HST and IP to compare data across both programs. However, taking a flexible approach was crucial to the further tailored development of the questionnaires to best suit each HTA process.
The questions were first developed by the PIP team in house, as none could be found in the literature. They were clear and precise with response options, also leaving the opportunity to the respondents to answer open-ended questions with free text. The questionnaires were then validated by each of the two programs and piloted in the IP program for 6 months.
To ascertain the level of impact for both programs, a five-point Likert scale was used, ranging from no impact to significant impact. A follow-up question to determine the type of impact was asked only where an impact was identified. The following (nonmutually exclusive) options were: whether it corroborated other evidence, whether it helped interpret other evidence, or whether it provided new evidence. Respondents were asked to provide qualitative examples to explain their answers about the type of impact (Supplementary Files 2–5).
The original aim of the questionnaire for HSTs was to capture the impact of the PAG submission of evidence. However, because the committee completed the questionnaire after they discussed the entirety of the evidence, the committee's responses encompassed the PAG's submission, and also the patient experts’ written statements, their oral testimonies and discussions during the meeting (Supplementary Files 2–3 for the forms used in both phases of the HST study). Phase one included a question on whether the input had had a significant impact on the committee's recommendations; for phase two, this was replaced by a question on whether the input had helped clarify the quality-adjusted life-years (QALYs) and incremental cost-effectiveness ratios (ICERs).
After the 6-month IP pilot, the questionnaire was reviewed and amended, particularly in terms of presentation and question order (see Supplementary Files 3 and 5).
Participants and Recruitment
Reponses were sought from IP and HST committee members who were present for the discussion of each guidance topic that had patient input during the study period. The IP and HST committees are different and each specializes in a different type of HTA. Both committees are independent of NICE and members are openly recruited. The committees are of different sizes and membership consists of NHS staff, academics, clinicians, industry, and lay members (committee membership by role type and quoracy is available in Supplementary File 6).
Only the committee received the questionnaire for the HST phases of the study, whereas for IP, both the committee and the technical team (NICE staff with responsibility for the technical aspects of the assessment process) received them.
Data Collection
For phase one of the HST study (retrospective), an online version of the questionnaire was produced with an optional Word version. Responses were sought only from those committee members present for each HST discussion. Copies of the PAG submissions and the patient issues slides were provided to refresh their memories. For phase two, it was decided to move to the IP model of distributing the questionnaires.
For IP, the questionnaires were distributed at the end of the committee discussion, in hard copy, on colored paper (to be distinct from other papers). They were completed and collected immediately after the discussion of the procedure had concluded, and before the next agenda item. This proved efficient in terms of response rate and quality of information.
Data Analysis
After the questionnaires had been collected, the data were entered into an online system. This was originally SurveyMonkey (7), but due to a change in corporate policy, it changed on 10 June 2019 to Snap Surveys (8). Reports were produced and the data were exported into a Microsoft Excel database for descriptive statistical analysis, mainly calculating proportions and percentages to describe the amount of impact and the type of impact of patient input. The explanations provided by the respondents were grouped together per response given to illustrate their choices, without further analysis.
Once the data had been analyzed for phase one of HST and the first 6 months of IP, the method for capturing the data and the questionnaire itself were reviewed (see Questionnaire Development above).
Results
HST—Phases One and Two
Tables 1 and 2 present the data of both phases of the HST study together; one question differs between study one and two.
All quantitative data from both HST study phases
a Numbers prefixed by HST are phase one of the study, and those prefixed by ID are phase two.
Summary quantitative data alongside explanations of responses for both HST study phases
As shown in Table 1 (all quantitative data for the HST study), the HST part of the study consisted of ten pieces of guidance and eighty-seven responses (twenty-one responses in total for the first four HST topics [retrospective study] and sixty-six responses for the other six HSTs). For the first phase, the mean response rate was 50 percent (standard deviation ±12%) and for the second, it was 94 percent (±4%) (the committee needs to be quorate, but attendance for each committee meeting varies—see Supplementary File 6). None of the respondents stated that there was no impact for patient input. Across both studies, the same four HSTs scored the highest on the amount of impact and on bringing new evidence (and in the same order, HST3, HST2, ID926, and ID1242).
Table 2 shows the summary of quantitative responses for the two phases of HST study combined with the explanations for the quantitative responses. The explanations are given according to HST or ID number depending on the phase of the study. The explanations for helping to interpret the evidence included an increased understanding of the nature of the disease and context, whereas the explanations about new evidence emphasized the impact of the condition and treatment on the lives of patients and carers, including maintenance of certain functions, carer disutility, and the acceptability of the treatment compared with existing treatment. For phase one, patient input helped with wider factors being considered, whilst the committee would have liked to have been presented with more systematic data from patients. For phase two, over a third of respondents stated that patient input helped clarify the data for the health economic modeling (QALYs and ICERS), whereas the committee would have liked to have seen more quality-of-life (QoL) information, more about carers’ QoL, and a range of different patient experiences.
Interventional Procedure
Twenty-six pieces of IP guidance including patient commentaries were scoped between February 2016 and October 2018. A total of 440 responses were collected for these IPs, and the mean response rate was 94 percent (±7%). On average, there were ten patient commentaries per IP (± ten). The data collected before the questionnaire were changed (representing 7 pieces of IP guidance/133 responses) and were analyzed separately for the type of impact, but they were pooled to the other data for the amount of impact. However, there was no option for the respondents to explain their choice for the level of impact on the first version of questionnaires.
The majority of respondents (60%) said patient input had no impact on the guidance (see Table 3 for the amount of impact of patient input on the guidance and the explanations given by the respondents for no impact or very little impact).
Results for IP—Amount and type of impact of patient input
The five IP guidance for which patient input had the most impact (moderate or significant) were IP 780/2 (radiation therapy for early Dupuytren's disease, 42%), IP 1193 (minimally invasive sacroiliac joint fusion surgery for chronic low back pain, 39%), IP660/2 (surgical repair of vaginal wall prolapse using mesh, 36%), IP 1012/2 (subcutaneous implantable cardioverter defibrillator for the prevention of sudden cardiac death, 36%), and IP 311/3 (sacrocolpopexy using mesh to repair vaginal vault prolapse, 32%) (Figure 1). The respondents could explain their choices for only two of these IPs where the second version of the questionnaire was used. For IP 660/2, the respondents said the committee “spent a long time discussing it and made significant changes,” it “led to a committee comment” (extra information that the committee sometimes adds to the guidance), and it “supported what [the respondent] thought already.” For IP 1012/2, the respondents said it was “helpful given the impact on patient's life,” it “affirmed positive coherence with device,” and it “confirmed the committee's decision.”
Amount of impact for IPs.
The five IPs scoring the highest on the amount of impact also scored the highest on the type of impact responses about supporting the other evidence (for IPs with the first version of the form) and on responses about being inconsistent with the other evidence (for other IPs) (please see Table 3 for the results for the type of impact of patient input along with the explanations given by the respondents). For four of these IPs, respondents said that the comments received were positive and helped confirm the evidence, sometimes because the published evidence was poor. For one of these five IPs (IP 660/2), respondents indicated that “the patient commentaries highlighted long-term complications and resulted in a committee comment.”
Discussion
Core-Summary Findings
On the 536 questionnaires analyzed across the HST and IP programs, respondents agreed that the most common type of impact for patient input was that it helped interpret other evidence by providing context. Whereas 30 percent of respondents in IP said that patient input provided new evidence by offering patients’ personal experience versus trial data, this was reduced to 17 percent for the HSTs in this study, although this varied considerably by individual HST. Committee members from both programs appear to not only like the reassurance that patient input can offer for the decision making (for HST, the second most popular type of impact was that it corroborated the other data), but they also welcome the “color” and real-life patient experience it adds to the other information available.
For HST, nobody indicated it had no impact on the guidance. For IP, however, most of the respondents (35%) did. For IP, they explained their choice by the uncertainty around patient commentaries (small numbers, patient population unclear, unsure if the patient had the procedure assessed, intervention done and device used on sedated patients, generalizability of data unclear) and by saying that the commentaries added minimal information. When we looked at the pieces of guidance that scored the highest on the amount of impact, the most common explanation for HST was that it brought new evidence to the committee about the impact of the condition and treatment on the lives of patients and carers. For IP, the explanations were more around the validation of the committee's initial decision.
These differences could be explained by three factors: the different remit of the two programs, the level (Reference Carman, Dardess, Maurer, Sofaer, Adams and Bechtel9) and amount of patient involvement, and the quantity of published evidence. First, HST has a wider remit as it considers clinical and cost-effectiveness, whereas IP considers safety and efficacy. Second, the level and the amount of patient involvement are higher for HST as patients are involved in the scoping process, participate in the committee meetings, and provide written statements. For IP, the primary input is of lower level as it is mainly patient commentary. Lastly, there is usually less published evidence available for HST than for IP, so it offers new real-world information which might explain why it provides moderate or significant impact. For IP, there is usually more published evidence available, so it has less apparent impact on the guidance decision making (it changed the IP recommendation for 7% of respondents), but it still helps the decision makers by providing QoL information that is often missing from published studies and by confirming trial evidence.
Comparison with Existing Literature
The published literature on this topic (Reference Rozmovits, Mai, Chambers and Chan10–Reference Boivin, Lehoux, Burgers and Grol18) is scarce and usually indicates that more robust evidence of impact is needed. The impact of written submissions of evidence from patients on HTA has been assessed at NICE before (Reference Staley and Doherty19) but with a different method (qualitative interviews of nine members of NICE appraisal committee). To the best of our knowledge, this is the first study looking at both the amount and type of impact of patient input in HST and IP guidance. It is also the only study where the decision makers were all systematically surveyed at the same time for every guidance with patient input. This allowed us to gather a lot of data with a high response rate. Our conclusions are in line with previous findings (Reference Dipankui, Gagnon, Desmartis, Légaré, Piron and Gagnon16;Reference Staley and Doherty19;Reference Berglas, Jutai, MacKean and Weeks20): patient input helps interpret the other evidence by providing context. In addition, from our study, nearly a third of respondents stated that the patient input had provided new evidence that has not yet been brought out elsewhere in the literature.
Limitations
There were several limitations to this study. For HST, firstly, there were two separate studies, one retrospective and the other contemporaneous with the committee. This meant that there were fewer respondents for the first study and people had to rely on their memory rather than completing the questionnaire immediately after their discussions.
Secondly, the original aim of the study was to measure the impact of the PAG submission. However, because the committee also received statements from individual patient experts and heard testimony and discussion from the patient experts at the committee meetings, it was not possible to assess the impacts of the three types of input separately, as they were considered simultaneously.
Thirdly, the committee was not included in the design and review of the questionnaire. Lastly, ten pieces of HST guidance were not enough to show trends in what was most useful, and the disease areas were diverse except for the two-amyloidosis guidance.
Additionally, the questionnaire changed during the study to reflect slight methodological changes in the committee's remit (5), so there was a subset of data that could not be pooled.
For IP, firstly, the questionnaire changed during the study, so there were two sets of data that could not be pooled for some questions. The presentation format of the patient commentaries also changed during the study, and this may have influenced the respondents. Secondly, patient commentaries were not always considered together within a topic (sometimes commentaries were received late), so respondents may not have recalled all the patient commentaries while answering the questionnaire. Thirdly, the number of patient commentaries received per IP varied from one to thirty-eight, and this could have influenced the impact. Fourthly, although they were asked to explain their choices, most respondents did not do so. Fifthly, the amount of published evidence varied between IPs. Finally, we did not look at the influence of the disease areas of the IPs studied on patient input either.
For both HST and IP, providing response options for the type of impact to the respondents may have restricted the range of impacts and the qualitative responses submitted. However, in the HST questionnaire, one response option was “other,” so respondents could possibly add to the response options provided for the type of impact.
Next Steps
As patient input becomes more and more crucial for HTA, this exercise has highlighted its use in the HST and IP programs at NICE. As a result, the committee has shown a preference for patient input to be underpinned by patient surveys for HST. The committee questionnaire responses are also now used to provide feedback to PAGs. In future, it would be helpful to include the NICE technical team in the questionnaires for any type of guidance so that their responses can also help inform the feedback letters. In IP, asking committee members to complete a form at the end of the meeting for each IP with patient commentaries has amplified the committee's focus on the patient's voice and is reflected in the increased presence of committee comments emphasizing the usefulness of patient commentaries for the guidance.
Committee involvement in the design of questionnaires and the regular feedback given on the results have also increased the buy-in of the project by the committee, thereby enhancing the response rate. Moreover, this study has allowed the collection of case studies of patient input impact in IP. For certain topics (e.g., mesh), it also provided a more balanced view compared with the patient comments received during consultation.
In IP, the next step could be to assess the impact of the PAG submissions and ultimately to compare the impact of patient survey (commentaries) and PAG submissions. Furthermore, it will be important to increase the robustness of patient survey data, for example, by double-checking that the patient answering the survey had undergone the procedure under discussion. Finally, trying to find out if a minimum number of patient surveys has an influence on the level of impact would be helpful.
Further collection of impact data is needed to see whether there are any trends of impact across treatments, devices, procedures, disease areas, methods of patient input, types of HTAs, or whether some combinations of these factors are more important or efficacious. This would mean systematically collecting data in HTAs and introducing impact data collection to NICE's other HTA types such as medical technologies (MedTech), technology appraisals, and diagnostics. A pilot has already been done for MedTech on one topic (gammaCore for cluster headache) (21). Because it included a patient survey, it was conducted via the same paper format, asking which method of patient input was the most helpful, how helpful it was, and if it provided new information. All respondents said that patient input helped determine the recommendations. The results are intended for future publication.
More comparable questionnaires could be designed to generalize and compare data between programs. The qualitative data could also be analyzed. If trends can be identified and predicted, then both HTA bodies and PAGs could arguably be directed to put more resource into the aspects of patient input with the most impact, and less where input has less influence. Supplementary analysis could also be conducted to see if certain types of respondents have preferences for specific patient inputs. Additionally, it should be considered whether to ask an open question for the type of impact to check if the proposed types of impact in the questionnaires are relevant and to possibly discover another type of impact.
Conclusion
This study has shown that the impact of patient input is not necessarily explicit in changing a recommendation but provides context, reassurance, and new information to the committee for the decision-making process in HTAs. It is, therefore, important that the collection of patient input data in HTA remains and expands. The analysis of the impact of patient input by NICE's public involvement and guidance-producing teams along with committee involvement should also continue, so as to monitor and increase the quality of the patient data collected. Patient input should then evolve according to these analyses to serve HTA decision making at its best.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462320002214.
Acknowledgments
The authors thank John Powell, Chloe Kastoryano, Kimberly Carter, and Mandy Tonkinson for their participation in the studies, their input into the manuscript and their helpful comments and, additionally, Sarah Scott, Victoria Thomas, and Jane Cowl for their helpful comments on the manuscript.
Funding
This research received no specific grant from any funding agency, commercial or not-for-profit sectors.
Conflicts of Interest
All authors have nothing to disclose.
Open access
