Background
The European Network for Health Technology Assessment (EUnetHTA) was established to enable and support the production and use of jointly produced Relative Effectiveness Assessments (Reference Erdös, Ettinger, Mayer-Ferbas, de Villiers and Wild1). Its aim is to create an effective and sustainable network for Health Technology Assessment (HTA) across Europe as stated in the Cross Border Directive (2011) (2). This directive provided the political and regulatory framework for EUnetHTA Joint Action 1 (JA1) and the succeeding JA2 and JA3, which are (partially) funded by the European Commission. The main objective in JA1 was to establish an effective and sustainable HTA collaboration in Europe (3), whereas the objective in JA2 was to strengthen the practical application of tools and approaches (4). In JA3, the main objective was to define and implement a sustainable model for cooperation on HTA across Europe after the end of the project period (5).
The EUnetHTA assessments focus on the clinical aspects only, summarizing the available evidence on effectiveness and safety and comparing the technology under assessment to other existing alternatives (6). Hence, EUnetHTA assessments do not provide recommendations on added value or reimbursement of a technology; such reimbursement decision remains within the national autonomy of a country.
EUnetHTA recognizes that different groups and organizations can provide key information, expertise, and relevant experience to the production of EUnetHTA assessments. For this reason, the EUnetHTA Joint Action Stakeholder Policy was developed during JA1 to facilitate a transparent and responsible stakeholder involvement process (7). According to the Stakeholder Policy, the appropriate involvement of stakeholders has to be defined according to the work in question (7).
In EUnetHTA JA2, only one of six assessments on pharmaceutical technologies (PT) and three of six assessments on other technologies (OT) involved patients or consumers. In addition, it revealed difficulties in recruiting patients and that the workload was overwhelming for involved patients (e.g., review draft assessments). To support the development of a sustainable process for patient involvement in EUnetHTA assessments and Early Dialogues, the EUnetHTA Task Group on Patients, Consumers, and Health Care Professionals (TG P/C&HCP, hereafter referred to as TG) was established. The TG consisted of representatives from work groups on Network Coordination, Dissemination, Joint Production, Life Cycle Approach to Improve Evidence Generation and Quality Management, Scientific Guidance and Tools (5), and was led by the EUnetHTA Senior Scientific Officer. It was important for the TG to build on the experience and knowledge that was already available in EUnetHTA and to agree on some preferred approaches for patient involvement that could be applied within EUnetHTA assessments. Experiences with patient involvement in assessments were recorded before, during, and after the TG formed recommendations for patient involvement in EUnetHTA assessments. The recommendations are described in the document “Patient Input in Relative Effectiveness Assessments” that is available for access, and this document was published in 2019 (8). Stakeholders representing European patient and consumer organizations, as well as other organizations from the HTA network stakeholder pool, were consulted with regard to the recommendations for patient involvement (8).
This article aims to provide a comprehensive overview of the experience with patient involvement gained by EUnetHTA in EUnetHTA assessment production from June 2016 until the end of November 2019, by describing and quantifying the approaches used and outlining the challenges and avenues for improving current processes. At this time point, the majority of the project period of EUnetHTA JA3 has passed and this allows for capturing learning points. The views of patients who were involved in EUnetHTA assessments were not systematically gathered at the time of writing this manuscript, and are, therefore, not part of this article.
Methods
Assessments related to PT and OT (e.g., medical devices, nonpharmaceutical procedures, etc.) were reviewed, and patient involvement approaches and experiences were quantified and described separately due to differences in organization of the PT and OT assessment processes.
Assessments related to PT and OT conducted from June 2016 until the end of November 2019 within EUnetHTA JA3 were reviewed. The assessments were either finalized or in an advanced stage of the scoping phase when information was collected. Assessments in which stakeholder involvement was not sufficiently advanced were not included in this article. Information about patient involvement procedures was retrieved from the EUnetHTA Web site and by contacting EUnetHTA project managers.
Experiences with patient involvement were collected from EUnetHTA project managers. The project management in PT assessments is done centrally, and information on patient involvement is kept in a database. In OT assessments, the project management can be done centrally or can be conducted by selected activity centers in a decentralized way. In both cases (centralized and decentralized project management), a set of questions was sent to respective OT project managers to gather relevant information. If needed, follow-up or clarifying questions were posed to the project managers.
Approaches for patient input
By reviewing assessments from June 2016 until the end of November 2019 and assessing information about patient involvement approaches, it was revealed that twelve out of fourteen PT assessments and eleven of twenty-two OT assessments successfully involved patients. All fourteen PT assessments attempted to seek patient input and only two assessment teams were unsuccessful in involving patients (Figure 1a). In OT, fifteen assessments attempted to seek patient input and eleven were successful in including patients (Figure 1b). Seven OT assessments did not attempt to seek patient input for various reasons. In this article, successful involvement is related to assessment teams’ perspective and means that patients could be identified/recruited and/or the pursued approach was completed, and unsuccessful involvement means that no patients could be identified/recruited and/or the pursued approach was not completed.
Figure 1.(a) Patient input in PT assessments. (b) Patient input in OT assessments.
The TG reviewed existing procedures and (grey) literature in regard to patient involvement approaches and collected relevant experiences from TG members. An overview with possible approaches was created, where further information was outlined: a description of the approach, the preferred time point, the target group, resource use, the expertise required by the assessment team and the patient, advantages, challenges and limitations, and additional considerations relevant for the PT and/or OT production process. Subsequently, these approaches were discussed in the TG e-meetings focusing on the applicability and feasibility within EUnetHTA. This method did not aim to be exhaustive in listing all possible patient involvement approaches but aimed to identify common practice in both (previous) EUnetHTA assessments and national HTA experiences. Based on this initial examination and discussion, five main approaches to obtain patient input in EUnetHTA assessments during JA3 (Figure 2) were identified and outlined in the document “Patient Input in Relative Effectiveness Assessments” (8): (i) (online) patient input template, (ii) one-on-one conversation, (iii) group conversation, (iv) participation in scoping e-meeting, and (v) other approaches. In several assessments, more than one approach was used. However, if one approach was used multiple times within a single assessment, it was still counted as one.
Figure 2.Pursued approaches for patient involvement.
For PT assessments, pursued approaches for patient involvement were (online) patient input template (n = 12), one-on-one conversation (n = 4), and other approaches (n = 1) (Figure 2). Of these, four did not succeed in involving patients, due to nonresponse or unwillingness of patient organizations to participate. This was the case for the online patient input template (n = 2), one-on-one conversation (n = 1), and other approaches (i.e., ranking of outcomes, n = 1). In two PT assessments, more than one approach was pursued.
For OT assessments, pursued approaches for patient involvement were as follows: (online) patient input template (n = 6), one-on-one conversation (n = 5), group conversation (n = 2), scoping e-/f2f meeting with patients/patient representatives (n = 1), and other approaches (i.e., a review of the draft Population, Intervention, Comparator and Outcomes (PICO), review of draft project plan or draft report, involvement as a team member, the use of national surveys, n = 9) (Figure 2). Not all led to successful patient involvement in assessments. This was the case for the use of the patient input template (n = 3), one-on-one conversation (n = 1), and other approaches (a review of scope or feedback on the selection and importance of outcomes, n = 3).
Experiences with patient input
Figure 3 presents a simplified process flow of the EUnetHTA assessment production process and shows the necessary steps for the patient involvement approaches.
Figure 3.Process flow of the EUnetHTA assessment production.
Experiences from PT
The responsible project manager coordinated the identification of patients and/or patient organizations to be involved in the PT assessments and documented this in a database. Patients or organizations were identified based on suggestions from the European Medicines Agency (EMA) (n = 11) via an Internet search (n = 4), via the HTA Network Stakeholder Pool (n = 14), or via other approaches (n = 3) such as taking suggestions from the authoring team or selecting organizations based on previous assessments. For nine assessments, several identification approaches were used.
In total, over thirty-six patient organizations were contacted for the fourteen PT assessments, and in twelve assessments, patient involvement approaches were successfully applied. The reasons for not including patient input included a lack of response, too tight timelines to identify suitable patients and/or patient organizations, or unwillingness from patient organizations to participate in assessments.
In eleven assessments, the patient input template was used as a basic approach in an early phase of the assessment, to ensure that patient input was gathered at the time of drafting the assessment scope. If deemed relevant by the authoring teams, the patient input template could be complemented with different approaches. In two assessments, this was the case (i.e., ranking of outcomes (n = 1) and one-on-one conversation (n = 1)). The median number of patient organizations that responded to the online patient input template was two patient organizations (range: 0–7). In the eight assessments in which more than one patient organization completed the online call for patient input, these organizations came from different countries. The patient organizations provided a synthesized patient view. This allows capturing divergent views due to national differences in, for example, the treatment pathway. In all assessments, the online patient input template was in English and responses in the national language of the respective authoring team were allowed. In two out of these eleven assessments, none of the approached patient organizations completed the patient input template.
Four PT assessments attempted to identify individual patients for a one-on-one conversation. For this approach, the project managers reached out to the patient pillar of the HTA Network Stakeholder Pool (n = 3) or the author identified national patients (n = 1). The identification of individual patients was very challenging, due to a tight production timeline, the burden of disease on patients, as well as the inclusion of a specific population group in the assessment. In one of these four assessments, the attempt to conduct a one-on-one conversation was unsuccessful, due to the time constraints and the burden of disease on patients. In three out of these four assessments, one individual patient was identified for each assessment. In two assessments, involvement occurred in an early phase of the scoping phase (i.e., the drafting of the research question), whereas in the other assessment, it was in a late stage (i.e., the finalization of the assessment report). The one-on-one conversation was conducted by the Project Manager (n = 1) or the author (n = 2) and was conducted in the local language of the patient (n = 1) or in English (n = 2). The experience was that a one-on-one conversation with individual patients in the scoping phase was more helpful than such a conversation later in the assessment phase. In the assessments where the one-on-one conversation approach was used, the experience was that there was uncertainty on how to interpret the results with only one single patient and how to account for cultural deviations. The patient input template, however, is believed to enhance the use of the patient input, because it represents the view of a broader group rather than the input from one individual patient. In addition, the online patient input template also ensured that various national perspectives are heard, as in the majority of assessments, more than one country was represented.
Experiences from OT
Sixteen OT assessment teams reported information about patient involvement in assessments through a set of questions when those were either finalized or in an advanced stage of the scoping phase. The questions were not sent to the project managers of six assessments, because the centralized project management already received the information beforehand that five assessment teams were not able to involve patients or patient organizations due to tight timelines and because one assessment team sent the information on successful patient involvement approaches via e-mail. In the following, information from the sixteen questionnaires that were sent to the assessment teams is presented.
Out of sixteen assessments, fourteen assessment teams planned patient involvement. One assessment team contacted individual patients, seven assessment teams contacted organizations, and six assessment teams contacted both individual patients and patient organizations. In nine of fourteen assessments, both national and European patients and patient organizations were approached. In total, the fourteen assessment teams contacted over 40 patient organizations and several individual patients (some were directly approached via patient organizations). The reason that two assessment teams did not plan for patient input was a tight timeline and a lack of a specific patient group organization. Patients or organizations were identified after receiving suggestions from coauthors (n = 3), dedicated reviewers (n = 6), external clinical experts (n = 5), via an Internet search (n = 8), via previous experiences at their HTA agency (n = 8), or via other identification approaches (n = 7). Other identification approaches included suggestions from a patient support expert, from a European patient organization, from own networks, or from the Ministry of Health. Several assessment teams used more than one identification approach. Selection was based on PICO relevance.
Ten assessment teams managed to recruit and include individual patients or patient organizations in their assessment. The reasons for unsuccessful recruitment of patients included a lack of response or unwillingness to participate, which was the case in four assessments.
The patient input template was successfully applied in three assessments, whereas one of them was published in the form of an open call for patient input on the EUnetHTA Web site (one patient organization completed the template); for the other two assessments, the template was sent to relevant patient organizations via e-mail. In one case, however, a statement was received back instead of a completed template, and in the other case, two patient organizations completed the template. On three occasions, the template could not successfully be used, whereas one of them was published on the EUnetHTA Web site as well, but did not result in any response from a patient organization, despite several efforts in contacting relevant patient organizations in order to make them aware of the open call.
In two assessments, group conversations were performed, whereas one assessment included two separate groups, one with adults and one with children and their carers. The group conversations included five or more individuals per group. One-on-one conversations were performed in four assessments, where one and up to five individuals were involved in each of the assessments. All group and one-on-one conversations were done in the local language of the patients. In one assessment, a scoping meeting was performed, and in six assessments, other approaches like a review of the draft PICO, a review of the draft project plan or draft report, participation as an assessment team member, or the use of national surveys were used. Seven assessment teams reported positive experiences with patient involvement, and one assessment team narrated its experience with one-on-one conversation in the following way:
“It was a really interesting experience because after the interview there is a sensation of 'closeness' to the patients suffering from the condition. It helps to understand who is benefiting (or not) from the technology under assessment.”
Management of patient input
In both PT and OT, the assessment teams primarily used patient input to inform outcomes. Sometimes, they also used patient input in the GRADE—Grading of Recommendations, Assessment, Development and Evaluation—ratings, in discussions, or as a supplement to information provided in the literature. Some assessment teams provided a summary of patient input in the assessment and added all of the inputs in appendix to the assessments.
Suggestions for improvement of patient involvement in assessments included standardization of patient input procedure, specific criteria for the use of group conversations, and request for assistance in the identification of patients, or a pool of relevant patients or patient organizations to contact. One assessment team that was unable to include patients or patient organizations in the scoping phase noted that patient input in the assessment phase also proved to be useful.
Challenges and avenues for improving current processes
It is widely recognized that patient involvement throughout the medicines life cycle is important to understand the needs of the target patient population. However, patient involvement in national HTA activities varies. Some HTA agencies engage patients in their HTA activities but not always routinely (Reference Hailey, Werkö, Bakri, Cameron, Göhlen and Myles9). To bring the patient perspective closer to the center of medicine development, authorization, HTA, and reimbursement processes, many initiatives have been started on a European Union level. For instance, under the Innovative Medicines Initiative (IMI), public-private collaborative initiatives specifically on patient involvement have been launched, such as the IMI-PREFER (10) or the IMI-PARADIGM (11) research projects. Within EUnetHTA, the effort to strengthen patient involvement was initiated during JA2, although it was during the current JA3 that criteria and processes for collecting patient input were well defined and structured (8). PT and OT piloted different patient involvement approaches in their assessments that led to EUnetHTA-recommended approaches for patient input and visibility of patient involvement (8), an (online) patient input template (8), a patient information flyer (8), and an evaluation questionnaire for patients. Even though EUnetHTA has now developed recommendations for patient input, it does not prevent EUnetHTA partners from using other approaches for patient input in HTA assessments. EUnetHTA partners could choose the approach that fits best to the topic under assessment, considering national practices and the EUnetHTA recommendations.
For assessments in PT, it is mandatory to seek patient input. The (online) patient input template is now used as a basic approach, and this can be complemented with other approaches. In OT, it is obligatory to discuss patient involvement in the assessment team and to provide a sound rationale in case their involvement is not feasible or is not pursued due to other reasons. The approaches to be used are to be agreed upon within the assessment team. In some instances, the defined population might not have a clear (patient) organization or self-help group behind them. The identification process of individual patients or patient organizations depends on the topic. The assessment teams’ experiences showed that the identification of individual patients and patient organizations proved to be challenging as many organizations did not respond or were not willing to participate. The project managers and assessment teams tried to use different existing sources like the pool from the EMA in PT and the HTA Network stakeholder pool in PT and OT, in addition to searches on the Internet. They aimed at involving both European and national patient organizations. For national organizations, national databases for patients or patient organizations (if existent), national contacts, and the assessment team members often approached them in their national language, which facilitated the development of contacts. To facilitate the recruitment process, EUnetHTA created a patient flyer (8) to inform patients and patient organizations about EUnetHTA and how they can contribute in EUnetHTA assessments. To further facilitate the identification and recruitment of patients for involvement in assessments, some of the assessment teams suggested that EUnetHTA could consider a database of patients/patient organizations, which could be used as an additional source or as an approved list like a “patient and consumer pool.” In addition, it could be explored if a stakeholder involvement officer would facilitate the identification of relevant patients/patient organizations within EUnetHTA assessments. A wide consultation and collaboration between patient organizations and EUnetHTA could also help clarify why a lack in response and difficulties in collaborating with patient organizations arises.
Which assessments successfully involved patients is visible on the EUnetHTA Web site (12), because a respective patient sign is placed next to the title of the assessment in case of actual involvement. Both PT and OT assessment teams provided a summary of the patient input in the assessment if the teams had access to patient validated summary and/or key statements. Some PT assessment teams tried to actively show which outcomes were stressed by patient organizations. In OT, some assessment teams also added patient input in an appendix to the assessments. It can be difficult to measure if patient input has a direct and quantifiable impact, but the assessment teams reported that patient input had an indirect impact on the assessment process because they better understood the patient situation and experiences. Experiences from PT assessment teams in EUnetHTA showed that there were challenges associated with using patient input if it was obtained from one individual patient, as they felt that this was not a generalizable perspective to other patients of the same nationality or a different nationality. Others also argue that the patient perspective may depend on their experience with the health system and that it could be influenced by the social and political environment that the patient lives in (Reference Carman, Dardess, Maurer, Sofaer, Adams and Bechtel13). However, regardless of their nationality and social/political background, patients bring experiential knowledge of living with the disease (Reference Gagnon, Desmartis, Lepage-Savary, Gagnon, St-Pierre and Rhainds14) and can, therefore, inform which outcomes are most relevant for them and provide information about the treatment process. Also, the EMA reported after having piloted engaging patients in the CHMP Benefit/Risk discussions that patient input of two individual patients was very valuable for the scientific discussion due to the patient's experience of living with the disease (15). Understanding the societal burden across countries and cultures could, therefore, be beneficial for assessment teams when interpreting patient input to the PICO. How to interpret patient input is currently not streamlined across assessment teams, but a group within EUnetHTA is working on a guidance on how to use the patient input among assessment teams. This is in line with one of the outcomes of the Fourth Edition of the European Network for Health Technology Assessment Forum, which called for a more robust measurement of contributions and visibility of the added value given by patients in the assessment process (Reference Tafuri, Bélorgey, Favaretti, Frénoy, Giorgio and Guardian16). EUnetHTA assessments are very technical and may not be easily understood by patients. Consequently, EUnetHTA came up with a plain language summary template that reflects the needs of EUnetHTA. It is currently being piloted and the aim is to use it for upcoming PT and OT assessments. Plain language summaries may empower patient populations (17), offer a valuable tool to communicate to the public, create awareness, and increase value for assessments.
How EUnetHTA experiences can influence national patient involvement practices
To the best of the authors’ knowledge, much research has been conducted on mapping methods for national stakeholder involvement in HTA activities or guideline development, but only limited research has been done on how patient involvement could be incorporated in international collaboration on HTA activities. However, the main challenges for patient involvement identified by PT and OT assessments are identifying patients or patient representatives and obtaining representative input and burden of work, and these correspond to challenges reported in studies on national HTA activities (Reference Hailey, Werkö, Bakri, Cameron, Göhlen and Myles9) and guideline development (Reference Eccles, Grimshaw, Shekelle, Schünemann and Woolf18).
Not all EUnetHTA partners have experience with patient involvement at a national level. For some, experience might be limited, or there are no standard involvement practices in place. By collaborating in a EUnetHTA assessment, national agencies could gain expertise with involving patients and patient organizations. This could then influence their processes at the national level, because some might not have involved patients so far due to insufficient awareness of the value of patient involvement or the lack of any recommended involvement approaches. However, given that reimbursement decisions remain in the national autonomy and are not part of the EUnetHTA assessment, the context of patient involvement could be different on a European level than on a national level. In EUnetHTA assessments, the focus of patient input lies more on understanding the patient perspective. In a national context, patient input could be used in the appraisal process, for example as arguments in the reimbursement decision-making process.
Summary
Although it is widely recognized that patient involvement throughout the medicine life cycle is important to understand the needs of the target patient population, challenges remain with timely patient involvement. Additionally, further efforts are needed to guide assessment teams on how to implement and enhance the visibility of patient input in EUnetHTA assessments.
You have
Access
Background
The European Network for Health Technology Assessment (EUnetHTA) was established to enable and support the production and use of jointly produced Relative Effectiveness Assessments (Reference Erdös, Ettinger, Mayer-Ferbas, de Villiers and Wild1). Its aim is to create an effective and sustainable network for Health Technology Assessment (HTA) across Europe as stated in the Cross Border Directive (2011) (2). This directive provided the political and regulatory framework for EUnetHTA Joint Action 1 (JA1) and the succeeding JA2 and JA3, which are (partially) funded by the European Commission. The main objective in JA1 was to establish an effective and sustainable HTA collaboration in Europe (3), whereas the objective in JA2 was to strengthen the practical application of tools and approaches (4). In JA3, the main objective was to define and implement a sustainable model for cooperation on HTA across Europe after the end of the project period (5).
The EUnetHTA assessments focus on the clinical aspects only, summarizing the available evidence on effectiveness and safety and comparing the technology under assessment to other existing alternatives (6). Hence, EUnetHTA assessments do not provide recommendations on added value or reimbursement of a technology; such reimbursement decision remains within the national autonomy of a country.
EUnetHTA recognizes that different groups and organizations can provide key information, expertise, and relevant experience to the production of EUnetHTA assessments. For this reason, the EUnetHTA Joint Action Stakeholder Policy was developed during JA1 to facilitate a transparent and responsible stakeholder involvement process (7). According to the Stakeholder Policy, the appropriate involvement of stakeholders has to be defined according to the work in question (7).
In EUnetHTA JA2, only one of six assessments on pharmaceutical technologies (PT) and three of six assessments on other technologies (OT) involved patients or consumers. In addition, it revealed difficulties in recruiting patients and that the workload was overwhelming for involved patients (e.g., review draft assessments). To support the development of a sustainable process for patient involvement in EUnetHTA assessments and Early Dialogues, the EUnetHTA Task Group on Patients, Consumers, and Health Care Professionals (TG P/C&HCP, hereafter referred to as TG) was established. The TG consisted of representatives from work groups on Network Coordination, Dissemination, Joint Production, Life Cycle Approach to Improve Evidence Generation and Quality Management, Scientific Guidance and Tools (5), and was led by the EUnetHTA Senior Scientific Officer. It was important for the TG to build on the experience and knowledge that was already available in EUnetHTA and to agree on some preferred approaches for patient involvement that could be applied within EUnetHTA assessments. Experiences with patient involvement in assessments were recorded before, during, and after the TG formed recommendations for patient involvement in EUnetHTA assessments. The recommendations are described in the document “Patient Input in Relative Effectiveness Assessments” that is available for access, and this document was published in 2019 (8). Stakeholders representing European patient and consumer organizations, as well as other organizations from the HTA network stakeholder pool, were consulted with regard to the recommendations for patient involvement (8).
This article aims to provide a comprehensive overview of the experience with patient involvement gained by EUnetHTA in EUnetHTA assessment production from June 2016 until the end of November 2019, by describing and quantifying the approaches used and outlining the challenges and avenues for improving current processes. At this time point, the majority of the project period of EUnetHTA JA3 has passed and this allows for capturing learning points. The views of patients who were involved in EUnetHTA assessments were not systematically gathered at the time of writing this manuscript, and are, therefore, not part of this article.
Methods
Assessments related to PT and OT (e.g., medical devices, nonpharmaceutical procedures, etc.) were reviewed, and patient involvement approaches and experiences were quantified and described separately due to differences in organization of the PT and OT assessment processes.
Assessments related to PT and OT conducted from June 2016 until the end of November 2019 within EUnetHTA JA3 were reviewed. The assessments were either finalized or in an advanced stage of the scoping phase when information was collected. Assessments in which stakeholder involvement was not sufficiently advanced were not included in this article. Information about patient involvement procedures was retrieved from the EUnetHTA Web site and by contacting EUnetHTA project managers.
Experiences with patient involvement were collected from EUnetHTA project managers. The project management in PT assessments is done centrally, and information on patient involvement is kept in a database. In OT assessments, the project management can be done centrally or can be conducted by selected activity centers in a decentralized way. In both cases (centralized and decentralized project management), a set of questions was sent to respective OT project managers to gather relevant information. If needed, follow-up or clarifying questions were posed to the project managers.
Approaches for patient input
By reviewing assessments from June 2016 until the end of November 2019 and assessing information about patient involvement approaches, it was revealed that twelve out of fourteen PT assessments and eleven of twenty-two OT assessments successfully involved patients. All fourteen PT assessments attempted to seek patient input and only two assessment teams were unsuccessful in involving patients (Figure 1a). In OT, fifteen assessments attempted to seek patient input and eleven were successful in including patients (Figure 1b). Seven OT assessments did not attempt to seek patient input for various reasons. In this article, successful involvement is related to assessment teams’ perspective and means that patients could be identified/recruited and/or the pursued approach was completed, and unsuccessful involvement means that no patients could be identified/recruited and/or the pursued approach was not completed.
(a) Patient input in PT assessments. (b) Patient input in OT assessments.
The TG reviewed existing procedures and (grey) literature in regard to patient involvement approaches and collected relevant experiences from TG members. An overview with possible approaches was created, where further information was outlined: a description of the approach, the preferred time point, the target group, resource use, the expertise required by the assessment team and the patient, advantages, challenges and limitations, and additional considerations relevant for the PT and/or OT production process. Subsequently, these approaches were discussed in the TG e-meetings focusing on the applicability and feasibility within EUnetHTA. This method did not aim to be exhaustive in listing all possible patient involvement approaches but aimed to identify common practice in both (previous) EUnetHTA assessments and national HTA experiences. Based on this initial examination and discussion, five main approaches to obtain patient input in EUnetHTA assessments during JA3 (Figure 2) were identified and outlined in the document “Patient Input in Relative Effectiveness Assessments” (8): (i) (online) patient input template, (ii) one-on-one conversation, (iii) group conversation, (iv) participation in scoping e-meeting, and (v) other approaches. In several assessments, more than one approach was used. However, if one approach was used multiple times within a single assessment, it was still counted as one.
Pursued approaches for patient involvement.
For PT assessments, pursued approaches for patient involvement were (online) patient input template (n = 12), one-on-one conversation (n = 4), and other approaches (n = 1) (Figure 2). Of these, four did not succeed in involving patients, due to nonresponse or unwillingness of patient organizations to participate. This was the case for the online patient input template (n = 2), one-on-one conversation (n = 1), and other approaches (i.e., ranking of outcomes, n = 1). In two PT assessments, more than one approach was pursued.
For OT assessments, pursued approaches for patient involvement were as follows: (online) patient input template (n = 6), one-on-one conversation (n = 5), group conversation (n = 2), scoping e-/f2f meeting with patients/patient representatives (n = 1), and other approaches (i.e., a review of the draft Population, Intervention, Comparator and Outcomes (PICO), review of draft project plan or draft report, involvement as a team member, the use of national surveys, n = 9) (Figure 2). Not all led to successful patient involvement in assessments. This was the case for the use of the patient input template (n = 3), one-on-one conversation (n = 1), and other approaches (a review of scope or feedback on the selection and importance of outcomes, n = 3).
Experiences with patient input
Figure 3 presents a simplified process flow of the EUnetHTA assessment production process and shows the necessary steps for the patient involvement approaches.
Process flow of the EUnetHTA assessment production.
Experiences from PT
The responsible project manager coordinated the identification of patients and/or patient organizations to be involved in the PT assessments and documented this in a database. Patients or organizations were identified based on suggestions from the European Medicines Agency (EMA) (n = 11) via an Internet search (n = 4), via the HTA Network Stakeholder Pool (n = 14), or via other approaches (n = 3) such as taking suggestions from the authoring team or selecting organizations based on previous assessments. For nine assessments, several identification approaches were used.
In total, over thirty-six patient organizations were contacted for the fourteen PT assessments, and in twelve assessments, patient involvement approaches were successfully applied. The reasons for not including patient input included a lack of response, too tight timelines to identify suitable patients and/or patient organizations, or unwillingness from patient organizations to participate in assessments.
In eleven assessments, the patient input template was used as a basic approach in an early phase of the assessment, to ensure that patient input was gathered at the time of drafting the assessment scope. If deemed relevant by the authoring teams, the patient input template could be complemented with different approaches. In two assessments, this was the case (i.e., ranking of outcomes (n = 1) and one-on-one conversation (n = 1)). The median number of patient organizations that responded to the online patient input template was two patient organizations (range: 0–7). In the eight assessments in which more than one patient organization completed the online call for patient input, these organizations came from different countries. The patient organizations provided a synthesized patient view. This allows capturing divergent views due to national differences in, for example, the treatment pathway. In all assessments, the online patient input template was in English and responses in the national language of the respective authoring team were allowed. In two out of these eleven assessments, none of the approached patient organizations completed the patient input template.
Four PT assessments attempted to identify individual patients for a one-on-one conversation. For this approach, the project managers reached out to the patient pillar of the HTA Network Stakeholder Pool (n = 3) or the author identified national patients (n = 1). The identification of individual patients was very challenging, due to a tight production timeline, the burden of disease on patients, as well as the inclusion of a specific population group in the assessment. In one of these four assessments, the attempt to conduct a one-on-one conversation was unsuccessful, due to the time constraints and the burden of disease on patients. In three out of these four assessments, one individual patient was identified for each assessment. In two assessments, involvement occurred in an early phase of the scoping phase (i.e., the drafting of the research question), whereas in the other assessment, it was in a late stage (i.e., the finalization of the assessment report). The one-on-one conversation was conducted by the Project Manager (n = 1) or the author (n = 2) and was conducted in the local language of the patient (n = 1) or in English (n = 2). The experience was that a one-on-one conversation with individual patients in the scoping phase was more helpful than such a conversation later in the assessment phase. In the assessments where the one-on-one conversation approach was used, the experience was that there was uncertainty on how to interpret the results with only one single patient and how to account for cultural deviations. The patient input template, however, is believed to enhance the use of the patient input, because it represents the view of a broader group rather than the input from one individual patient. In addition, the online patient input template also ensured that various national perspectives are heard, as in the majority of assessments, more than one country was represented.
Experiences from OT
Sixteen OT assessment teams reported information about patient involvement in assessments through a set of questions when those were either finalized or in an advanced stage of the scoping phase. The questions were not sent to the project managers of six assessments, because the centralized project management already received the information beforehand that five assessment teams were not able to involve patients or patient organizations due to tight timelines and because one assessment team sent the information on successful patient involvement approaches via e-mail. In the following, information from the sixteen questionnaires that were sent to the assessment teams is presented.
Out of sixteen assessments, fourteen assessment teams planned patient involvement. One assessment team contacted individual patients, seven assessment teams contacted organizations, and six assessment teams contacted both individual patients and patient organizations. In nine of fourteen assessments, both national and European patients and patient organizations were approached. In total, the fourteen assessment teams contacted over 40 patient organizations and several individual patients (some were directly approached via patient organizations). The reason that two assessment teams did not plan for patient input was a tight timeline and a lack of a specific patient group organization. Patients or organizations were identified after receiving suggestions from coauthors (n = 3), dedicated reviewers (n = 6), external clinical experts (n = 5), via an Internet search (n = 8), via previous experiences at their HTA agency (n = 8), or via other identification approaches (n = 7). Other identification approaches included suggestions from a patient support expert, from a European patient organization, from own networks, or from the Ministry of Health. Several assessment teams used more than one identification approach. Selection was based on PICO relevance.
Ten assessment teams managed to recruit and include individual patients or patient organizations in their assessment. The reasons for unsuccessful recruitment of patients included a lack of response or unwillingness to participate, which was the case in four assessments.
The patient input template was successfully applied in three assessments, whereas one of them was published in the form of an open call for patient input on the EUnetHTA Web site (one patient organization completed the template); for the other two assessments, the template was sent to relevant patient organizations via e-mail. In one case, however, a statement was received back instead of a completed template, and in the other case, two patient organizations completed the template. On three occasions, the template could not successfully be used, whereas one of them was published on the EUnetHTA Web site as well, but did not result in any response from a patient organization, despite several efforts in contacting relevant patient organizations in order to make them aware of the open call.
In two assessments, group conversations were performed, whereas one assessment included two separate groups, one with adults and one with children and their carers. The group conversations included five or more individuals per group. One-on-one conversations were performed in four assessments, where one and up to five individuals were involved in each of the assessments. All group and one-on-one conversations were done in the local language of the patients. In one assessment, a scoping meeting was performed, and in six assessments, other approaches like a review of the draft PICO, a review of the draft project plan or draft report, participation as an assessment team member, or the use of national surveys were used. Seven assessment teams reported positive experiences with patient involvement, and one assessment team narrated its experience with one-on-one conversation in the following way:
“It was a really interesting experience because after the interview there is a sensation of 'closeness' to the patients suffering from the condition. It helps to understand who is benefiting (or not) from the technology under assessment.”
Management of patient input
In both PT and OT, the assessment teams primarily used patient input to inform outcomes. Sometimes, they also used patient input in the GRADE—Grading of Recommendations, Assessment, Development and Evaluation—ratings, in discussions, or as a supplement to information provided in the literature. Some assessment teams provided a summary of patient input in the assessment and added all of the inputs in appendix to the assessments.
Suggestions for improvement of patient involvement in assessments included standardization of patient input procedure, specific criteria for the use of group conversations, and request for assistance in the identification of patients, or a pool of relevant patients or patient organizations to contact. One assessment team that was unable to include patients or patient organizations in the scoping phase noted that patient input in the assessment phase also proved to be useful.
Challenges and avenues for improving current processes
It is widely recognized that patient involvement throughout the medicines life cycle is important to understand the needs of the target patient population. However, patient involvement in national HTA activities varies. Some HTA agencies engage patients in their HTA activities but not always routinely (Reference Hailey, Werkö, Bakri, Cameron, Göhlen and Myles9). To bring the patient perspective closer to the center of medicine development, authorization, HTA, and reimbursement processes, many initiatives have been started on a European Union level. For instance, under the Innovative Medicines Initiative (IMI), public-private collaborative initiatives specifically on patient involvement have been launched, such as the IMI-PREFER (10) or the IMI-PARADIGM (11) research projects. Within EUnetHTA, the effort to strengthen patient involvement was initiated during JA2, although it was during the current JA3 that criteria and processes for collecting patient input were well defined and structured (8). PT and OT piloted different patient involvement approaches in their assessments that led to EUnetHTA-recommended approaches for patient input and visibility of patient involvement (8), an (online) patient input template (8), a patient information flyer (8), and an evaluation questionnaire for patients. Even though EUnetHTA has now developed recommendations for patient input, it does not prevent EUnetHTA partners from using other approaches for patient input in HTA assessments. EUnetHTA partners could choose the approach that fits best to the topic under assessment, considering national practices and the EUnetHTA recommendations.
For assessments in PT, it is mandatory to seek patient input. The (online) patient input template is now used as a basic approach, and this can be complemented with other approaches. In OT, it is obligatory to discuss patient involvement in the assessment team and to provide a sound rationale in case their involvement is not feasible or is not pursued due to other reasons. The approaches to be used are to be agreed upon within the assessment team. In some instances, the defined population might not have a clear (patient) organization or self-help group behind them. The identification process of individual patients or patient organizations depends on the topic. The assessment teams’ experiences showed that the identification of individual patients and patient organizations proved to be challenging as many organizations did not respond or were not willing to participate. The project managers and assessment teams tried to use different existing sources like the pool from the EMA in PT and the HTA Network stakeholder pool in PT and OT, in addition to searches on the Internet. They aimed at involving both European and national patient organizations. For national organizations, national databases for patients or patient organizations (if existent), national contacts, and the assessment team members often approached them in their national language, which facilitated the development of contacts. To facilitate the recruitment process, EUnetHTA created a patient flyer (8) to inform patients and patient organizations about EUnetHTA and how they can contribute in EUnetHTA assessments. To further facilitate the identification and recruitment of patients for involvement in assessments, some of the assessment teams suggested that EUnetHTA could consider a database of patients/patient organizations, which could be used as an additional source or as an approved list like a “patient and consumer pool.” In addition, it could be explored if a stakeholder involvement officer would facilitate the identification of relevant patients/patient organizations within EUnetHTA assessments. A wide consultation and collaboration between patient organizations and EUnetHTA could also help clarify why a lack in response and difficulties in collaborating with patient organizations arises.
Which assessments successfully involved patients is visible on the EUnetHTA Web site (12), because a respective patient sign is placed next to the title of the assessment in case of actual involvement. Both PT and OT assessment teams provided a summary of the patient input in the assessment if the teams had access to patient validated summary and/or key statements. Some PT assessment teams tried to actively show which outcomes were stressed by patient organizations. In OT, some assessment teams also added patient input in an appendix to the assessments. It can be difficult to measure if patient input has a direct and quantifiable impact, but the assessment teams reported that patient input had an indirect impact on the assessment process because they better understood the patient situation and experiences. Experiences from PT assessment teams in EUnetHTA showed that there were challenges associated with using patient input if it was obtained from one individual patient, as they felt that this was not a generalizable perspective to other patients of the same nationality or a different nationality. Others also argue that the patient perspective may depend on their experience with the health system and that it could be influenced by the social and political environment that the patient lives in (Reference Carman, Dardess, Maurer, Sofaer, Adams and Bechtel13). However, regardless of their nationality and social/political background, patients bring experiential knowledge of living with the disease (Reference Gagnon, Desmartis, Lepage-Savary, Gagnon, St-Pierre and Rhainds14) and can, therefore, inform which outcomes are most relevant for them and provide information about the treatment process. Also, the EMA reported after having piloted engaging patients in the CHMP Benefit/Risk discussions that patient input of two individual patients was very valuable for the scientific discussion due to the patient's experience of living with the disease (15). Understanding the societal burden across countries and cultures could, therefore, be beneficial for assessment teams when interpreting patient input to the PICO. How to interpret patient input is currently not streamlined across assessment teams, but a group within EUnetHTA is working on a guidance on how to use the patient input among assessment teams. This is in line with one of the outcomes of the Fourth Edition of the European Network for Health Technology Assessment Forum, which called for a more robust measurement of contributions and visibility of the added value given by patients in the assessment process (Reference Tafuri, Bélorgey, Favaretti, Frénoy, Giorgio and Guardian16). EUnetHTA assessments are very technical and may not be easily understood by patients. Consequently, EUnetHTA came up with a plain language summary template that reflects the needs of EUnetHTA. It is currently being piloted and the aim is to use it for upcoming PT and OT assessments. Plain language summaries may empower patient populations (17), offer a valuable tool to communicate to the public, create awareness, and increase value for assessments.
How EUnetHTA experiences can influence national patient involvement practices
To the best of the authors’ knowledge, much research has been conducted on mapping methods for national stakeholder involvement in HTA activities or guideline development, but only limited research has been done on how patient involvement could be incorporated in international collaboration on HTA activities. However, the main challenges for patient involvement identified by PT and OT assessments are identifying patients or patient representatives and obtaining representative input and burden of work, and these correspond to challenges reported in studies on national HTA activities (Reference Hailey, Werkö, Bakri, Cameron, Göhlen and Myles9) and guideline development (Reference Eccles, Grimshaw, Shekelle, Schünemann and Woolf18).
Not all EUnetHTA partners have experience with patient involvement at a national level. For some, experience might be limited, or there are no standard involvement practices in place. By collaborating in a EUnetHTA assessment, national agencies could gain expertise with involving patients and patient organizations. This could then influence their processes at the national level, because some might not have involved patients so far due to insufficient awareness of the value of patient involvement or the lack of any recommended involvement approaches. However, given that reimbursement decisions remain in the national autonomy and are not part of the EUnetHTA assessment, the context of patient involvement could be different on a European level than on a national level. In EUnetHTA assessments, the focus of patient input lies more on understanding the patient perspective. In a national context, patient input could be used in the appraisal process, for example as arguments in the reimbursement decision-making process.
Summary
Although it is widely recognized that patient involvement throughout the medicine life cycle is important to understand the needs of the target patient population, challenges remain with timely patient involvement. Additionally, further efforts are needed to guide assessment teams on how to implement and enhance the visibility of patient input in EUnetHTA assessments.
Acknowledgments
We thank Giovanni Tafuri, who was leading the EUnetHTA P&C/HCP TG until April 2020, for reviewing and providing valuable inputs to the draft manuscript.
Funding
This work received funding from the European Union's Health Program (2014–2020) (grant number “724130 / EUnetHTA JA3”).
Disclaimer
The content of this paper represents the views of the authors only and is their sole responsibility; it cannot be considered to reflect the views of the European Commission and/or the Consumers, Health, Agriculture and Food Executive Agency or any other body of the European Union. The European Commission and the Agency do not accept any responsibility for use that may be made of the information it contains.