The Spanish Network of Agencies for Assessing National Health System Technologies and Performance (RedETS) is an instrument to inform health policy decisions from the Spanish Ministry of Health on the technologies and provisions to be included in the Common Services Portfolio of the National Healthcare System. RedETS was officially created in 2012, but has been driven and funded by the Spanish Government since 2006 (Reference Serrano-Aguilar, Asua-Batarrita, Molina-López, Espallargues, Pons-Rafols and García Armesto1). Among other tasks, RedETS informs decision making on how to allocate funds regarding nonpharmaceutical technologies, helps to determine the best conditions for their funding and appropriate use, and identifies candidate technologies for disinvestment (2).
Health technology assessment (HTA) organizations are increasingly incorporating patient involvement (PI) into their assessment processes (Reference Lopes, Street, Carter and Merlin3). PI is an inclusive term referring to the incorporation of people with health problems, their family and caregivers, users of healthcare technologies or citizens, and patient representatives in the HTA process (Reference Toledo-Chávarri, Perestelo-Pérez, Álvarez-Pérez, Abt-Sacks, Santoro Domingo, Villalón and Trujillo4;Reference Facey, Hansen and Single5). PI can provide relevance, balance, justice, and legitimacy to those assessments (Reference Toledo-Chávarri, Perestelo-Pérez, Álvarez-Pérez, Abt-Sacks, Santoro Domingo, Villalón and Trujillo4;Reference Facey, Hansen and Single5). Moreover, it can lead to a better HTA by allowing collaboration in the identification and prioritization of what has to be evaluated, introducing experiences of living with illnesses and technologies; in short, it includes the patient's perspective to complement clinical and healthcare system standpoints (Reference Barr, Scholl, Bravo, Faber, Elwyn and McAllister6). Despite the growing interest in the relevance of incorporating perspectives, experiences, and values of patients in HTA processes and products (Reference Toledo-Chávarri, Perestelo-Pérez, Álvarez-Pérez, Abt-Sacks, Santoro Domingo, Villalón and Trujillo4;Reference Facey, Hansen and Single5), few studies have assessed the process and impact of PI strategies (Reference Royle and Oliver7–9). In a recent international survey, it was observed that approximately half of the HTA agencies that systematically incorporate patients in their product development performed process and/or impact evaluations (Reference Weeks, Polisena, Scott, Holtorf, Staniszewska and Facey10).
In the last 10 years, RedETS has worked to include patients in its activities, both those related to HTA reports and the development of clinical practice guidelines and other evidence-based products (Reference Toledo-Chávarri, Perestelo-Pérez, Álvarez-Pérez, Abt-Sacks, Santoro Domingo, Villalón and Trujillo4;9). In HTA, some relevant initiatives for PI have been the development of recommendations on how to adapt the content of HTA reports for patients (11) and the compilation of prior PI experiences within the network (Reference Bayón, Reviriego, Gutiérrez and Galnares-Cordero12).
In 2017, RedETS published the methodological framework for PI in HTA and, together with the Spanish Ministry of Health, a public declaration was issued on the gradual strategy for the involvement of patients in the HTA process at a national level (2;Reference Toledo-Chávarri, Perestelo-Pérez, Álvarez-Pérez, Abt-Sacks, Santoro Domingo, Villalón and Trujillo4). This declaration recognizes the need and value of active participation and collaboration of patients, caregivers, and users to improve decision making about nonpharmaceutical technologies taken within the Spanish Healthcare System. With this purpose in mind, six of the eight RedETS units and agencies implemented PI processes in their 2017 HTA annual work program.
The aim of this study was to assess, from the HTA researcher's perspective, the processes and impact of PI initiatives that were implemented in the first year following the publication of the methodological framework.
Methods
Design
PI experiences implemented during 2017 (n = 6) were presented at the RedETS annual meeting in Tenerife in an internal workshop, in December of the same year. A preliminary discussion of each PI initiative was held. At the time, the experiences were at different stages of development. During the workshop, the need for a deeper evaluation of the PI initiatives' impact was identified. A survey was conducted with this aim.
The survey (Supplementary File 1, in Spanish) included open-ended questions and was divided into two parts. The first part was based on the Public Involvement Impact Assessment Framework Guidance (Reference Popay and Collins13) and included the following aspects: approaches to PI, focus on design, practical aspects, and impact of participation. The second part of the survey adapted and based on the work by Weeks et al. (Reference Weeks, Polisena, Scott, Holtorf, Staniszewska and Facey10) addressed the lessons learned, including barriers and facilitators perceived in the incorporation of patients and the general population in HTA (Reference Weeks, Polisena, Scott, Holtorf, Staniszewska and Facey10). The survey was piloted by one of the participants to test its readability, suitability, and applicability. All received comments were used for its improvement.
Data Collection Process and Participants
The aforementioned survey was emailed in September 2018 to the six RedETS members that implemented PI strategies in at least one of their products that year. The RedETS agencies/units were: the Agency for Evaluation of Healthcare Technologies in Andalusia (AETSA), the Agency for Healthcare Quality and Assessment of Catalonia (AQuAS), the Scientific-Technical Advice Unit, Galician Agency for Health Knowledge Management (Avalia-t), the Basque Office for Health Technology Assessment (Osteba), the Evaluation Unit of the Canary Islands Health Service (SESCS), and the Healthcare Technologies Evaluation Unit of Madrid (UETS).
PI was considered as any activity aimed at consulting or involving patients, family members, and/or general population undertaken in the context of RedETS projects. Responders could give multiple answers if PI was implemented in more than one project.
Data Analysis
The responses to each survey were reviewed and analyzed qualitatively by two of the authors. For this purpose, an adapted thematic analysis, which meant that answers were not codified but were categorized and summarized in themes in a first descriptive analysis, was performed and reviewed by all authors. A second order interpretation was discussed by means of a teleconference, and the final analysis was reviewed and approved by all the authors.
This study did not require formal approval of a research ethics committee, given that the approach was focused on procedures of organizations and patients did not participate.
Results
The six invited agencies/units responded (6/6; 100 percent response rate) and provided information on nine different RedETS evidence-based products in which PI were implemented. There were seven HTA reports (Reference Bayón, Reviriego, Gutiérrez and Galnares-Cordero12;Reference Baños-Álvarez, Molina Linde and Benot López14–Reference Vallejo Torres, Linertova, Castilla, Ramos García, Castilla Catalán and Barricarte Gurrea19), one methodological report (Reference Almazán, Arroyo and Astorga20), and one evidence-based protocol (21). The participants were six senior HTA researchers with different levels of expertise in PI.
PI Characteristics
Table 1 shows the characteristics of PI projects, including the following information: the agency/unit responsible for the report, the bibliographic reference of the report, the product type (assessed technology), the aims of PI, the characteristics of patients involved, the phase of the reports' development in which they participated, and the methodology for participation.
Characteristics of PI processes
AETSA, Agency for Evaluation of Healthcare Technologies in Andalusia; AQuAS, Agency for Healthcare Quality and Assessment of Catalonia, Avalia-t, Scientific-Technical Advice Unit, Galician Agency for Health Knowledge Management; WGEDCD, Working Group for the Early Diagnosis of Celiac Disease Protocol; Osteba, Basque Office for Health Technology Assessment; SESCS, Evaluation Unit of the Canary Islands Health Service; UETS, Healthcare Technologies Evaluation Unit of Madrid.
The main and specific aims of PI were inclusion of the perspectives, experiences, and values of patients on the technology that was being assessed (n = 6), discovering the important outcome measures for patients (n = 5), analyzing barriers, facilitators, or suggestions for implementation (n = 2), and obtaining relevant information on legal aspects and other patient preferences (n = 2).
The methodology for PI included surveys, focus groups, semi-structured interviews (telephone and face-to-face), and patient participation on expert panels.
The technologies assessed included: the depression in young people (TIDY) program, transcatheter aortic-valve implantation (TAVI), nipple-areola complex micropigmentation for women affected with breast cancer, platelet-rich plasma (PRP), early detection of disease-related malnutrition, pneumococcal vaccines, celiac disease early diagnosis, leadless pacemaker, and health apps and other mobile health solutions.
Lessons Learned, Challenges, and Opportunities
The following five main themes emerged from the analysis: (i) recruitment challenges, (ii) needs identified in the PI pilot process, (iii) PI impact, (iv) lessons learned, and (v) suggestions for the future.
Recruitment Challenges
Patient recruitment was perceived as the most challenging aspect of PI in all cases. Active tactics to recruit patients were put in place. On the one hand, engaging committed clinicians (Reference Jurado López, Larman Telechea, Moreno Gómez, Rodríguez Salvanés and Novella Arribas17;Reference Triñanes Pego, Rial Boubeta, Varela Reboiras, Mazaira Castro and Atienza Merino18) or active patient associations with broad bases or professionalized organizations (Reference Bayón, Reviriego, Gutiérrez and Galnares-Cordero12;21) as brokers facilitated recruitment. However, not all patient associations responded to the invitation to participate. For instance, six patient associations were contacted at least twice by email for the PRP HTA report, but only two responded and neither of them decided to participate (Reference del Pino Sedeño, Trujillo Martín, Linertová, Toledo Chávarri, Andia Ortiz and Kaiser Girardot16). One patient organization declined to be involved in a report regarding pneumococcal vaccination (Reference Vallejo Torres, Linertova, Castilla, Ramos García, Castilla Catalán and Barricarte Gurrea19) citing lack of time and resources. Recruitment processes failed and PI was not possible in two reports (Reference Brito García, Trujillo Martín, García Pérez, Ramos García, León Sala and Kaiser Girardot15;Reference Vallejo Torres, Linertova, Castilla, Ramos García, Castilla Catalán and Barricarte Gurrea19).
On the other hand, direct recruitment of users was not free of difficulties either. In the case of the report on the use of mobile applications for the healthcare sector (Reference Almazán, Arroyo and Astorga20), direct recruitment of consumers without the mediation of patient organizations or clinicians was difficult and resulted in a low participation. In fact, only nine app users were finally included in the focus group.
The characteristics of the technology assessed and the target population defined the recruitment process and success. Recruitment was easier in technologies with a greater impact on patient experiences, values, or preferences or those that were used in more prevalent diseases. A hypothesis drawn from these results was that some technologies less relevant to patient experiences, values, or preferences made the recruitment process more difficult. For example, engaging patients to assess the values and preferences regarding personalized decisions to use TAVI (Reference Triñanes Pego, Rial Boubeta, Varela Reboiras, Mazaira Castro and Atienza Merino18) may have been difficult as they had had no knowledge or experience related to the technology.
Needs Identified during the PI Pilot Process
The PI processes identified a number of capacity building needs both for HTA researchers and patients. On the part of HTA researchers, the involvement of internal/external researchers with experience in qualitative research techniques was identified as a facilitator of PI in HTA. On the contrary, lack of expertise in PI or qualitative research experience was a barrier. In relation to patients, lack of knowledge regarding HTA and evidence-based products, their scope, assessment methods, and the ways in which they can engage in the process were identified. To counteract this aspect, in the leadless pacemaker HTA report, researchers developed easily readable explanatory materials to facilitate PI (Reference Popay and Collins13).
Unexpected consequences of PI were identified in two cases. In the first case, extra economic resources and time were required for travel (Reference Baños-Álvarez, Molina Linde and Benot López14). In the second case, a high number of responses meant an additional analysis workload, which was insufficiently planned (21).
Recruitment processes found some limitations regarding the representativeness of perspectives. None of the projects sought statistic representativeness, but a maximum variation of samples to include a wide range of experiences and preference was desired. In some reports, PI took place in local or regional contexts, whereby results may not be generalized to other contexts (Reference Bayón, Reviriego, Gutiérrez and Galnares-Cordero12;Reference del Pino Sedeño, Trujillo Martín, Linertová, Toledo Chávarri, Andia Ortiz and Kaiser Girardot16), as for example, access to technologies may differ according to region. This was the case of nipple-areola complex micropigmentation for women affected with breast cancer which was only funded in some of the Spanish regional Health Systems. In one case, recruitment by means of patient organizations might not have been the most adequate way to address information needs, as associated patients may have more information than most patients (Reference Bayón, Reviriego, Gutiérrez and Galnares-Cordero12).
PI Impact
All participants agreed that PI provided relevant information on patient and carer values, preferences, and experiences that enriched RedETS products (Reference Bayón, Reviriego, Gutiérrez and Galnares-Cordero12;Reference Baños-Álvarez, Molina Linde and Benot López14;Reference del Pino Sedeño, Trujillo Martín, Linertová, Toledo Chávarri, Andia Ortiz and Kaiser Girardot16–Reference Triñanes Pego, Rial Boubeta, Varela Reboiras, Mazaira Castro and Atienza Merino18;Reference Almazán, Arroyo and Astorga20;21). Moreover, patients and caregivers provided input, in some HTA reports, regarding the following aspects: the need to include the technology or procedure in the Spanish National Health System's (NHS) Provisions Portfolio (Reference Bayón, Reviriego, Gutiérrez and Galnares-Cordero12;Reference Triñanes Pego, Rial Boubeta, Varela Reboiras, Mazaira Castro and Atienza Merino18;Reference Almazán, Arroyo and Astorga20), the most important outcomes to be measured from their perspective (Reference Bayón, Reviriego, Gutiérrez and Galnares-Cordero12;Reference del Pino Sedeño, Trujillo Martín, Linertová, Toledo Chávarri, Andia Ortiz and Kaiser Girardot16), problems with respect to accessing the technologies, as well as ethical (Reference Baños-Álvarez, Molina Linde and Benot López14;Reference del Pino Sedeño, Trujillo Martín, Linertová, Toledo Chávarri, Andia Ortiz and Kaiser Girardot16–Reference Vallejo Torres, Linertova, Castilla, Ramos García, Castilla Catalán and Barricarte Gurrea19) and implementational (Reference Bayón, Reviriego, Gutiérrez and Galnares-Cordero12) aspects to be considered. In one case (21), patient representatives took part as authors, contributing to the design, preparation, and review of the report and its recommendations.
Nevertheless, the impact of PI on HTA reports and other evidence-based products was notably heterogeneous. Contributions to some reports were perceived as more limited by HTA researchers. Furthermore, collecting relevant information that improves the report seemed easier in technologies with a greater impact on patient experiences or that were used in more prevalent diseases. Contributions concerning technologies less linked to patient experiences, preferences, and values were more general and more related to their disease, but less specific to the evaluated technology. Therefore, those contributions were less able to provide input to the comparison between evaluated technology and usual care. This variability on impact led to a discussion regarding the relevance of including PI in all HTA and evidence-based reports, and which methods would be suitable in different cases.
Regarding the impact of PI, the information obtained from patients and carers provided the researchers and clinicians involved in the development of the HTA with a wider perspective of the assessed technologies. Patients highlighted key outcomes that were not always common to those pointed out by clinicians (Reference Baños-Álvarez, Molina Linde and Benot López14). Participation also meant an opportunity for training in qualitative research methodology for some RedETS researchers.
According to those who responded to the survey, patients felt that their opinion was considered when evaluating inclusion of the technology in the NHS (Reference Bayón, Reviriego, Gutiérrez and Galnares-Cordero12), and in some cases, they actively disseminated the results of the process (21).
Lessons Learned
Overall, PI in HTA reports and other evidence-based products was feasible and satisfactory for the different stakeholders (researchers, clinicians, patients, and carers), and provided specific contributions which improved results. However, this required time as well as financial and technical resources that were not always available and should be integrated into the planning process.
PI strategies and methodologies were diverse in the RedETS agencies and units. This diversity was not always related to HTA reports or evidence-based products' objectives but also to the interests, resources, and capabilities of the agencies/units. HTA researchers identified training needs in obtaining patient input and qualitative research techniques.
Patient recruitment may be one of the most important challenges for PI and was identified as a clear barrier for PI mainstreaming. Collaboration with engaged patient associations and clinicians can facilitate the PI process. Most patients lacked awareness in the HTA process and how they can be involved in it. Information or training facilitated their engagement and the impact of their contributions.
Suggestions for the Future
Improvements were suggested in five areas related to the implementation of PI strategies: planning the PI, patient invitation and recruitment processes, training for HTA researchers and patients, securing resources, and dissemination. The suggestions are shown in Table 2.
Suggestions for the future
A common roadmap or algorithm for PI implementation in RedETS products was suggested in order to improve the planning of PI. This roadmap or algorithm would help to transparently decide and prioritize ways in which RedETS products PI is needed or should be more intensive. This decision-making process should be reported in the publications for increased transparency.
Regarding the invitation and recruitment processes, participants suggested the promotion of wider and more diverse involvement. Three proposals were made in this regard: to search for ways to strengthen and standardize the invitation process through a common procedure; to build collaborative networks for patient recruitment among RedETS agencies/units; and to develop innovative forms to facilitate the recruitment process.
Capacity building development for PI of both patients and HTA researchers was another suggestion. For this purpose, resources should be allocated both to facilitate PI and for capacity building. Other technical needs were also mentioned, such as accessibility to software for qualitative analysis. Capacity building among HTA researchers would require engaging external experts to train them in participatory approaches and qualitative methodology. Capacity building among patients would require the development of educational materials and training actions in collaboration with patients and their representatives.
A final suggestion was related to increasing the dissemination of the results of the report among patients and their representatives at regional and national levels.
Discussion
PI processes implemented in RedETS were adapted to the type of product (HTA reports or other evidence-based products), the assessed technology and the health problem they were targeting, their specific aims, as well as the individual needs and resources of each HTA agency or unit. This evaluation has allowed the collection of relevant information on different aspects mainly related to the processes (practical and procedural), evaluation of the impact from the HTA researchers' perspective, in addition to important aspects that should be considered in future PI initiatives.
Overall, the results obtained are similar to those from recent studies evaluating PI in HTA (8;9;11). As for the most contextual factors, the importance of having sufficient human and financial resources and HTA researchers with experience in PI was noteworthy. The most important challenges also appear to be similar to those identified in other studies and related to patient recruitment and the effective contribution of patients. In this respect, the need to continue evaluating PI initiatives nationally is highlighted. Organizations such as the National Institute for Health and Care Excellence (NICE) reported that they are promoting PI evaluations for the entire organization, not only in HTA; the Canadian Agency for Drugs and Technologies in Health (CADTH) has introduced formal evaluations on participation processes every 5 years; whereas in France, the French Healthcare Authority (HAS, in French) has reported initiatives to broaden PI and its evaluation for HTA reports (Reference Royle and Oliver7;8;Reference Weeks, Polisena, Scott, Holtorf, Staniszewska and Facey10).
The results of the present study also show how PI has a relevant impact on HTA by providing key information on the experiences, values, and preferences of patients, contributions that benefit the HTA and the process of drawing up recommendations.
Some authors propose that including PI must be considered in all HTA reports, unless the evaluation concerns technologies that do not require direct interaction with patients and that, therefore, could not directly affect the patient experience (8). In any case, a more technocratic approach according to which patient participation is justified by the contributions they may make in the form of experiential knowledge must be complemented with a democratic approach that facilitates the transparency, opening up, and representation of a diverse range of experiences of disease (8;Reference Martin22). Other authors have also suggested the ethical obligation of including patients in the evaluation processes, bearing in mind that they are the end-users of healthcare technologies (Reference Esmail, Moore and Rein23).
One of the most important limitations of the present study is that PI initiatives have only been analyzed from the perspective of the RedETS researchers. Evaluation from the perspective of other relevant stakeholders, such as clinicians, managers, or the patients themselves has not been incorporated. Therefore, this may imply an analysis more related to procedural and methodological aspects (e.g., feasibility, resources needed, or methodologies to use) rather than other PI aspects (e.g., overall impact on decision-making process or overall efficiency of the process). However, it is worth noting that such broader issues are more difficult to evaluate (8).
In addition, the number of projects evaluated was limited and heterogeneous due to the different methodologies used and strategies adopted. Even if conclusions were drawn from rich responses to the survey and further discussions, they may not be generalizable to other contexts.
A way to move forward on improving processes of PI in HTA is by documenting and analyzing the results of different involvement strategies in order to contribute to increasing knowledge and practice during evaluation processes (Reference Facey, Hansen and Single5). In this context, the authors hope that this study provides information to strengthen PI in HTA. RedETS strives to optimize the process of PI in those areas in which opportunities for improvement have been identified. Training materials on HTA addressed to patients and citizens are currently being drawn up. In addition, an algorithm supporting the PI decision-making process is being developed and implemented. Future developments will enable the standardization of the process of PI within RedETS and the improvement of technical capacities to allow more meaningful patient contribution.
Supplementary Material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462320000586.
Acknowledgments
We would like to thank Marie-Pierre Gagnon for reviewing a draft of this paper.
Financial support
This work was performed in the framework of the Spanish Network of Agencies for Assessing National Health System Technologies and Performance (RedETS) and under its funding.
Conflict of Interest
The authors have no conflict of interests to declare in relation to this work.
