Healthcare decision makers across Europe have to make decisions on the national introduction, use or decommissioning of health technologies. As a result, all EU Member States basing their decisions on an informed process, need to collect and assess the same evidence about the effects of a technology. Some European countries developed public agencies for this kind of assessment providing evidence to decision makers. Nevertheless, the evaluation of technologies comes with a cost, as it is based on high professional profiles and it takes time. A closer collaboration among European HTA agencies, based on shared methods and tools, could avoid duplication of efforts, allow resource savings and represent a first step toward providing more uniform healthcare supplies throughout European countries also in the view of the Cross Border Healthcare Directive (24/2011/EU) (1). In fact, collaboration among Member States on HTA had been promoted by the European Commission which in the past had funded several projects starting since the middle nineties and the beginning of the noughties: EUR_ASSESS project (1994–97) (Reference Banta2) which established a common understanding of HTA and first identified the nine domains of the HTA Core Model (see below), HTA Europe Project (1997–98) consisting in an overview of HTA in European Countries (Reference Banta and Oortwijn3), and ECHTA/ECAHI Project (2000–02) analyzing the opportunity of a permanent coordinating structure for HTA in Europe (Reference Jonsson, Banta, Henshall and Sampietro-Colom4). From 2002 to 2005, some European developments at policy level paved the way for the establishment of the European Network for HTA (EUnetHTA) which started in 2006 with the objective of a more effective use of resources, increase HTA input into decision making, and support countries with limited experience in HTA (Reference Kristensen5;Reference Kristensen, Mäkelä, Neikter, Rehnqvist and Håheim6). The activities of the EUnetHTA has been organized by means of the EUnetHTA Project (2006–08) (7), the EUnetHTA Collaboration 2009 (8), and the EUnetHTA Joint Action 1 (2010–12) (9). The EUnetHTA Joint Action 1 aimed at finding strategies, tools and methods for an effective collaboration among European HTA agencies and comprised eight Work Packages. Work Package 4 on Core HTA was coordinated by the National Institute for Health and Welfare - Finland (THL) and co-coordinated by the National Agency for Regional Healthcare - Italy (Agenas). This Work Package was built on the earlier efforts of Work Package 4 of EUnetHTA project 2006–08 led by the Finnish Office for HTA (FinOHTA) which had developed the HTA Core Model, a novel method for producing and reporting HTA. The HTA Core Model enabled effective national and transnational production and sharing of HTA results in a common, structured format and from a wide range of perspectives as it consists of nine domains of research: health problem and current use; description and technical characteristics; safety; effectiveness; costs and economic evaluation; ethical; organizational; social; and legal aspects. For each domain, questions that could be asked and answered have been structured and methodologies highlighted. The actual assessment of a technology produced using the Core Model as a framework is called Core HTA. Its production suggests the search and examination of information and evidence on the technology at stake from several angles by answering a set of the questions provided in each domain (Reference Lampe, Mäkelä and Garrido10;Reference Pasternack, Anttila, Mäkelä, Ikonen and Räsänen11).
Work Package 4 on Core HTA's objectives were thus to produce, publish, store and retrieve structured HTA information, to develop two joint Core HTAs using the HTA Core Model and find and test methods for their joint production. Its work was organized into two strands. Strand A focused on transforming the above described HTA Core Model from a paper format to an online tool (HTA Core Model Online) with an electronic service for producing, publishing, storing and retrieving HTA information (On-line Tool & Service) and on a new application of it. Strand B's main deliverables were two real Core HTAs on topics relevant to several HTA agencies, to be produced by testing both the online tool and different collaborative models. This strand had three main purposes. First, to test the functioning of the HTA Core Model as a Web based toolkit. This meant that researchers from different countries had to learn how to use a same structured way to produce and store evidence and information, and exchange it, giving also feedback to new online toolkit developers. Second, to explore ways to make different European agencies’ researchers work together for the production of a Core HTA, according to specific collaboration models that had to be figured out and tested. These modalities had to be field-tested during the actual Core HTA production process and difficulties, problems and strengths of each had to be highlighted. A final purpose was to enhance collaboration among researchers who operate in different contexts, countries and have different backgrounds.
Our objective in this article was to provide a historical record of the activities of the strand coordinated by Agenas. We aimed at reporting what worked and what did not, in the coordination of groups of researchers with different backgrounds. In the Results section, we provide a description of all the joint Core HTAs production phases and an account of researchers’ and coordinators’ views of the production phases.
METHODS
For the description of joint Core HTAs production process, each phase (procedures for topic selection, collaborative models, organization of the works, etc.) was described and a historical account on how decisions were made was given. A retrospective analysis is provided of collaboration experiences of agencies participating in joint production of Core HTAs in the EUnetHTA Joint Action 1 project (2010–12).
To highlight what worked and what did not in the process, all primary investigators were contacted for a questionnaire based interview apart from the broader validation process (12). Each primary investigator works usually with two investigators and is responsible for research coordination and final text input in the online model relative to his/her domain.
Considering the qualitative nature of the information we wanted to collect, we decided to use a semi-structured interview supported by a questionnaire (Reference Cohen and Crabtree13–15). Available documentation (feedback questionnaires on Core HTA production, meeting's minutes, Interim Technical Reports, Surveys’ results, emails, and discussions in EUnetHTA team in Agenas, etc.) relative to EUnetHTA Joint Action 1 was analyzed to select topics for the questionnaire based interviews. Selection criteria were: issues that were not well represented elsewhere in available documentation, issues that were very relevant in the process of implementing the project, or issues that were not clear. The questionnaire was reviewed by the Work Package 4 leader and co-leader for major inconsistencies and then piloted with two interviews. The final questionnaire, which covers all production phases and not only the collaborative aspect reported in this article, is available upon request. An invitation to book an interview by means of phone, Skype, or Saba platform was sent to all seventeen primary investigators. One primary investigator acted as primary for two domains, so primary investigators were seventeen, instead of eighteen (nine domains for each Core HTA). For domains linked to more than one primary investigator one answer per question was requested after they conferred. A total of two reminders were sent. For those that could not book an interview, the questionnaire was collected by means of email. Estimated time for interviews was 30 minutes each. Coordinators point of views were gathered by internal meetings at Agenas. Main concepts arisen during meetings were shared and validated by coordinators.
RESULTS
Description of Joint Core HTAs Production Phases
The whole HTA process involves a first part where a technology has to be notified and then chosen according to explicit criteria and a second part where the technology is assessed (Reference Busse, Orvain, Velasco, Perleth and Drummond16). Our working group focused on both these steps: it developed a shared procedure to notify and prioritize technologies (procedures for topic selection) and proposed two different collaborative models to allow HTA agencies to work together during scientific assessment. A description of those two lines of activities is provided below.
Procedures for Topic Selection
A transparent procedure had first to be found for notification and selection of topics so that two technologies of common interest could be picked up for being assessed. This resulted in a complex pathway, made of different attempts, leading the group to finally agree on a procedure to select and choose topics.
First an attempt was made to use the Planned and Ongoing Projects database (POP database) from Work Package 7, Strand B (9), as a starting point, as this was already available. Joint Action 1 partners had to fill POP database with planned or ongoing HTA projects so technologies on which a project was planned could be selected and all planned projects prioritized according explicit criteria. Prioritization criteria established by the group at this point, were: degree of interest in Europe, comprehensiveness of the assessment, potential uses of the technologies, stakeholders pressures, burden of disease. A feasibility test for the use of POP database was made and it resulted not to be useful for the specific purposes, so a scoring of the criteria has not been carried out and a new procedure was drawn. POP database was indeed primarily aimed at allowing EU agencies to know what the others were doing or were about to do and, at that time, only few agencies had added their projects (Figure 1, first procedure).
Topic selection and priority setting procedures in EUnetHTA Joint Action 1.
Henceforth, the new procedure was followed, based on collecting notifications of technologies from all partners through a questionnaire. Three categories of technologies were identified on the basis of existing or under development HTA Core Model applications: medical and surgical interventions, diagnostic technologies and screening technologies.
The two categories of technologies with the higher number of notifications had to be selected and within those sets, the most notified technologies were chosen. In case of two technologies having the same number of notifications, prioritization criteria defined in the previous procedure, would have been applied according to their rank. A questionnaire was submitted to members to rank criteria and their application within the process. The order of ranking from the most important to the less was: burden of disease, potential uses, comprehensiveness, stakeholder pressure. The criterion degree of interest in Europe was not considered because it was indirectly applied by selecting technologies that were the most notified by EUnetHTA partners. Two topics emerged out: “Treatment of patients with back pain” (medical and surgical interventions) and “Genetic test for cancer” (diagnostic technologies) (Figure 1, second procedure).
These results were discussed during a meeting with all partners and were not endorsed, because selected topics were too generic and needed to be better specified. In addition, a need was expressed to test the new Core Model application on screening that was developed by Strand A and a more efficient use of human resources suggested testing this application while producing one of the two Core HTAs.
As a result, a new survey was made. The generic topics were sent by e-mail to their notifiers asking them to narrow. The list of all technologies, comprising the narrowed topics and the other ones resulting from the second procedure, clustered by technology's category, was sent to all partners asking simply to express their preferences. Answers from twenty-three partners of thirty-seven, with a response rate of 62 percent, identified two topics: “Genetic Test for breast cancer” and “Abdominal Aorta Aneurysm” for screening (Figure 1, third procedure).
Collaborative Models
The joint production was not only a way to train researchers in the use of Core Model online tool, but also the circumstance to test on field two ways of making researchers from different countries and backgrounds work together. The group theorized two different collaborative models relying on the previous work done by FinOHTA as leader of EUnetHTA 2006–08 Work Package 4 which had piloted the HTA Core Model with two topics and organized the production by assigning one (or more) domain to one agency (Reference Pasternack, Anttila, Mäkelä, Ikonen and Räsänen11) and on the experience Agenas had had in leading the Italian Network for Health Technology Assessment (RiHTA) (Reference Moirano, Cerbo and Lo Scalzo17). In the first model, partners willing to participate in the production process had to appoint one or more investigators in each domain. This meant that each domain team was made up of a cross national group of researchers. The basic idea was to form groups where both more trained researchers from long term established agencies and less trained ones were present. Agencies were asked to choose researchers with a specific expertise in the domain they were about to work in. In the second model, a whole domain was managed by a single agency with its team of experts. The two collaborative models were tested while producing the two joint Core HTAs to highlight probable limits and positive aspects of each of them. Both collaborative models were based on setting up domain teams, one for each of the nine Core Model domains. Model 1 was supposed to be tested during the assessment of the first technology at stake and Model 2 for the second one (Table 1).
Collaborative Models, Core HTAs, Domains and Partners in EUnetHTA Joint Action 1
Cur, health problem and current use of the technology; Tec, description and technical characteristics of technology; Saf, safety; Eff, clinical effectiveness; Eco, costs and economic evaluation; Eth, ethical aspects; Org, organizational aspects; Soc, social aspects; Leg, legal aspects.
Organization of the Joint Work
Two editorial teams were appointed for each Core HTA with the remit of taking major decisions and give approval to ongoing and final drafts. Each editorial team was formed by primary investigators from each domain, a Chair and a Vice-Chair. The Chair had a role of scientific coordination while Vice-Chair was middle-man contact among editorial team, the nine domain teams and Chair. The composition of each editorial team reflects the composition of the domain teams and so it includes investigators from different agencies and different countries.
A Project Manager from Agenas was in charge of coordinating domain teams work by facilitation of intra-domains communication and by managing projects in the On-line Tool & Service. The project manager assigned roles and functions to the different researchers involved in each project as primary investigator, investigators and reviewers (different “rights” for editing the information in the assigned domain). Human resources and timelines were managed according to the availability of each researcher involved and to the amount of person/days budgeted in the Joint Action 1 for the specific work package. Each Core HTA group defined: the title of the project (Core HTA 1: Genetic Test for Breast Cancer; Core HTA 2: Abdominal Aorta Aneurysm Screening), an abstract with scope of assessment based on Population-Intervention-Comparison-Outcome and a timeline of next steps. After the creation of the two projects in the online tool, each team created its Core HTA Protocol, including all assessment elements domain teams had selected as relevant. Core HTAs protocols were agreed through an internal discussion by means of e-meetings among primary investigators. Each domain's primary investigators shared information with the respective team. The selected assessment elements were then translated into research questions (one or more for each assessment element) to be answered both using scientific literature and other information sources (registers, manufacturer information, etc.). An important step in this phase was the definition of overlapping questions among domains: one research question could be answered by more domains. For this reason, coordinators organized e-meetings to reach consensus among primary investigators on overlaps management avoiding duplication of work. The Core HTA Protocols were then reviewed and shared with the Stakeholder Advisory Group (SAG) and comments and feedback added before locking the protocols.
For the research phase, a search strategy was agreed among primary investigators and a general literature search was made by one researcher in each Core HTA team. This was then shared with each domain team by email because the online tool did not provide methods for sharing this kind of electronic data. Domain teams needing further evidence retrieved new references and articles for the specific domain on their own. Word templates with research questions formulated by domain teams were used to facilitate exchange of information within each domain using the Word track changes. Information gathered from the literature search and from answers to questions made to EUnetHTA partners and manufacturers about the technologies were used to answer research questions, for example, Core HTA 1 used information collected from manufacturers. A final version of answers to research questions was entered in the online tool by primary investigators, after revision made by domains’ reviewers.
All assembled Core HTA information were published in the form of collections, each of which contains: (a) a set of result cards with concise answers to research questions, (b) general content (e.g., summary, introduction, and discussion) combining cards into a coherent information package, (c) appendices for any additional information to result cards. Then for each domain, a summary, introduction and discussion chapter was written by the domain teams and included in the online Core HTA. A Collection Summary and a Collection Introduction was also written by editorial teams as a general part of the Core HTA. Final versions of both Core HTAs were reviewed by a medical writer to homogenize text, because it was written by different authors with different styles.
POSITIVE AND NEGATIVE ASPECTS OF A JOINT PRODUCTION ACCORDING TO RESEARCHERS AND COORDINATORS
Researchers Views
The aim of the interviews was to draft a history and record of important decisions made and chronicle participants’ experience and views on those decisions during the EUnetHTA Joint Action 1. Of the expected seventeen final questionnaires and interviews, only twelve (70 percent) were accomplished by answering all the questions. Of the responding researchers, five have worked with collaborative model 1 (domain work shared across different agencies) and seven with model 2 (domain work produced within one agency). Following is a list of involved agencies: Agenas (Italy), THL (Finland 2), Gemelli (Italy 2), NICE (UK), HVB-LBG (Austria 2), ISCII (Spain), NOKC (Norway), AAZ (Croatia), Infarmed (Portugal). Of these, ten booked an interview and two only submitted the questionnaire by means of email.
Most relevant questions about topic selection and priority setting and collaborative models are reported in Figure 2. The majority (75 percent) of respondents thought topic selection and priority setting procedure was not clear and was not well managed (50 percent) and time consuming (66 percent). Ninety-one percent (91 percent) of respondents felt the need of early clinical expert involvement and the majority (Reference Pasternack, Anttila, Mäkelä, Ikonen and Räsänen11/12) thought they should be involved from the scoping phase and a few suggested also from the topic selection phase. Also, some suggested that stakeholders should provide a conflict of interest declaration. Others thought the list of criteria for topic selection and prioritization should be further supported by literature.
Primary investigarors views on topic selection and priority setting (TSPS) collaborative models in EUnetHTA Joint Action 1.
For collaborative models’ questions respondents, independent from their own experience, when asked if they thought the first model of collaboration (different agencies’ researchers develop one domain) should result in having a wider perspective, 25 percent agreed, 25 percent disagreed, and 50 percent neither agreed or disagreed or could not say. Furthermore, when asked if they thought that in a situation of scarcity of resources the first model would enable small agencies to participate, 50 percent agreed and the rest neither agreed or disagreed or could not say. A mix of the two models was proposed as a possible future development and 41 percent thought it could be possible. Instead when considering their own experience the second collaborative model (one agency develops one domain) was more satisfactory than the first (see Figure 3). During oral consultations, relevant comments were made about language barriers and organizational difficulties in collaborative model 1. Some also thought that whatever the model, small groups would enable a high level of exchange as long as there is a transparent knowledge of commitments.
Selection of primary investigators with collaborative models (CollMod) 1 and 2 in EUnetHTA Joint Action 1.
Coordinators Views
The experience of coordinating large and heterogeneous groups was challenging and allowed the co-leader to refine processes and procedures for future collaborative Core HTAs.
A first major obstacle highlighted by coordinators and encountered during the Joint Action project was availability of researchers. Even if the 3-years workplan and relative budget allocation was discussed and agreed within the EUnetHTA Network, some activities turned out to be more demanding (i.e., literature screening, assessment element overlapping, etc.) so that their accomplishment required an amount of time and work that has not been estimated at the beginning of the Joint Action. Shifting of researchers involved in Core HTA production was another major concern. As a researcher changed his/her job position within the agency or left it for a new position, his/her replacement in the research group came sometimes with a certain amount of difficulties both in terms of domain expertise or role (e.g., for some domains is more difficult to find trained researchers and finding new primary investigator) and experience in working with the Core Model.
Different languages, novelty of the online tool, and different ways of producing HTA were the main barriers coordinators’ identified as major obstacles during the research work. At the beginning of the HTA production process, different practices of performing HTA came to surface and misunderstandings arose about simple steps such as literature search. In some agencies, experienced external resources carry out this search, while in others this is done internally. So the group had to agree also on simple but fundamental steps like that. The development of specific tools for the collaborative production of Core HTA information (i.e., EUnetHTA intranet, Online tool and service, etc.) was useful for a productive collaboration even if some of these new tools needed an initial training period and an adaptation of the researchers way of producing HTA. Notwithstanding different languages, backgrounds, and ways of working on HTA, researchers had a very positive and keen attitude which facilitated the development and the production of the Core HTAs. People involved were always available to help and collaborate even for tasks not on schedule, which shows that human resources from participating countries are open and skilled for this cross national work.
DISCUSSION
The experience of producing two real Core HTAs using the online tool highlighted pros and cons for topic selection and the collaborative work. In general, our experience showed that the coordination of a collaborative work needed a strong project management effort as it had to consider not only the time, budget and resources constraints but also the existence of several partners with different expertise, background and language.
Topic selection procedure to select technologies for joint assessment resulted in a complex pathway, made of different attempts (Figure 1). What was clear at the end of the experience was that using explicit shared prioritization criteria to choose among the notified topics, complicated the procedure. It cannot be excluded that this was due to the experimental aims of the project. However, technologies for the two Core HTAs have been eventually selected among the panel of notified topics by involved partners expressing their preferences. By applying this simpler procedure, the real interest of partners was better captured. In the 2nd Joint Action on HTA, based on this experience, the topic selection procedure has been revised and streamlined. The working group identified a three steps procedure that should allow an easy and shared identification of topics to be assessed. The procedure consists now of a simple notification form, followed by a selection form in which participants can choose up to three notified topics; the most voted topic is then selected for the assessment. In the case of two or more topics with the same score, a further voting round is carried out.
With regard to collaborative models, they were built on the previous work of FinOHTA leading Work Package 4 in EUnetHTA 2006–08 and Agenas’ experience within RiHTA. Investigators thought the second model of collaboration, one Agency develops one domain, was more satisfactory. This may be to the absence of barriers such as language and working at distance, and the sharing of well-known work routines. Nonetheless, the practice showed that smaller and less experienced HTA Agencies had difficulty in developing an entire domain and lacked necessary expertise. Therefore, smaller agencies benefited from working alongside more experienced researchers as in the first collaborative model (more agencies work on one domain). However, for some highly specialized domains in HTA (e.g. legal, ethical) it was difficult to find a team of investigators from different agencies, and the one agency-one domain model could fit better in that case.
Our experience from the Joint Action 1 showed that a mix of the two models could be a possible solution for future organization of the work. Such a model allots work on the basis of agencies’ resources, rather than adopting only one approach, so some domains can be developed by a single agency, some others can be developed by investigators from different agencies allowing the mixing of different skill levels and expertise. While apparently more difficult to coordinate, we found the mixed working model to be relatively easy to implement and strong in output once the members of each team had been working together for a while. It provides a chance for “hands on” learning by junior members from their seniors.
CONCLUSIONS
In conclusion, this is the first overview of operating the Core Model online tool based on everyday practice. The topic selection and prioritization procedure is a sensible task, a quite simplified procedure was set up and it has been revised and streamlined in the 2nd Joint Action on HTA. Different collaborative models are possible, although the appropriate collaborative model should be chosen according to the interests, availability and expertise of partners. On the basis of our experience, the mixed working model may be the best compromise between efficiency of production and learning needs.
Collaborative production of Core HTA information by means of the online tool and service was useful, although the new tool needed an initial training period and an adaptation of the researchers’ way of producing HTA. The Core Model online showed to be a useful production platform. More standardized facilities could improve collaborative working processes.
CONTACT INFORMATION
Alessandra Lo Scalzo, M.Phil, MSc, Nicola Vicari, BA, Mirella Corio, PhD, MSc, Maria Rosaria Perrini, MSc, Tom Jefferson, MD, Francesca Gillespie, MD, Marina Cerbo, ScD, Agenzia Nazionale per i Servizi Sanitari Regionali (Age.na.s), Via Puglie 23, Rome, Italy 00187
CONFLICTS OF INTEREST
The authors have no conflicts of interest to declare.
