Innovation is important to meet the challenges faced by healthcare systems and in improving quality, productivity and patient outcomes. The crucial element of transferring knowledge from research to clinical practice needs to be recognized and supported by national health systems. However, these health systems need to be able to effectively manage the introduction of emerging health technologies as they may generate new costs and infrastructure requirements so will require support themselves. Time is a crucial factor here. It is important to provide early information to decision makers to allow them to plan the introduction and adoption of innovation.
Early Awareness and Alert (EAA) activities, also known as horizon scanning (HS), is the systematic activity aimed at identifying new and emerging health technologies that may have a relevant impact on a populations health and/or on the health system (Reference Simpson, Hiller and Gutierrez-Ibarluzea13). The main phases of an EAA process are the following (Reference Simpson, Packer and Carlsson15;Reference Wild and Langer18): (i) Identification of new and emerging technologies; (ii) Filtration of the technologies identified; (iii) Prioritization of the technologies to be evaluated; (iv) Assessment of selected technologies; (v) Dissemination of results; (vi) Monitoring of the assessed technologies (7;Reference Simpson, Hiller and Gutierrez-Ibarluzea13).
In Italy, Agenas's COTE Project (Centro Osservazione Tecnologie sanitarie Emergenti – observatory of new and emerging health technologies) is the first attempt to establish an EAA system for new and emerging health technologies (primarily devices) at a national level. The need for a national EAA system is a reflection of the policy framework in which healthcare providers and managers operate in Italy. Within the EU device market access is mainly regulated by the CE mark and Italy itself has no additional hurdles; in addition, no mandatory coverage with evidence generation programs are running at a central level.
There are some initiatives operating at a regional level, for example, in the Emilia-Romagna Region (the ORI, regional observatory for emerging technologies) (2) and Veneto Region (the IHSP, Italian Horizon Scanning Project, limited to pharmaceuticals and aimed at predicting which new drugs are likely to have a significant impact on the national health system) (Reference Joppi, Demattè and Menti9). However, these are local initiatives that have limited impact at a national level and fulfill a different need. The COTE Project was a 12 months project funded by the Ministry of Health (MoH) planned during 2007 and started in 2008. After the project ended, the EAA system became fully operative and is now a permanent organization. We describe here the methods used to establish and implement an EAA system, and the main findings of the COTE Project.
AIMS AND OBJECTIVES
The aim of the Agenas's COTE Project was the development of a national EAA system in Italy to inform health service decision makers on the adoption of new and emerging nonpharmaceutical health technologies; and to inform decisions on undertaking primary as well as secondary research (trials, health technology assessments, systematic reviews) on these technologies. To ensure transparency and long term sustainability, the development of the system focused on the involvement of relevant stakeholders throughout the process.
METHODS
The initial part of the project involved identifying stakeholders that would have an interest in the output of the project and would potentially be involved in the EAA process. Stakeholders were identified in the following institutions: Universities (all the faculties of medicine and all the faculties of engineering and economics within the Italian universities that offer healthcare-related courses, for example, biomedical engineering, health economics), regions and regional healthcare agencies, Medical Associations (the FISM, the Italian federation of medical associations), industry representative associations (ASSOBIOMEDICA, the main industry representative in Italy), Istituto Superiore di Sanità (ISS, Italian National Institute of Health), and the Italian MoH. At the end of 2007, all the potential stakeholders received an invitation to participate in the project. One or more key people could be nominated by each institution and a mailing list was created. These key people were invited to participate in two workshops that were structured around the main phases of the EAA process: Workshop I) Identification of new and emerging technologies, and prioritization of the technologies to be evaluated; Workshop II) Assessment of new and emerging technologies and dissemination of results. All workshops were held in 2008 and 2009, and for each workshop delegates were provided with information to consider in advance.
Thirty days before workshop I, focused on identification, filtration, and prioritization of new and emerging health technologies, all the participating stakeholders were sent a brief document prepared by Agenas describing the different EAA systems worldwide, and in particular, the different tools used to identify and submit topics. The stakeholders were invited to express their preference for the different identification methods classified in advance of the workshop. Feedback from each key-person was collected by e-mail (for those that did not send the feedback, their preference was registered by real-time voting during the workshop). In addition, a study by Noorani et al. (Reference Noorani, Husereau, Boudreau and Skidmore12) was attached to help the stakeholders familiarize themselves with the concepts of prioritization and with the criteria used in the selection of the technologies to be assessed. This study was selected because it sought to identify and compare the various practical and current approaches for priority setting in health technology assessment.
The first COTE workshop (September, 2008) presented the project to stakeholders, introduced relevant terminology, presented the results of the voting for the identification methods and tools to be used in the proposed EAA system, proposed the filtration criteria, and allowed delegates to define the prioritization method by which technologies would be selected for assessment. This latter part of the workshop was designed as a dynamic ideas-exchange activity in which the participants were randomly split into groups and guided through a simulated prioritization process involving three technologies. Each group had to express a final statement about its choice (i.e., which technology had been selected as a priority and why).
Thirty days before workshop II, a technology notification form was sent to all delegates. Participants were asked to suggest new and emerging technologies that they were aware of. The aim of this was to pilot the form, gather feedback, and also identify technologies that could be prioritized using the method and criteria identified during workshop I. The aims of the second COTE workshop (January, 2009) were to present the results of the pilot on identification methods, to propose a report structure (the key information that would be presented in the final output of the EAA system), and to investigate the role of the stakeholders in the dissemination of results.
The Project's results were presented at a final workshop: “Final considerations on the future of COTE” held in Rome in November, 2009. In this workshop, the first three EAA reports prepared by Agenas in 2009 were also presented and the main milestones that were agreed by the COTE Project were highlighted.
RESULTS
Results of the COTE Workshop I – “To Identify and Prioritize”
Fifty-two key-persons attended workshop I (Table 2).
Sharing Terminology
The use of the term “health technologies” was restricted to the research fields in which Agenas operates: “devices, procedures, programs, settings, and public health activities but not pharmaceuticals or vaccines.” It was agreed that the EuroScan glossary of terms (http://euroscan.org.uk/outputs/terminology-and-understanding-of-the-activity/) would be adopted with specific variations on the definitions of new and emerging technologies as follows (Reference Douw and Vondeling6): “technologies that are new (e.g., in the phase of adoption that has only been available for clinical use for a short time and is generally in the launch or early postmarketing stages), or are emerging (e.g., not yet adopted by the healthcare system because in premarketing stages, or marketed but not diffused or localized to a few centers), or represent a change in indication or use of an existing technology, or are part of a group of developing technologies.”
Defining Methods and Tools for the Identification and Filtration of Technologies
Stakeholders veered toward an integrated reactive method for the identification of new and emerging technologies, based on a combination of open proposals, results of literature searches, technologies identified by other international EAA agencies, and suggestions from an expert network across the whole territory. A structured notification form was designed to allow interested parties to propose technologies for filtration (Table 1). The structured form was chosen as the only means of technology notification. Filtration was based on the type of technology (i.e., it had to be a nonpharmaceutical), the completeness of the information provided (i.e., information reported in the form had to allow the unambiguous recognition of the technology and its use), year of CE marking, available evidence from the EuroScan database, and diffusion in Italy.
Notification Form Developed to Allow People to Propose New and Emerging Health Technologies to Agenas for Evaluation
Prioritizing the Evaluations
Key-persons from each stakeholder group were invited to discuss the prioritization phase. The stakeholders recognized the difficulty of choosing a limited number of technologies to be assessed from a long list. The EAA systems from other countries show that one “perfect way” to prioritize cannot be realized as this phase strongly depends on the setting in which the EAA system operates and on the issues that the decision makers perceive as priorities. On the basis of the findings of Noorani et al. (Reference Noorani, Husereau, Boudreau and Skidmore12) the stakeholders at the workshop selected five prioritization criteria: clinical-epidemiological, economical-organizational, social-ethical, level of evidence, and risk of inappropriate use. It was decided that, in this experimental stage, no weights would be assigned to the different criteria. Stakeholders agreed the national committee for medical devices (CUD), a multidisciplinary body composed of seventeen members (Ministry of Health, eight members; Ministry of Economy, one member; Regional representatives, seven members; ISS, one member) would be the most appropriate group to prioritize technologies.
Results of the COTE Workshop II: “To Assess and Disseminate”
Forty-seven key-persons attended workshop II (Table 2).
Institutions Participating in the Workshops as Stakeholders
Results of the Pilot Stage on the Identification and on Prioritization
The notification form was sent to seventy-seven key-persons from all stakeholder groups. Forty-three responses were received within the allocated 1-month period. The reasons for the low response rate emerged during subsequent discussions and were mainly related to the short time given for the response and the delayed communication with the medical associations and the industry representatives. The forty-three proposals collected through the pilot were filtered. This left thirty-five technologies that were summarized and proposed for prioritization. As agreed, prioritization was done by consensus by CUD. On the basis of the five criteria CUD members selected three technologies to be assessed in 2009 (Figure 1).
Flow-chart of the filtration process.
The HS Report's Structure and Content
Agenas proposed that COTE EAA reports would be as follows: (i) brief (around 7 pages); (ii) appropriate for a lay audience; (iii) written in Italian and in English (to facilitate the integration in the EuroScan network); (iv) openly downloadable (as Agenas is a public body and works with public funds); (v) transparent, describing the methods for evidence searches and disclosing authors’ conflict of interests.
Moreover Agenas proposed a possible structure; the COTE reports would contain the following information: target population; description of the procedure and technology; clinical importance and burden of disease; products, manufacturers, and approval; setting; roll out in Italy, comparators, effectiveness and safety; potential benefits to patients; cost of the technology; potential impact (both structural and organizational); conclusions; future prospects. All stakeholders agreed with the proposals for the HS report structure and content.
Dissemination of Results
As each stakeholder has a clearly defined target audience, it was proposed that the stakeholders use their own channels to disseminate the HS reports, while direct dissemination would be carried out by the Agenas and MoH Web portals. All stakeholders were invited to post the reports on their Web portal. Dissemination toward international audiences would be carried out exclusively by Agenas as a EuroScan member, and entered in the EuroScan database as “freely downloadable to all” (without user restrictions). Again all stakeholders agreed with these proposals.
Results of the COTE Workshop III: “Final Considerations and Future of the COTE Project”
Fifty-one key-persons attended workshop III (Table 2).
The First HS Reports
The first three HS reports (“Transapical transcatheter aortic valve implantation”, “Tele-home care system based on digital video broadcasting terrestrial technology”, and “Minimally invasive treatment for atrial fibrillation by high intensity ultrasound ablation”) were produced by Agenas after consultation with external experts (both physicians and manufacturers). The reports were also reviewed by external experts and comments were analyzed, discussed, and integrated if judged appropriate. The final documents were made available through the resources agreed in Workshop II (3–5).
Final Consideration of the COTE Project
The COTE Project set the basis for a permanent national EAA system in Italy. Involving relevant stakeholders in the development of the system from the start enabled the methods for submitting new and emerging health technologies for evaluation to be established successfully and has resulted in a strong network of key people. An open call for notification of new and emerging technologies will be launched to all the stakeholders on a 6-month basis. Transparent methods are in place for the assessment of prioritized technologies and the involvement of the various stakeholders in identification, assessment and dissemination has been defined. During the project positive feedback has been received from stakeholders. In particular, the involvement of Regions and Regional Agencies as well as Industry (manufacturers and their representatives) increased from the first to the third workshop (Table 2) and the number of representatives continues to grow as other manufacturers have requested enrolment on the mailing-list following the final workshop. Conversely, a decrease in the involvement of universities and medical associations was observed.
Collaboration from manufacturers was satisfactory. All the manufacturers involved were open to the exchange of ideas and point of views, providing also some data from “gray literature” (e.g., registers, conference abstracts). Most importantly, the three resulting report's “conclusions” were written without any pressure from these stakeholders.
DISCUSSION
The concept of “new means better” should not be assumed in today's clinical practice. In Europe the access to market is regulated by CE marking. In general, medical devices are progressively regulated taking into account their complexity and associated risk factors of invasiveness, time of contact, body districts, or local versus systemic effects (Reference Schuh16). However, this regulatory environment is strictly focused on safety and efficacy and, in most cases effectiveness does not need to be proven before commercialization (Reference Migliore, Ratti, Cerbo and Jefferson10). New medical devices with an uncertain profile of clinical effectiveness may be introduced and widely marketed in uncontrolled ways (Reference Spencer, Nicklin and Wickramasinghe17). This will have an impact on patients (e.g., in terms of health benefits) and on health systems (e.g., in terms of expenditures).
The COTE Project has now ended and a national EAA system has been implemented. To gain recognition of the system and related activities, stakeholders were kept informed and involved from the early planning phases and all aspects of future developments were discussed. In particular, the regional stakeholders showed increasing interest and they are currently involved in other projects and activities managed by Agenas. Stakeholder involvement has been a key issue for the projects development and sustainability. The Italian context is highly regionalized, with a low level of interference from central government, so a decision to exclude stakeholders could have resulted in a detrimental outcome. EAA is a new concept in Italy so Agenas's major effort has been targeted toward the “education” of stakeholders and ideas-sharing. Agenas is carrying out EAA activities as a nonprofit organization, publicly funded, with an ongoing, officially recognized technical role in supporting Regional and National governments.
It is well known that the instructions to build the “perfect” EAA system do not exist. Every EAA system is part of a health system and works in symbiosis with it and the wide variability showed in the recent comparative analysis by EuroScan are consistent with this concept (Reference Simpson and Packer14). An EAA system suitable for Italy was therefore developed by taking the best from other systems and adapting particular elements to the Italian national health system. While further studies are ongoing to assess the impact of the assessments produced, COTE is now working and will work in the future using the same general framework. Table 3 summarizes some characteristics of the Agenas-COTE EAA system (see Supplementary Table 1, which can be viewed online at www.journals.cambridge.org/thc2012036). According to the Agenas's inspiring principles, the COTE's outputs have to be used as an information tool and in particular, to support decision makers (at all the institutional levels); to support clinical practice (for health professionals); and to guide health research (for the HTA Agencies, and research centers).
Brief Summary of the Agenas-COTE System Characteristics and Approach (Full Table is Available as a Supplementary Table 1)
As ideas are constantly exchanged between Agenas and stakeholders, the network will evolve in the future and new roles will be defined for the Regional authorities.
The initial low participation rate registered among some stakeholders was considered to be a possible lack of interest toward the project. For example a decrease in the involvement of universities and medical associations was observed. A need to investigate this was recognized, however, when the pilot project ended and the first outputs were made available, there was an increased interest from stakeholders that were initially reluctant to participate. Continuing to promote EAA activities and gather support will contribute to the future success of the system. The COTE's outputs (i.e., EAA reports) can be downloaded from the MoH Web portal (11), the EuroScan Web site (in English) (8) and from the Agenas Web portal (in Italian) (1).
CONTACT INFORMATION
Maria Rosaria Perrini, M.Sc., Tom Jefferson, MD, FFPHM, Marina Cerbo, M.Sc., Innovation and Development Director. All the authors are from Agenas, Agenzia nazionale per i servizi sanitari regionali, via Puglie 23, 00187 Rome, Italy.
CONFLICTS OF INTEREST
All authors report they have no potential conflicts of interest. The authors declare that they will not receive benefits from the publication of this study. The COTE Project was funded by the Italian Ministry of Health.
