In spring 2020, during the first wave of the outbreak of the COVID-19 pandemic, all countries around the world struggled to produce and to use best evidence to inform health policy (Reference O’Rourke, Orsini and Guerino1). Within a short time, clinical trials testing single agents or tests, large international platform trials testing multiple medicines (i.e., WHO Solidarity and UK Recovery), were started in the shortest time ever seen in the history of clinical trials (Reference Pearson2). As of July 2021—18 months since the pandemic’s onset—3,208 Randomized Controlled Trials (RCTs) on therapeutics, preventive measures or vaccines were running and 330 evidence syntheses had been published (https://covid-nma.com/). A new format was arisen, the “living reviews,” to digest the multiple sources of evidence and to permanently update the reviews with incoming RCTs (Reference Pearson2;Reference Boutron, Chaimani and Meerpohl3). Alongside the increase of primary and secondary data, coordinated initiatives started to avoid duplication of efforts and to make tailored evidence available to health policy in an efficient manner. Among others, the European network for Health Technology Assessment (EUnetHTA) decided to join forces for collaborative activities aimed at providing scientific data and information to support decision makers in counteracting the COVID-19 pandemic, through the development and publication of rolling collaborative review (rolling CR) and rapid collaborative review (rapid CR) formats (https://www.eunethta.eu/services/covid-19/).
Furthermore, the EUnetHTA COVID-19-response included the setting up of a publicly accessible repository of the publications and work of national HTA agencies on COVID-19. To maintain this database, the Web sites of all European HTA agencies were screened on a bi-weekly basis.
The aim of this paper is to describe the process and output of the coordinated and collaborative activities of EUnetHTA in response to the COVID-19 pandemic, to discuss both HTA capacity to meet national as well as European decision makers’ needs and to reflect on missed opportunities and suggestions for the near future.
Methods
We used a descriptive analysis of key documents related to the coordinated and collaborative EUnetHTA process, methodology and reports related to the response to the COVID-19 pandemic on diagnostics and therapeutics from April 2020 to May 2021. The main aims and the methodology applied to rapid CR and rolling CR are described in detail. Relevant documents were identified by searching the Web sites of the EUnetHTA and through the analysis of the EUnetHTA’s internal documents.
Results
Process: Coordinated and Collaborative Activities on Diagnostics and Therapeutics
In April 2020, with most EU countries in lockdown, a COVID-19 task force was set up by EUnetHTA partners and a survey was sent to all partners to collect pressing health policy questions across Europe. Two coordinating agencies for diagnostic tests (RER/IT) and therapeutics (AIHTA/AT) were assigned. Soon after, the team on COVID-19 diagnostic presented a work plan to address some of the questions resulted from the survey, namely the screening of asymptomatic persons, the diagnosis in people with symptoms and how to best monitor the course of the disease to inform decisions on management, at the individual and public health level. The call for collaboration resulted in a joint effort for the production of two joint rapid CRs (Reference O’Rourke, Orsini and Guerino1;Reference Pearson2) on diagnostic tests with the involvement of five HTA agencies from four European countries (Table 1).
Rapid Collaborative Reviews: Diagnostics and Therapeutics
AIHTA, Austrian Institute for Health Technology Assessment/AT; DVSV, Dachverband der Sozialversicherungsträger (formerly HVB)/AT; GÖG, Gesundheit Österreich/AT; HIS, Health Improvement Scotland/UK; HIQA, Health Information and Quality Authority/IR; HTW, Health Technology Wales/UK; INFARMED, National Authority of Medicines and Health Products/PT; IQWiG, Institute for Quality and Efficiency in Health Care/GE; MoH Ukraine, Ministry of Health of Ukraine/UA; NA, not applicable; NCPE, National Centre for Pharmacoeconomics/IR; RER, Reg Emilia-Romagna/IT; SESCS, Servicio de Evaluación del Servicio Canario de la Salud/ES; SNHTA, Swiss Network for HTA/CH; SUKL, State Institute for Drug Control/CZ; UCSC Gemelli, Università Cattolica del Sacro Cuore/IT.
The main objectives of the EUnetHTA Task Force on SARS-CoV-2 diagnostics were to answer the following health policy questions:
-
1. How to best screen asymptomatic subjects and monitor close contacts in order to promptly detect infections among the general population and healthcare workers.
-
2. How to best test patients with clinical manifestations of SARS-CoV-2 in order to confirm the diagnosis of COVID-19.
-
3. Which tests should be used to monitor the course of disease and inform decisions on treatment, hospitalization, and to determine viral clearance of recovered patients in order to allow re-entry into the community.
Besides the evidence-based support on testing policies, EUnetHTA decided to start with a series of so-called “rolling” CRs, as well as rapid CRs (Table 1) related to treatment/pharmaceuticals. The expression “rolling” was borrowed from the European Medicines Agency (EMA) (4), conveying that all incoming clinical trial data would be assessed at the time of their publication aiming at providing swift policy support for eventual promising therapeutics or—equally—on therapies failing in treating COVID-19 patients. For this new format of rolling reviews, a group of eleven EUnetHTA partners from eight European countries (Table 2) was formed to monitor candidate therapies and to collect trial data according to a predefined methodology. The endeavor was supported by two partnering institutions, the Department of Epidemiology Lazio Regional Health Services—DEPLazio/IT (https://www.deplazio.net/farmacicovid/index.html) and by the Living Map initiative, a collaboration of NIPH/ N and EPPI-centre/ UK (https://www.fhi.no/en/qk/systematic-reviews-hta/map/).
EUnetHTA Rolling Collaborative Reviews (May 2021, End of EUnetHTA JA3)
AEMPS, Agencia Española De Medicamentos Y Productos Sanitarios; AETSA, Andalusian HTA Agency, Ministry of Health/SP; AIHTA, Austrian Institute for Health Technology Assessment/AT; GÖG, Public Health Institute/AT; HTW, Health Technology Wales/UK; KCE, Health Care Knowledge Centre/BE; NIPH, Norwegian Institute of Public Health/N; NIPN, National Institute of Pharmacy and Nutrition/HU; SESCS, Evaluation and Planning Unit of the Canary Islands/SP; SMCA, State Medicines Control Agency/LT; SNHTA, Swiss Network for HTA/CH.
Comparison of EUnetHTA Products
The objectives of the EUnetHTA activities on therapeutics were
-
1. To inform health policy at the national, regional, and European levels at an early stage in the lifecycle of therapeutics, on which interventions are currently undergoing clinical trials,
-
2. To monitor permanently—in the format of a living document—potential therapies against COVID-19,
-
3. To support preparations for an evidence-based purchasing of regional/national health policy makers, if necessary.
Formats and Methodology: Rapid CRs (Diagnostics and Therapeutics) and Rolling CR (Therapeutics)
In order to increase a timely response to the public health emergency, it was agreed that rapid CR would differ from Joint Relative Effectiveness Assessment in so far as the consultation of manufacturers and the involvement of external experts (clinicians and patients) would not be mandatory. They would not differ, however, in the methods and conduct of the systematic search, review, and synthesis of all available evidence, nor in the requirement for dedicated reviewing. In the rapid CRs, an assessment team (author, coauthor, and dedicated reviewers) was convened to conduct a systematic literature review and a synthesis of the evidence, following a reporting template similar to that of the Relative Effectiveness Assessments. Moreover, for a rapid CR a Population, Intervention, Comparator, and Outcomes (PICO) question was established.
During the first pandemic outbreak most of the scientific literature was made available in great haste and without being peer-reviewed and less appropriate study designs have been included in order not to lose any potentially useful data (e.g., case control and retrospective cohort studies were included in the diagnostics reviews). However, quality assessment and risk of bias assessment were performed rigorously and appropriate comments on quality and uncertainty of the results were made for each research question.
Rapid CRs are not living documents as in the case of rolling CRs, and thus subject to update on a case-by-case basis. For COVID-19 treatments, the rapid CRs can be a follow-up from the rolling CRs using the information from that report as a starting point. Furthermore, in the pharmaceutical rapid CRs, patient input was also sought and evidence gaps tables were populated. One major difference is that although, for the pharmaceutical rapid CRs, the manufacturers were always requested to participate by submitting a dossier or relevant information, the rapid CRs—in contrast to joint assessments—could continue without manufacturer involvement. Details on rapid CR and rolling CR formats can be found in Table 3.
The coordinator (project manager) in close collaboration with the EUnetHTA partner engaging in this activity developed a process and method to prioritize and select therapeutics for rolling CR, to decide when to update them as well as when to stop them. Figure 1 explains the steps in the process in detail.
Continuous process of rolling collaborative review production. CR, collaborative review; EMA, European Medicines Agency.
Prioritization and selection was based on the Horizon Scanning System of the AIHTA established in April 2020, scanning different sources such as clinical trial registries for ongoing studies on repurposing of existing compounds as well as on the development of new medicine for COVID-19 (a detailed description is provided elsewhere (5)). From the long-list of compounds in Phase 2 or 3 a short-list for those medicines to be uptaken as EUnetHTA rolling CR was generated based on detailed starting rules and discussions in the EUnetHTA COVID-19 task force (see Figure 1):
-
• Starting rule 1: Published results from ≥1 phase III RCT with positive efficacy and safety results in the indication and population under review (high or moderate quality, nonpeer-reviewed, or peer-reviewed article). Confirmed by RCTs found on covid-nma.com and ClinicalTrials.gov.
-
• Starting rule 2: Upcoming (promising) evidence of ≥2 phase III trials.
-
• Starting rule 3: Compound included as trial drug in a platform trial on COVID-19 treatments.
-
• Starting rule 4: Combination therapy of ≥2 promising pharmaceuticals—the combination is reviewed as a separate intervention.
-
• Starting rule 5: Interest from ≥2 EUnetHTA Partners (e.g., compassionate use in a country) OR Interest from MAH to seek marketing approval OR Interest from EC, HTA Network or EUnetHTA Stakeholder Groups.
Conduct of rolling CRs: The rolling CRs are living documents descriptive in nature, updated on a (bi)monthly basis and centrally coordinated (6). They are based on three sources of information:
-
1. Published RCTs presented in a summary of findings of efficacy and safety data (synthesized for a network meta-analysis conducted by DEPLazio, Italy). Two reviewers from DEPLazio screen search results, assess full texts of studies and extract study characteristics and outcome data according to predefined criteria. Risk of bias is assessed using the criteria outlined in the Cochrane Handbook for Systematic Reviews of Interventions, and the GRADE approach is used for rating the certainty of the evidence.
-
2. Published prospective observational studies for safety results, provided by the Living Map of COVID-19 evidence conducted at NIPH, Norway. Two researchers from NIPH carry out title and abstract screening and assess the full texts of all potentially eligible studies. The individual author extracts the data and assesses the risk of bias using the Robins-I tool.
-
3. Ongoing RCTs registered in clinical trial registries (ClinicalTrials.gov; EudraCT Register; ISRCTN). The individual author searches and extracts the data from the eligible studies. Detailed stopping and starting rules for rolling CRs were defined to guarantee a transparent selection of reviewed compounds.
Stopping a rolling CR (and eventually starting an early assessment): The decision to stop a rolling CR is taken collectively in the EUnetHTA COVID-19 task force based on either stopping rule 1 OR stopping rule 2:
-
• Stopping rule 1: The compound is in EMA’s marketing authorization process or has a positive marketing authorization decision.
-
• Stopping rule 2: No clinical benefit: ≥2 RCTs OR treatment arms in platform trials (i.e., Recovery) with negative efficacy and/or safety results in the indication and population under review (phase III, of high or moderate quality/high or moderate certainty of evidence, well powered) OR ≥ 1 RCT with negative efficacy and/or safety results in the indication and population under review (phase III, of high or moderate quality/high or moderate certainty of evidence, well powered) AND stopped enrollment of participants to the treatment arm of interest in a platform trial, because no evidence of beneficial effects.
-
• Additional to be considered for the discussion: Amount and number of included patients of ongoing RCTs and upcoming RCTs.
Final Reports: Diagnostics and Therapeutics
Diagnostics
Two rapid CRs were conducted in order to provide a synthesis of available evidence on clinical benefit and safety of two main classes of tests commonly used in management of the pandemic, antibody tests and molecular tests (Table 1). The first rapid CR was carried out between May and June 2020, with aim to answer whether antibody test can be a reliable tool in following clinical uses (7): (i) early detection of new asymptomatic cases of SARS-CoV-2 acute infection in the general population and/or specific subpopulations; (ii) diagnosis of SARS-CoV-2 acute infection in patients presenting symptoms suggestive of SARS-CoV-2 infection; (iii) measuring seroprevalence in the communities; (iv) ruling out risk of transmission in patients who had recovered from acute infection, and (v) assessing protective immunity in subjects with past infection. At the time of publication, only data for the first two diagnostic accuracy questions were available and retrieved.
The second rapid CR on the current reference standard molecular tests, published in December 2020 (8), reviewed whether alternative molecular tests methods could be used to scale up the current COVID-19 testing protocols, that is, how well they work to diagnose patients that are suspected to have COVID-19 compared to what is currently used to diagnose these patients. Precisely, it addressed two policy priority questions: (i) How to best test patients with clinical manifestations of SARS-CoV-2 in order to confirm a diagnosis of COVID-19 and (ii) How to best screen asymptomatic subjects and monitor close contacts in order to promptly detect infections among the general population.
As data on important patients’ outcomes from comparisons of different testing strategies were not available, a clear description for both risks and benefits were provided (Table 4) and comparative natural frequencies were calculated to inform decision making.
Patient Important Outcomes to Consider When Using Tests to Diagnose SARS-CoV-2 Acute Infection in Patients Presenting Symptoms
Therapeutics
As of May 2021, nineteen rolling CRs were ongoing (6). From the initial list of RCRs, one was suspended due to evidence on lacking effectiveness (Lopinavir + Ritonavir) and three moved on to rapid CRs owing to EMA’s conditional approval or endorsement of use (Remdesivir, Dexamethasone, REGN-COV2, Bamlanivimab monotherapy, and bamlanivimab plus etesevimab combination) (Table 1). Four rolling CRs are updated on a bi-monthly basis due to lack of high-quality evidence.
EUnetHTA Joint Action 3 ended in May 2021. Nevertheless, six partner agencies have decided on a voluntary basis to continue conducting the rolling CRs on a reduced number of compounds (in total nine compounds were continued: camostat, nafamostat, anakinra, APN01, baricitinib, vitamin D, mavrilimumab, ivermectin, and aspirin). A particular focus is repurposed drugs, especially those used in the European platform trial, as well as those antibody drugs selected by the EU Health Union Resilience and Health Emergency Response Authority (HERA) for developing, manufacturing, and procuring at EU level (9).
Discussion
Summary and Reflection on Diagnostics
The first rapid CR on antibody tests was carried out between May and June 2020, when decision makers were searching for testing strategies that could supplement the shortage of molecular tests, and the authors were able to complete the work in just five weeks (7). This review was informed by a quite tight liaison with the other European institutions working on COVID-19 testing. The second rapid CR on the current reference standard molecular tests, published in December 2020, proved particularly valuable as it compared different types of molecular tests and provides the basis for assessing the comparative effectiveness of all upcoming new diagnostic devices (8).
The very fast deployment of antigen rapid tests and consequent bulk procurement by national health systems, while clinical data were still in development, did not allow for timely comparative assessment and no joint collaborative rapid review was carried out on these tests, although planned.
Response to EC-information needs: EUnetHTA was invited at the meetings of the European Commission COVID-19 Testing Working Group, and had an opportunity to liaise with the experts of the Joint Research Centre (JRC), the European Centre for Disease Prevention and Control (EDC), and the World Health Organization (WHO) Collaboration Network. Their input was very valuable in shaping the scope of our review and its research questions, two of them on the diagnostic use of antibody tests (for symptomatic and asymptomatic persons) and three other indications of use (use in seroprevalence studies, in establishing virus clearance and in assessing immunity). Only evidence on the diagnostic use of the tests was available and, in coherence with knowledge on how antibodies develop and become detectable, quantitative results were pooled according to different timings from symptoms’ onset showing how diagnostic accuracy improves as time and number of weeks since symptoms onset increase.
Response to national needs: To aid transferability to decision making, results of the rapid reviews on COVID-19 diagnostic tests have also been presented in terms of risk/benefit trade-offs, highlighting the benefits of having a correct test result against the risk associated with an incorrect test result. Aware of the differing outbreak situations of the countries across Europe, and of the sudden changes that might occur within a country, these calculations were presented for different sizes of target population and different prevalence estimates.
Summary and Reflection on Therapeutics
Since there was such an urgent need to publish accessible information on the COVID-19 therapeutics, EUnetHTA had fairly limited time to develop a process or template. Therefore, this was developed on-the-go (6). Collaboration was sought with an external party (DEPlazio/Italy) for the rolling CR, which was performing a network meta-analysis for COVID-19 studies (Reference De Crescenzo, Amato and Cruciani10). In addition, a collaboration was established with a EUnetHTA partner (NIPH/Norway) who would conduct a systematic literature search for observational studies for the COVID-19 treatments (11). Having these two tasks centralized with one party was very helpful and allowed the authors of the rolling CRs to focus on analyzing the results. To keep the rolling CRs readable for the audience, a version history was included which shows the major changes compared to previous versions. Also, to ensure the rolling CRs were focusing on the most important treatments, starting and stopping rules were developed (6). One crucial stopping rule was when the treatment was about to receive Marketing Authorization (or had already received it) from the EMA. In that event, the treatment was eligible to continue into a rapid CR. Four reports of rapid CRs were published, with one update related to remdesivir as the important new evidence was published (12–16). Three rapid CRs were published before the compounds under review were granted Marketing Authorisation by the EMA (14–16). At time of publication, to the best of our knowledge of EUnetHTA, the EMA rolling reviews were still ongoing on two compounds. Under Article 5 (3)1 the EMA issued an advice on the use of these compounds in European Member States. Therefore, EUnetHTA decided to publish the rapid CRs to support the Member States in potential HTA activities on these compounds. However, once Marketing Authorisation is granted, these rapid CRs need to be read with caution as the indication used in these reports may be different from the indication approved by EMA. The authors of these reports reserve the right to edit the reports at a later point in time if necessary.
To perform rolling CRs on a monthly basis, a flexible but clear process was needed with as little resource requirements as possible. The experience is that it takes a lot of coordinative resources to bring together all partners and to develop a template, process, and methodology that is fit-for-purpose. As one might imagine the coordination not only of eight EUnetHTA partners as authors for monthly or bi-monthly updates of rolling CRs on the monitored compounds, but also of the collection and distribution of information from the external partners is a complex undertaking, but worthwhile undertaking, considering the number of countries in Europe served with this information. Since the start of the rolling CRs in August 2020 until May 2021, up to nine versions of 23 therapies were published, three compounds moved on toward process of conditional market approval and, accordingly, to rapid CRs as succession for policy support, one monitoring of a compound was stopped due to proof of lack of efficacy.
Response to EC-information needs: As a “byproduct” of these rolling CRs EUnetHTA was asked to inform the large adaptive platform trials in Europe on which therapies are promising, which treatments might be considered for inclusion in new study arms and when to expect results from national trials. The European Clinical Research Infrastructure Network (ECRIN) promoted the idea of developing multinational, multiarm “adaptive” platform trials in Europe to rapidly recruit patients and test multiple treatment options simultaneously. The platform studies that have been running since spring 2020 (DisCoVeRy as an extension of Solidarity in Europe, Remap-Covid, Recovery/UK) or started in 2021 (EU-Response) are based on the WHO guidelines on defined patient cohorts and validated core outcome sets (Reference Jin, Pang and Zhang17) and therefore lead to comparable and policy-relevant research results more quickly. Joining forces is an example of ensuring complementarity of all major European COVID-19 adaptive platform studies and increasing their validity. Additionally—in summer 2021—EUnetHTA was asked to support the EC-HERA in its Therapeutic Strategy to develop a clear overview of potential candidates and to advise on the five most promising COVID-19 therapeutics for an eventual EU-wide joint purchase of some of those medicines (9).
Response to national needs: In early 2020 each country started national programs to support national decision making. Part of the national needs was to respond to—often very regional—promising offers of repurposed therapies in national trials or of regional providers. It was part of the EUnetHTA coordinative efforts and adaptive flexibility to serve (contentwise and formatwise) the national needs as much as the EC-information needs. According to the EUnetHTA implementation survey (on average, fourteen agencies responded) the reports on diagnostics were used by twelve to sixteen agencies and the reports on therapeutics by five to nine agencies.
Reflection on Determinants of EUnetHTA Response Matching Needs
During the pandemic decision makers, confronted with the up-surging of affected people’s numbers and the struggling of their health systems, were further hastened and pressurized by a landslide of media’s reported advice, opinions, and declarations seldom supported by biological plausibility and hardly ever based on clinical evidence. In this context, the role of Health Technology Assessment in informing decisions with timely and scientifically robust information was put under great strain.
To meet the need for fast and timely delivery of information, EUnetHTA proved capable of tight cooperation for a coordinated work and a quick response, showing flexibility in restructuring its products without jeopardizing their quality and trustworthiness.
However, while timeliness could be dealt with by partners’ collaboration and the rapid/rolling reviews’ initiative, the lack of robust clinical data compromised the endeavor of providing conclusive scientific evidence. Given the overload of bad quality studies and the lack of coordination within the scientific community, uncertainty could not be avoided. The haste for answers overturned the cautiousness with which health systems usually deal with uncertainty. For example, the limited role of antibody tests in early diagnosis of the illness did not prevent their widespread use in case finding when molecular tests came short, while many patients were administered compassionate use of single or in combination drugs, despite the absence of clinical safety and efficacy data.
Finally, fast track approval pathways for both diagnostics and therapeutics rushed health systems in the run for procurement, allowing little or no time for comparative assessment of risks and benefits.
On the basis of this past experience, we believe that EUnetHTA should strive for (i) the formalization of the process and methods of conducting rapid and rolling CRs, (ii) a clear discipline of the relations between HTA and regulatory bodies during the entire lifecycles of health technologies, and (iii) the consolidation of HTA’s role in Scientific Advice in the design and development of clinical trials. Despite the successful outcomes of the Early Dialogue line of activity of the EUnetHTA Joint Actions (18), during the pandemic it had not been possible for the Network to be involved in any Scientific Advice for trials assessing COVID-19’s health technologies. Such an early involvement in the development of urgently needed technologies could prevent wasting much time in inconclusive research, improve the quality and relevance of the evidence produced, narrow the time lag needed to overcome uncertainty and assure fast access to safe and effective drugs and tests.
Conclusion
Throughout the current COVID-19 pandemic, EUnetHTA partners have been under great pressure to provide timely responses to support their own national decisions makers. In such a turmoil, the partners and the EUnetHTA Network proved capable of prompt collaboration and trust, which allowed speeding up the production and release of high-quality EUnetHTA outputs, while the relationships with the other European institutions facilitated their quick dissemination. The capacity to aid decisions toward the most effective and safe health technologies has been however hindered by the scarcity of good quality scientific evidence. In order to reduce the burden of uncertainty, that the onset of a new disease typically carries, HTA should have a formal role during the different phases of development of health technologies, in order to ensure prompt collection of adequate evidence and fast effective and safe response to the health emergency.
Conflicts of Interest
The authors declare that they have no conflict of interest.
Open access
