Introduction
COVID-19 disease, caused by the coronavirus SARS-CoV-2, has gradually spread around the world since the last months of 2019 (Reference Wiersinga, Rhodes, Cheng, Peacock and Prescott1). Because the disease is asymptomatic in most cases, several diagnostic strategies are available to discriminate infected from noninfected people and define which people have developed antibodies to the infection (Reference Gao, Xu, Sun, Wang, Guo and Qiu2). The first step in diagnosing SARS-CoV-2 infection is RT-PCR, which detects viral nucleic acids present in nasopharyngeal fluids, allowing viral detection in asymptomatic infected individuals (Reference Loeffelholz and Tang3). In vitro diagnostic tests, also known as serological tests, detect the presence of antibodies produced in response to SARS-CoV-2 infection and reveal past infections by identifying antibodies. The synthesis of antibodies is a primary immune response to infections (Reference Pan, Li, Yang, Fan, Tang and Zhao4): immunoglobulin A (IgA) responses appear between 4 and 10 days after infection; immunoglobulin M (IgM) levels rise during the first week after SARS-CoV-2 infection, peak after 2 weeks, and then return to near-background levels in most individuals; immunoglobulin G (IgG) is detectable after 1 week and is kept at a high level for an extended period.
Serological tests can help assess protection against subsequent viral exposure and/or for contact tracing purposes; therefore, these tests are essential for epidemiological assessments and forecasting estimates of global therapeutic needs (Reference Lin, Liu, Zhang, Hu, Yang and Guo5;Reference Lipsitch, Swerdlow and Finelli6).
In vitro diagnostic serological tests have rapidly spread in traditional, automated, and point-of-care forms; however, mostly single-center evaluations of these tests have been performed, without any systematic validation of the tests or health technology assessment (HTA) approach (Reference Vashist7–Reference Udugama, Kadhiresan, Kozlowski, Malekjahani, Osborne and Li9).
In the last months of 2020, Covidiagnostix, a multicenter national project funded by the Italian Ministry of Health, was launched with the aim of evaluating and comparing some of the serological tests available on the market for SARS-CoV-2 through the HTA approach.
In this work, by retracing the methodological (related to the stepwise method used in HTA) and analytical (related to the systematic revision of the literature) phases, we report the rapid HTA related to the serological tests of SARS-CoV-2 evaluated and validated in the Covidiagnostix project.
Materials and Methods
Five Italian Scientific Institute for Research, Hospitalization and Healthcare (IRCCS) and one Cooperative were involved in the study: IRCCS Ospedale San Raffaele Hospital (OSR) and IRCCS Orthopedic Institute Galeazzi (IOG) (Milano, Italy); IRCCS Casa Sollievo della Sofferenza (CSS) (San Giovanni Rotondo, FG, Italy); Fondazione IRCCS Policlinico San Matteo (Pavia, Italy); IRCCS Ospedale Pediatrico Bambino Gesù; and OSA Cooperative a r.l. (Roma, Italy). The participants are involved with the COVID-19 epidemic, in terms of both assistance and scientific research, providing a network of skills ranging from analytical evaluation to the management of processes related to laboratory methods.
To yield a timely and appropriate evaluation, we focused our study on safety, clinical efficacy, and organizational and economic impacts. The report was developed using the EUnetHTA (European Network for Health Technology Assessment) Core Model® version 3.0 (https://eunethta.eu/hta-core-model/) as the reference method.
From the entire list of assessment elements, only those relating to the diagnostic test were selected. Each selected assessment element was then evaluated considering evidence gathered through literature search.
The tests subject to the assessment and their features are:
• ADVIA Centaur SARS-CoV-2 Total (COV2 T) (Siemens Healthcare Diagnostics), based on high-throughput chemiluminescent microparticle immunoassay (CMIA) and targeted on spike protein;
• Alinity i SARS-CoV-2 IgG (Abbott), based on high-throughput CMIA and targeted on nucleocapsid protein;
• LIAISON SARS-CoV-2 S1/S2 IgG (DiaSorin), based on chemiluminescence (CLIA) and targeted on spike protein;
• Elecsys Anti-SARS-CoV-2 (Roche), based on high-throughput electro-chemiluminescence (ECLIA) and targeted on nucleocapsid protein; and
• SARS-COV-2 ELISA (IgA) and SARS-COV-2 ELISA (IgG) (EUROIMMUN) based on enzyme-linked immunosorbent assay (ELISA) and targeted on spike protein.
The research and collection of the information were carried out relating to each aspect of assessment. In particular, for the description of the technologies and technical characteristics related to each test, both the information contained in the EUA Authorized Serology Test Performance issued by the Food and Drug Administration (FDA) and those provided by each manufacturer in the technical data sheets (method manuals, protocol, illustrative sheet) were used. The technical data sheets also allowed us to make conclusions regarding safety and organizational aspects.
Information on clinical effectiveness was obtained through a systematic review of the scientific literature by consulting the international database MedLine, through PubMed, and using the following keywords: antibody tests for SARS-CoV-2, serological tests for SARS-CoV-2, and commercial tests for SARS-CoV-2. The systematic review included all types of methodological studies (systematic reviews, overview, case studies, qualitative studies, methodological documents), published before 31 December 2020, which met the following criteria: methodological documents strictly related to antibody tests to SARS-CoV-2; studies in which a description or a comparison of the methods used for the antibody determination for SARS-CoV-2 and their features were provided. Studies in which antibody tests for SARS-CoV-2 were only mentioned without a description and those written in a language other than English were not considered. The studies identified by the research strategies were managed and shared using the Zotero program (version 5.0.95); each bibliographic reference was made identifiable by the surname of the first author and the year of publication.
The assessment of the economic aspect was conducted through direct evidence research based on the analysis/estimate of the costs: each participant routinely feeds an internal database with data relating to testing cost, batch size, cost of personnel involved, consumables, and waste disposal. Such data were shared with the study group to derive an estimate of the full cost for a single test.
Finally, based on all the information collected, the final report relating to each test was drawn up.
Results
The domain “Description and characteristics of the test” (Figure 1) reported the test characteristics that impact the purposes of use, level of health care, and administration methods. In particular, none of the six analyzed tests or the competitors can be administered by the patients themselves or their caregivers. Still, at all stages (from the blood sample to the report), the intervention of health personnel is required. The patient can decide for him/herself to undergo the serological test; however, it is important to know that when used, these tests in an individual are more sensitive as time passes from the time of exposure to infection or the appearance of symptoms.
Description of the topics relating to the “test description and characteristics” domain.
Serological tests aiming to detect the immune response to SARS-CoV-2 rather than detecting the virus itself should not be used as the sole basis for diagnosing COVID-19; they could play a role in the fight against COVID-19 by helping healthcare professionals identify people who may have developed an immune response.
Finally, just tests that target the spike (ADVIA Centaur SARS-CoV-2 Total, Siemens Healthcare Diagnostics; LIAISON SARS-CoV-2 S1/S2 IgG, DiaSorin; SARS-COV-2 ELISA IgA and SARS-COV-2 ELISA IgG, EUROIMMUN) could play a role in evaluating the vaccine response.
The information in the “Test safety” domain reports any unwanted or harmful effects caused by the use of the test; issues of safety for patients (Figure 2, panel A), healthcare professionals, and the environment (Figure 2, panel B) were also addressed.
Description of the topics “patient safety” (panel A), “occupational safety” and “environmental safety” (panel B) into the “test safety” domain.
In particular, in laboratory tests, indirect damage, that is, false negative or false-positive results, is taken into account (Figure 2, panel A). In the case of false-positive results, the risk is that the test leads to the need to isolate and monitor individual people, his/her family, and the people he/she has been in close contact with, limiting their possibility to work. Risks associated with a false negative result include a lack of monitoring for him/her, his/her family, or people with whom he/she has been in close contact and consequently lead to an increased risk of spreading COVID-19 within the community.
The occurrence of adverse events is not expected for any of the tests analyzed; however, only a low possibility should be considered in the preanalytical phase due to possible inconveniences or complications during sample collection (venous blood collection). Furthermore, the technology is based on a fully automated platform, so there is a low risk of operator-related test failure. Also, the reagents are supplied in closed containers, which must be loaded onto the system, thus minimizing the risk of errors related to reconstitution, dosage, administration, or storage.
All six tests require the handling of human blood, so both human source materials and all consumables contaminated with human source materials should be considered potentially infectious and should be handled following applicable and appropriate biosecurity practices. Accidental contact with the reagents can cause an allergic skin reaction. Incidental contact between reagents and acids could release toxic gases. Finally, all tests produce biohazardous waste that must be disposed of according to local regulations; other reagents must not be released into the environment and/or disposed of following local regulations.
In the domain “Clinical effectiveness of the test” (Figure 3, panel A), the evaluation of health benefits through clinically significant end points such as mortality, morbidity, and quality of life was evaluated. An analysis of the literature revealed no differences between the various tests evaluated. As for their impact on the test-treatment chain, serological tests do not affect the treatment of the disease due to their low sensitivity during the first 14 days after the infection. Serological tests could help estimate the mortality rate. Also, all serological tests identify individuals who may have contracted the virus and developed an immune response to SARS-CoV-2 (Figure 3, panel B). Finally, this domain contains the parameters that allow a comparison of the test accuracy of each of the tests (Figure 3, panel C), estimated from the scientific literature, as previously described. In particular, a systematic review of the scientific literature identified 117 scientific papers published in international journals with peer review and an average 3.99 impact factor (IF). In 54 percent of the cases, the selected paper analyzed and described the antibody tests for SARS-CoV-2; the other article elaborated comparison models between tests.
Description of the topics “mortality” and “morbidity (panel A), “function” and “change-in management” (panel B), and “test accuracy” (panel C) into the “clinical effectiveness” domain. NPV and PPV stand for negative predictive value and positive predictive value, calculated in the figure under the hypothesis of 1 and 5 percent prevalence.
The “Organizational aspects for the execution of the test” domain (Figure 4), which concerns the ways in which different types of resources must be mobilized and organized when implementing a technology, and the consequences they may additionally produce in the organization and the healthcare system as a whole, underlines when and if the presence of a trained operator is necessary or if the test can be managed independently by the individual. In particular, no preparation is needed (strict fasting, rest) before taking the sample, and no involvement should be mobilized for patients and/or caregivers. For all six tests, regarding the analytical phase, laboratory technicians receive adequate education and training on the automated technological platform used for the analysis and the method is described in the attached sheet supplied with each kit; the quality control and monitoring system of the test is overseen for each new analytical session undertaken through the quality controls provided by the kit.
Description of the topics and issues relating to the “health delivery process” domain.
The evaluation of the economic domain has been first searched in scientific literature. However, as available publications did not report any data on the economic or financial aspect, direct queries on internal databases have been performed by institutions that participated in the project.
Before economic computation, a process analysis was developed to map the overall test delivery and identify the main (differential) cost drivers. Such analysis highlighted two major findings: (i) no difference was found for the COVID test process with respect to other standard serological tests performed in each center and (ii) no major difference was found among different centers involved in the present study.
Concerning infrastructural and technological facilities, the initial cost of the test equipment/point-of-care test equipment was not included in the economic analysis, because the centers involved in the study are all major hospitals, thus already equipped with the required technical and structural means. The same consideration applied to human resources’ endowment. Moreover, as each center was asked to comply with a Turn Around Time of 24 h, no other resources (technical or human) were needed. The only difference seen was relative to the cost of the test, which ranged from 1.4 to 12.5€.
Discussion
In Laboratory Medicine, the evolution of knowledge and the innovation of technologies are the basis of diagnostic progress. To guarantee the general optimization of the process and the organizational requirements of testing, technological progress must be supported by effective tests and by constantly updated operators.
The large number of new technologies developed in a short period entails the risk of overlap between the traditional technology already in use and the innovative one, which, if prolonged outside the validation period, leads to organizational irrationality and an unjustified increase in costs. Evaluation tools, such as the rapid-HTA method, which can adequately manage and tackle uncertainty, while guaranteeing timely and adequate evaluation, are still helpful when it is more critical to provide decision makers with an indication based on incomplete yet timely evidence, rather than waiting for the collection of complete evidence but with such a delay that could make the decision vain (Reference Vanni, Foglia, Pennestrì, Ferrario and Banfi10).
In the specific case of antibody diagnostic tests for SARS-CoV-2, the simultaneous placing on the market of trials with similar characteristics makes their choice particularly difficult. However, having detailed information is essential as serological tests are strategic in detecting the actual prevalence of the disease in the general population, as they also consider asymptomatic or paucisymptomatic individuals. It is known that there is a part of the population infected but not tested, as they did not have any typical symptoms of COVID-19, such as cough, fever, and breathing difficulties.
For the Covidiagnostix project, developed according to the multidisciplinary nature required by the EuNetHTA model, health professionals and researchers with different backgrounds (biomedical, economic, financial, managerial) collaborated with each other (Reference Franchin, Faggiano, Plebani, Muraca, De Vivo and Derrico11).
The results show very few differences among tests from different manufacturers, namely test costs and the possibility of being used in vaccination triage. With vaccine triage, the authors intended the opportunity to optimize (in particular due to the scarcity of vaccine doses and to logistic constraints) the vaccination campaign by defining vaccine inoculation priorities as a function of antibodies titer (i.e., postponing the vaccination of natural seropositive individuals or the second dose of higher-responsive individuals to the first dose of vaccine).
Conversely, a common feature, the very low sensitivity in the first days of infection, makes them quite useless (at least if not adequately supported with other types of tests) to diagnose the disease and control the pandemic evolution. This is the reason for the current indication of molecular testing and self-isolation after contact with a confirmed COVID-19 positive individual.
Finally, the rapid-HTA evaluation of serological tests for SARS-CoV-2 has been summarized in a report that allows its dissemination and communication, thus becoming a tool to support managerial decisions to promote high-value-added tests and reduce the variability of the diagnostic test offered (Reference Hivon, Lehoux, Denis and Tailliez12).
The EuNetHTA model is a flexible and adaptative method to evaluate various aspects of a medical procedure. In the case of serological tests for SARS-CoV-2, we completed the evaluation on the medical literature, which refers to the period when serological tests were used to indirectly identify individuals who came into contact with the virus and who had developed an immune response.
Then, the rapid-HTA report resulting from Covidiagnostix, the first project granted by the Italian Ministry of Health on the HTA approach, could be the basis to choose the tests to be used in the evaluation of the antibody titer developed by each individual against SARS-CoV-2 and of the efficiency of the vaccination campaign.
Conclusion
The HTA is confirmed as a valid tool to support managerial choices to promote high-value-added tests and reduce the variability of the diagnostic offer.
In particular, through Covidiagnostix, the importance of the rapid-HTA method is outlined, which, by adequately supporting the decision-making process, makes it possible to make the allocation of resources more efficient even in the event of an emergency.
Acknowledgments
The authors thank the members of the working group: Sami Al Bitar Nehme, Elena Bassanelli, Di Resta Chiara, Elena Cittera, Ilaria Cristiano, Liliana De Vivo, Francesco Cosimo Faggiano, Davide Ferrari, Francesco Giuffrida, Anna Lo Russo, Selenia Marino, Stella Pastore, Michela Pireddu, Francesco Ricciardi, Eleonora Sabetta, and Sara Zacchetti.
Funding
The study was granted by the COVID-2020-12371619 call funded by the Italian Ministry of Health.
Conflict of Interest
The authors declare no conflict of interest.
Open access
