Introduction
Health technology assessment (HTA) is a continuous process of analysis and production of evidence to support decisions regarding the use of technologies in health systems (1;Reference Tricco, Langlois and Straus2). Policy makers demand reliable evidence to support timely decisions regarding the use of health technologies (Reference Tricco, Langlois and Straus2). Usually, systematic reviews play a main role in healthcare decision making, guiding the production of informed, high-quality research evidence syntheses (Reference Garritty, Stevens, Gartlehner, King and Kamel3). However, the length of time and resource required to undertake systematic reviews, which can typically be 12–24 months (Reference Harker and Kleijnen4), is a significant barrier that can make their use impractical in knowledge synthesis in health policy- and decision making (Reference Silva, Silva and Barreto5).
Therefore, HTA studies focusing on rapid responses have emerged as useful options to provide evidence to support faster decision making than traditional systematic reviews (Reference Tricco, Langlois and Straus2;Reference Silva, Silva and Barreto5). Rapid reviews are generally not as methodologically robust as full systematic reviews, but their speed and lower resource requirement may mean that a greater proportion of healthcare and policy decisions can be informed by evidence than would be the case if full systematic reviews were always used.
It is important to note that a variety of terms have been used to describe HTA studies focusing on rapid responses, including mini-HTAs, rapid reviews, evidence summaries, rapid syntheses, and brief reviews (Reference Gough, Thomas and Oliver6–Reference Wilson, Lavis and Gauvin9). Kelly et al., through a modified Delphi method, identified important aspects that characterize rapid reviews, but they could not produce a single definition statement to distinguish rapid reviews from other forms of evidence synthesis (Reference Kelly, Moher and Clifford10). There are, therefore, no clear definitions of these studies, with a lack of standardization about how they should be performed and how reliable are their results (Reference Garritty, Stevens, Gartlehner, King and Kamel3;Reference Haby, Chapman, Clark, Barreto, Reveiz and Lavis11). Previous studies have demonstrated that the results of rapid reviews may be different from those of standard systematic reviews (Reference Marshall, Marshall, Wallace, Brassey and Thomas12–Reference Glasziou, Shepperd and Brassey15).
Therefore, the Brazilian Ministry of Health developed methodological guidelines to elaborate a type of mini-HTA report, called Parecer Técnico-Científico (PTC, Scientific Technical Report) (1). These mini-HTA reports have been used to support policy makers and health systems in Brazil with scientific evidence synthesis in a more rapid (within 90 days) and accessible way (1). They are the first step in the process of evaluating the coverage of new technologies, as well as to exclude technologies that are already being used in Brazil's national health system, the Sistema Único de Saúde (SUS)—the Unified Healthcare System. Mini-HTA reports in Brazil are performed by different sectors, such as industry, private medical insurance companies, and medical and patient associations—but they are also an important tool used by the Brazilian Ministry of Health to support timely decision making through the national agency—Comissão Nacional de Incorporação de Tecnologias (CONITEC, the National Committee for Health Technology Incorporation).
The lack of consistency in terminology and methodology of HTA studies focusing on rapid responses increases the importance of an adequate reporting. A comprehensively reported mini-HTA allows a decision maker to be clear about the limitations of the process used in assembling the evidence synthesis.
Objective
The objective of our study was to evaluate the quality of reporting of mini-HTA reports published in Brazil.
Methods
Study Design
This is a meta-epidemiologic study.
Search
An electronic search for all mini-HTA reports published in Brazil between 2014 and March 2019 was conducted in the Sistema de Informação da Rede Brasileira de Avaliação de Tecnologia em Saúde (SISREBRATS, the information system of the Brazilian Health Technology Assessment Network) and the Comissão Nacional de Incorporação de Tecnologias (CONITEC, the National Committee for Health Technology Incorporation) databases. These two databases are freely accessible and contain the majority of mini-HTA reports published in Brazil.
Study Selection
Two reviewers screened all mini-HTA reports published in these databases. The full texts of the mini-HTA reports were checked, and any duplicates were removed. There were no restrictions related to the characteristics of mini-HTA reports, such as their objectives (therapeutic, diagnostic, and others) or the type of the technology assessed (drugs, procedures, devices, and others). The mini-HTA reports found in the CONITEC database are part of clinical reports used to evaluate the possible coverage of technologies in SUS, by means of national clinical guidelines. However, in some of these national clinical guidelines, only abstracts of the mini-HTA reports were available. These abstracts of the mini-HTA reports were not included in our study because of the limited information available.
Data Extraction
As there is no standard tool to evaluate the quality of reporting of mini-HTA reports, we developed a specific data extraction form to assess them. Two reviewers independently extracted data from all included mini-HTA reports. Any disagreements were resolved by discussion or arbitration by a third reviewer when consensus could not be reached. The data extraction process was performed using the software Ragic.
We defined the items to be extracted based on the Brazilian methodological guidelines for conducting mini-HTA reports, previous studies that addressed this content, and some of the tools used for systematic reviews (1;Reference Silva, Silva and Barreto5;Reference Yuba, Novaes and de Soarez16–Reference Shea, Reeves, Wells, Thuku, Hamel and Moran18). The following data were extracted from the included mini-HTA reports:
1. Bibliographic data: title; year of publication; corresponding author; location of corresponding author.
2. The structured research question: whether the PICO acronym or other formats such as PIRO, or PECO were used to create the research question.
3. The characteristics of the population: health condition (e.g., neoplasms, diseases of the respiratory system, metabolic disorders); gender, age range (adults, children, or elderly).
4. The characteristics of the technology of interest: technology type (drug, procedure, device, diet, vaccines, equipment, nonmedical intervention, and others); technology life cycle phase (innovation, coverage/initial diffusion, wide diffusion, abandonment, and exclusion); and the purpose of the technology (preventive, diagnostic, therapeutic; rehabilitation).
5. The characteristics of the technology used as the comparator: reporting if the technology is used in SUS; and the nature of the technology (active intervention or not).
6. The characteristics of the outcomes assessed: whether authors provided any explanation of why they included the specific outcomes listed; discrimination between primary and secondary outcomes; the number of total outcomes; the number of beneficial and harmful outcomes.
7. Eligibility criteria: inclusion and exclusion criteria related to the study design; the reporting of the types of study design eligible (systematic review with and without meta-analysis, randomized controlled trials, prospective and retrospective cohorts, clinical guidelines, and others); and any restrictions related to the sample size of the included studies.
8. Searches: the reporting of the number and the names of databases searched; and any restriction related to the year and language of publication of the included studies.
9. Study selection: the reporting of the number of researchers that participated in the study selection; the number of publications in the screening of titles and abstracts; the final number of included studies; the number of primary and secondary studies.
10. The characteristics of the included studies in the mini-HTA report: whether the authors provided a rationale for choosing the study design in the mini-HTA report; the types of study design included; the presence of a table containing information on the characteristics of the included studies (such as bibliographic information, sample size or number of studies in case of systematic reviews, and interventions and outcomes from included studies); and the number of included comparisons.
11. Inconsistency between outcomes and comparisons reported in the Methods section and those reported in the Results section.
12. Risk of bias assessment.
13. Quality of evidence assessment: use of the GRADE approach; and reporting of the domains assessed.
14. Synthesis of results: reporting of effect size and precision measures; presence of a table containing the summary of findings (information about the outcome, the number of participants and/or studies involved in the comparison, the effect size and precision measures, and the quality of evidence).
15. Recommendation: direction of recommendation (in favor, against, or unclear); strength of the recommendation (weak, strong, or unclear); and agreement between the recommendation reported in the abstract and the conclusion.
Data Analysis
Descriptive analysis was used to summarize the information retrieved from all included mini-HTA reports. The normality of continuous data was evaluated using curve symmetry analysis. Data with normal distribution were described as mean and standard deviation, whereas median and interquartile ranges were used for non-normally distributed data. Categorical data were expressed in terms of number of studies and percentage. The software STATA 10 was used for all analyses.
Results
The search identified 125 mini-HTA reports published between 2014 and 2019. Of these, 103 were included in our meta-epidemiological study (Figure 1). We excluded seventeen mini-HTA reports because only the abstract was available, as they were part of clinical guidelines published in the CONITEC database, and five duplicate studies were excluded.
Study flow diagram.
Table 1 details the characteristics of the mini-HTA reports, focusing on the research question components. Of the 103 mini-HTA reports included, 99 (96.1 percent) used a structured research question (e.g., PICO or other formats). Most of the mini-HTA reports did not report important characteristics of the population of interest such as the gender (80.6 percent) and age range (70.9 percent). In respect of the technology of interest, the great majority of mini-HTA reports focused on the coverage (93.2 percent) of the technologies in the healthcare system, and more than 60 percent of the mini-HTA reports were about drugs.
Characteristics detailed of the research question component of the 103 mini-HTA reports included
HTA, Health Technology Assessment; IQR, Interquartile Range; PICO, Participants; Intervention; Control; and Outcomes; SUS, Sistema Único de Saúde, Unified Healthcare System from Brazil.
Regarding the outcomes assessed in the 103 mini-HTA reports included, only 5 (4.8 percent) provided any explanation of why they included the specific outcomes listed, and 15 (14.5 percent) discriminated between primary and secondary outcomes. In Table 2, we describe information regarding the eligibility criteria, and the search and selection processes of the mini-HTA reports included. Of the 103 mini-HTA reports, 95 (92.2 percent) reported inclusion criteria related to the study design. All mini-HTAs reported the databases searched, with Medline (99 percent) and Cochrane Library (87.4 percent) being the most accessed databases. Around 60 percent of the mini-HTA reports did not report whether there was any restriction related to the year or language of publication of the included studies. In respect of study selection, eighty-nine mini-HTA reports (86.4 percent) used a flowchart format, and fourteen (13.6 percent) reported the number of researchers involved in this process.
Information about eligibility criteria and search and selection processes of the 103 mini-HTA reports included
HTA, Health Technology Assessment; IQR, Interquartile Range.
Details of the assessment of the information about the results and analyses of the mini-HTA reports are given in Table 3. Of the 103 mini-HTA reports, 28 (28.2 percent) provided a rationale for choosing the study design of the included studies. Systematic review with meta-analysis was the most included type of study design (58.2 percent). In thirty-two mini-HTA reports (31.1 percent), we found inconsistencies between the comparisons reported in the methods and results, whereas for outcomes, we found inconsistencies in forty-three mini-HTA reports (41.7 percent). These inconsistencies included additional comparisons or outcomes not identified in the methods that appeared in the results or that appeared in the methods and were not presented in the results, without any explanation.
Information about the results and analyses of the 103 mini-HTA reports included
HTA, Health Technology Assessment; IQR, Interquartile Range.
Of the 103 mini-HTA reports, 62 (60.2 percent) reported assessing the risk of bias of included studies. Regarding the quality of evidence assessment, fifty-four mini-HTA reports (52.4 percent) reported the use of the GRADE approach. Of those mini-HTA reports that used the GRADE approach, 16 (29.6 percent) did not report the domains that may downgrade or upgrade the quality of evidence. Sixty-five mini-HTA reports (63.1 percent) synthetized their results using a summary of findings table. Most mini-HTA reports (66 percent) recommended the use of the technology of interest, whereas the strength of the recommendation was strong in eight mini-HTA reports (7.8 percent). In ten (9.7%) of the mini-HTA reports evaluated, there was disagreement between the recommendation in the abstract and the conclusion of the full text.
Discussion
The quality of reporting of mini-HTA reports performed in Brazil needs to be improved. Based on our findings, important methodological aspects are not reported in most of the mini-HTA reports evaluated. Specifically, the mini-HTA reports lacked details on the included outcomes, the choice of the study design of the included studies, the risk of bias and the quality of evidence assessment, and the presentation of results.
Some of the flaws that we detected in the reporting of mini-HTA reports assessed can be easily adjusted. We noted that around 60 percent of the mini-HTA reports did not report whether there was any restriction related to the year of publication of the included studies. In general, limiting the searches to the last 5 or 10 years may lead to a reduction in the number of studies found (Reference Marshall, Marshall, Wallace, Brassey and Thomas12). However, it is important to note that limiting the search to include only more recent studies can be justified. For example, when the mini-HTA report is evaluating the coverage of new technologies in the health system, the available literature is likely to have been published more recently. In these cases, it is important that the authors report the decision to limit the search. Another point is that only a few mini-HTA reports (13.6 percent) reported the number of researchers involved in the process of study selection. It is recommended that authors report the number of researchers in this process and, if possible, there should be at least two independent researchers to minimize the possibility of missing potential studies. However, it is important to consider the trade-off between the possibility of missing information and the high number of mini-HTA reports that can be done with one reviewer only. Finally, around 60 percent and 50 percent of the mini-HTA reports reported if risk of bias and quality of evidence assessments were performed, respectively. It is important that the mini-HTA reports are clear whether these assessments have been performed and if not, a justification is necessary. These are crucial aspects in any high-quality research evidence syntheses and there are no reasons for not incorporating them in the mini-HTA reports.
A worrying finding of our study was the high number of mini-HTA reports with inconsistencies reported in the methods and results regarding the comparisons and outcomes. Therefore, it is recommended that only the comparisons and outcomes previously defined in the research question be presented. When information about some comparison or outcome is not available in the literature, this should be reported by the authors in the text. Another kind of inconsistency found was that in around 10 percent of the mini-HTA reports evaluated, there was disagreement between the recommendation in the abstract and the conclusion of the full text. Ideally, mini-HTA reports should be internally consistent; otherwise, the credibility of such reports will be undermined, which may be unhelpful in the decision-making process.
Our findings are consistent with those of the few studies published in the literature that have evaluated the quality of reporting of HTA studies focusing on rapid responses. Kidholm et al. evaluated the mini-HTA reports produced in Denmark and concluded that there was a strong need for improvements in the quality of the reports produced (Reference Kidholm, Ehlers, Korsbek, Kjaerby and Beck19). Kelly et al. evaluated rapid reviews (another kind of HTA study focusing on rapid response) and also found that the quality of the conduct and reporting of the included studies (Reference Kelly, Moher and Clifford20) was not up to the mark.
Our study has some limitations that should be addressed. The first is that, as no specific tool to evaluate the quality of reporting of HTA studies focusing on rapid response has been published yet, we used a specific data extraction form developed by us, based on the recommendation of the Brazilian methodological guidelines for conducting HTA reports and previous studies (1;Reference Silva, Silva and Barreto5;Reference Yuba, Novaes and de Soarez16–Reference Shea, Reeves, Wells, Thuku, Hamel and Moran18). Another limitation is that our results are restricted to the Brazilian mini-HTA reports published in the SISREBRATS and CONITEC databases, which may limit the generalizability of our findings to other countries.
Despite the efforts from health managers, and the development of methodological guidelines in respect of the production of mini-HTA reports, there is still a lack of certainty about the replicability and transparency of these studies performed in Brazil (1). This was the motivation behind this study. There is a contrast between the increasing reliance on HTA studies focusing on rapid responses to define clinical decision making and health policy and the uncertainty shown by health managers about the reliability of their findings and whether they reach the desirable standards (Reference Marshall, Marshall, Wallace, Brassey and Thomas12).
Efforts should be made to develop standardized reporting in HTA studies focusing on rapid responses, with the aim of increasing the transparency, reproducibility, and reliability of their results (Reference Tricco, Zarin, Antony, Hutton, Moher and Sherifali21). Guidelines on mini-HTA studies are important as a reference for those carrying out these studies. In Brazil, for example, based on the results of the present study, we have developed a checklist, through a consensus between researchers, to guide the reporting of mini-HTA studies in partnership with the Brazilian Ministry of Health, which was also submitted to public validation (the general public, health professionals, patient associations, and others) (22). This checklist is currently available (22) and includes all the steps that should be reported in the mini-HTA study; it also emphasizes the importance of reporting steps that have not been conducted.
Conclusion
In conclusion, our findings show that the quality of reporting of the mini-HTA reports performed in Brazil needs to be improved. Most of the mini-HTA reports demonstrated flaws in the reporting of important aspects related to their methods and the presentation of their results.
Funding
This study was supported by the Support Program for the Institutional Development of the Unified Health System (Proadi-SUS) in partnership with the Ministry of Health of Brazil. The funder was not involved in the design, collection, analysis or interpretation of data, or in writing the manuscript. We would like to thank Tiago V. Pereira who provided insight and expertise that greatly assisted the research.
Conflict of Interest
The authors declare that they have no conflict of interest.
