The SARS-Cov-2 pandemic has challenged world health care systems by causing a novel COVID-19 disease with no known treatments. In response, investigators around the world launched clinical trials to evaluate the efficacy and safety of various products hypothesized to alleviate the symptoms associated with the disease and reduce the mortality rate.Reference Bauchner and Fontanarosa 1
The ecosystem of biomedical research involves multiple stakeholders with diverse interests involved in the design, conduct, and funding of clinical trials.Reference Macleod, Michie, Roberts, Dirnagl, Chalmers and Ioannidis 2 The characteristics of the trials organized in response to the pandemic strongly influence the nature of the evidence that can guide clinical and regulatory decision-making. The best evidence emerges from clinical trials that adhere to high levels of rigor. For example, blinded and randomized trials are considered the gold standard for trial design because they are most likely to minimize bias.Reference Higgins, Altman, Gøtzsche, Jüni, Moher and Oxman 3 However, other trial characteristics also influence the relevance of the evidence and the trials to support decision-making, such as the primary outcome, comparator (e.g., active vs. placebo), population studied, or source of funding.Reference Naci and Ioannidis 4
Some reports have evaluated the nature of the clinical trials that emerged in the wake of the recognition of the SARS-CoV-2 pandemic. One included 244 studies available worldwide by early April 2020, described interventions (treatment arms and preventive or curative objective), size, randomization and blindness, status of recruitment, and country, with a description of the trials per therapeutic class.Reference Chowdhury, Rathod and Gernsheimer 7 A second by authors affiliated with the European Medicines Agency on the trials registered in the European database of clinical trials,Reference Eichler, Cavaleri, Enzmann, Scotti, Sepodes and Sweeney 6 presenting however only limited details to support the comments of the authors. The third one reviews trials registered to test hydroxychloroquine only.Reference Chowdhury, Rathod and Gernsheimer 7 The fourth one includes 201 trials registered in the WHO’s International Clinical Trials Registry Platform by the end of March 2020.Reference Mehta, Ehrhardt, Moore, Segal and Alexander 8
However, none of these studies evaluates the rigor of the design of the trials being organized. To understand how well the international scientific community employed the principles of trial rigor in its initial response to the SARS-Cov-2 pandemic, we reviewed the characteristics of the clinical trials organized in France and the US. Analyzing the landscape of clinical trials can help judge how the clinical research community responded to the early outbreak and can help policymakers alter the response to the ongoing pandemic.
Methods
Database and Extraction
ClinicalTrials.gov, 9 a US-based clinical trials registry, was searched on May 8, 2020 to identify trials related to COVID-19, using the search tag provided by the registry. We used the registry ClinicalTrials.gov as the data source for several reasons, including the requirement by the FDA to register clinical trials in the database before the enrollment of patients in the US, and the real-time visibility of the registered trials in this primary database. The clinical trials organized in France and the US were selected as these countries were two of the hardest-hit with the virus and the two in the world registering the most clinical trials linked to the management of COVID-19 in ClinicalTrials.gov at the time of the study. We included only trials classified as “interventional (clinical trial)” originating from the US and France using the map display provided by the database. We excluded trials of procedures, blood sample collections, medical devices, diagnosis tests, and behavioral.
From qualifying trials, we extracted the following variables that were available in the database: status, results, interventions, outcome measure, phases, enrollment, funding source, study design, and primary completion date.
Figure 1
Data were extracted from the database, coded, analyzed, and organized using Microsoft Excel (Microsoft, Redmond, WA). The coding rules were discussed and defined by the four authors. A quality check was performed to ensure the correct classification of the observations.
Definition of the Variables
We assigned the variables extracted from the database to the following mutually exclusive categories:
-
• country in 3 categories (France, US, both countries),
-
• interventions in 3 categories (drug, product of human origin, other),
-
• status in 4 categories (recruiting, not yet recruiting, enrolling by invitation, active, not recruiting),
-
• enrollment in 2 categories (below or above the median),
-
• funding source in 3 categories (industry, mixed, other), with “industry” standing for the industry appearing as the sole funding source, “other” standing for no industry appearing among the funding sources and “mixed” standing for the industry appearing among other funding sources,
-
• primary purpose of the intervention in 3 categories (prevention, treatment, other),
-
• primary outcome of the trial in 3 categories (feasibility, safety, efficacy), in which any study including at least one efficacy outcome among the primary outcomes was classified as efficacy,
-
• design quality in 3 categories prone to discriminate the strength of evidence (low [non-comparative or not randomized], medium [randomized and open-label or single-blind], or high [randomized and either double, triple, or quadruple blind]),
-
• primary predicted completion date in 4 years (2020, 2021, 2022, 2023).
We developed three dichotomous variables related to the drug being tested:
-
• the drug under evaluation includes hydroxychloroquine/chloroquine (yes/no),
-
• the trial includes an arm with an active comparator (yes/no), in which placebo, standard of care and a different dosage of the similar drug are not considered active comparators,
-
• the trial includes at least one drug that was not approved on the market before the pandemic (yes/no).
For trials measuring the efficacy of the treatment for COVID-19, we classified the nature of the outcome in 4 categories related to the direct or indirect effect of the treatment on the clinical outcome (mortality, other clinical outcome, time to clinical evolution, surrogate), in which other clinical outcome includes any other clinical endpoint than mortality, including an ordinal scale of severity of COVID-19, or a type of care that is directly related to clinical status (e.g. admission to hospital or intensive care unit) or a surrogate endpoint that defines a clinical status (e.g. O2 saturation above or below 93%). Time to clinical evolution includes the outcomes defined as the number of days without ventilation. Surrogates include viral load measure and O2 levels without reference to a clinically meaningful threshold.
Descriptive Analysis
We presented data relative to all trials, with a focus on studies prone to guide clinical practice or regulatory approval by aiming to show efficacy in the treatment or prevention of COVID-19. We conducted a sub-analysis focused on trials likely to provide the highest level of evidence on efficacy defined as combining a “high” design, more than 200 patients enrolled, recruiting or enrolling by invitation and, for the trials evaluating the efficacy of treatment, a primary outcome of either mortality or another clinical endpoint.
Statistical Analysis
The exploratory analysis combined a Factor Analysis of Mixed Data (FAMD)Reference Pagès 10 and a Hierarchical Classification on the Principal Components (HCPC),Reference Husson, Josse and Pages 11 first to select a set of variables (dimension and sub-dimensions) that characterized the clinical trials and second to cluster clinical trials based on commonalities and differences relative to these dimensions.Reference Husson, Lê and Pages 12 The FAMD and HCPC were implemented using FactoMineR packages from R software and R softwareReference Lê, Josse and Husson 13 (see Supplemental Digital Content 1, that details the FAMD and the HCPC).
A logistic regression was performed to estimate if variables used in the descriptive analysis and different clusters of clinical trials identified in the HCPC had differential correlations with the design of highest rigor. The logistic regression was performed using R software (see Supplemental Digital Content 2 for details of the logistic regression).
This study was not submitted for institutional review board review as it was based on publicly available data and involved no health records (45 Code of Federal Regulations [CFR] 46.102).
Results
There were 1,324 entries in the original search, of which 200 trials met our cohort entry criteria: 151 in the US, 44 in France, and 5 in both, with more trials including drugs or related in the US compared to France (Figure 1). French trials were less often classified as having high rigor compared to the US (30% vs. 47%), much more often classified with a medium rigor (64% vs. 26%), slightly more often funded outside industry (86% vs. 74%), and much more often had a primary completion date in 2020 (89% vs. 53%). The industry funded alone 2% of the trials in France vs. 15% in the US, and all 5 binational trials.
Characteristics of New vs. Repurposed Drug Trials
As seen in Table 1, most studies (73%) evaluated the repurposing of drugs that were already on the market and were funded without the participation of industry (75%). About one-third (34%) were registered but not yet described as recruiting. Slightly more studies registered as recruiting were funded outside industry compared to the whole set of studies (71% vs. 75%) and studied a drug non-approved for any indication (32% vs. 28%).
The sum of the percentages might not reach 100% because of rounding.
a The column US-FR shows the trials appearing in both countries in the database.
b Other intervention includes 2 trials evaluating a vaccine in the US, 1 trial evaluating the discontinuation of a drug therapy in each country, and 1 trial evaluating a dietary supplement in the US.
c The funding source is classified as either “industry” if industry appears as the sole funding source, as “other” if no industry appears among the funding sources and as “mixed” if the industry appears among other funding sources.
d Any study including at least one efficacy outcome among the primary outcomes was classified as efficacy.
e Other purpose includes 1 trial evaluating the feasability of a drug for diagnosis and 1 trial evaluating the efficacy of a dietary supplement for supportive care, both in the US.
f The drugs that are registered in more than two trials are listed in the table.
g The design is classified according to the level of evidence, defined as “high” for randomized trials with either double, triple or quadruple blinding, as “medium” for randomized open or single-blind trials and, as “low” for non-randomized or non-comparative trials. h Placebo, standard of care and a different dosage of the similar drug are not considered active comparators.
h Placebo, standard of care and a different dosage of the similar drug are not considered active comparators.
i The drugs that are registered in more than five trials are listed in the table.
j Other clinical outcome includes any other clinical endpoint than mortality, including an ordinal scale of severity of COVID-19, or a type of care that is directly related to clinical status (e.g. admission to hospital or intensive care unit) or a surrogate endpoint that defines a clinical status (e.g. 02 saturation above or below 93%). Time to clinical evolution includes the outcomes defined as the number of days without ventilation. Surrogate includes viral load measure and O2 levels without reference to a clinically meaningful threshold.
Among the trials including a non-approved drug (n=55), 21 were funded by industry (19 in the US, 2 in both countries), 9 had a mixed funding source (2 in France, 7 in the US), and 25 were funded outside the industry (4 in France, 21 in the US). Among trials with efficacy as trial outcome (vs. safety or feasibility) (n=43), 19 were funded by the industry (17 in the US, 2 in both countries), 7 had a mixed funding source (2 in France, 5 in the US) and 17 were funded outside the industry (4 in France, 13 in the US).
Characteristics of Treatment vs. Prevention Trials
Among the 178 trials for which efficacy was the primary outcome, the primary purpose of the intervention under evaluation was treatment in 156 trials, prevention in 21 trials, and supportive care in 1 trial.
Among the 21 trials focusing on evaluating the efficacy of preventive strategies, we found no trials of vaccines, which were only in early phases evaluating safety; 18 were repurposing already-approved drugs, including chloroquine/hydroxychloroquine in 13 trials.
Among the 156 trials evaluating treatments for active COVID-19, 55 (35%) were not yet recruiting, 115 (74%) were funded without any support from the industry and 116 (74%) repurposed already-approved drugs. French trials are more likely to have mortality as a primary endpoint compared to the US (17% vs. 15%) less likely to have another clinical endpoint (41% vs. 51%), and more likely to compare several active treatment (22% vs. 16%). The countries are similar in the proportion of trials including hydroxychloroquine/chlo-roquine (73%).
Characteristics of Trials Based on Number of Patients Enrolled
The characteristics of the clinical trials vary when considering the number of individuals included in the trials instead of the number of trials, with a higher share of patients included in trials funded outside the industry, including hydroxychloroquine as a drug under investigation or in prevention trials (See Supplemental Digital Content 3, that mirrors Table 1, according to enrollment). However, this result was mainly driven by one large prevention trial aiming at enrolling 55,000 individuals.
Sub-Analysis
Thirty-one trials that evaluated the efficacy of a drug were classified as having a high-quality design, planned to enroll at least 200 individuals, were registered as either recruiting or enrolling by invitation, and for treatment trials, had either mortality or another clinical endpoint as primary outcome (See Supplemental Digital Content 4, the flow-chart of the selection of the sub-analysis). The subset of trials represents 9% of the trials in France and 17% of the trials in the US. Table 2 details the characteristics of the trials.
a The column US-FR shows the trials appearing in both countries in the database.
b The funding source is classified as either “industry” if industry appears as the sole funding source, as “other” if no industry appears among the funding sources and as “mixed” if the industry appears among other funding sources.
c The drugs that are registered in more than two trials in the main analysis are listed in the table.
d Placebo, standard of care and a different dosage of the similar drug are not considered active comparators.
e The drugs that are registered in more than five trials in the main analysis are listed in the table.
f Other clinical outcome includes any other clinical endpoint than mortality, including an ordinal scale of severity of COVID-19, or a type of care that is directly related to clinical status (e.g. admission to hospital or intensive care unit) or a surrogate endpoint that defines a clinical status (e.g. O2 saturation above or below 93%).
g The severity of disease in the inclusion criteria was classified as either mild (no hospitalization), moderate (hospitalization without oxygen need), severe (hospitalization with oxygen support), or critical (need for mechanical ventilation). One trial could include patients with different stages of the disease.
Among the 31 trials, the 4 studies comparing several active treatments included only drugs that were already approved and were all funded by non-industry sponsors. The industry funded, partially or completely, 7 of the 8 trials including non-approved drugs and 6 of the 23 trials including only drugs that were approved for another condition before the pandemic, as well as 11 of the 15 studies not including hydroxychloroquine/ chloroquine, and 2 of the 16 including hydroxychloroquine/chloroquine.
Factorial Analysis and Taxonomy
The FAMD resulted in the selection of three factorial axes — which account for 31% of the variance — and the HCPC, based on the FAMD, in a 3-cluster partition (see Supplemental Digital Content 1, which details the FAMD and the HCPC).
Cluster 1 (59% of the trials) over-represents trials repurposing drugs already approved, evaluating the efficacy of a treatment against COVID-19, funded outside the industry, having a design of medium rigor, being registered in France, having either a clinical outcome that is not mortality or a surrogate outcome, comparing several active treatments, evaluating a drug, not yet recruiting, and enrolling under the median number of patients.
Cluster 2 (19% of the trials) over-represents trials funded by the industry, evaluating an unapproved drug, recruiting, not including chloroquine or hydroxychloroquine, registered both in the US and in France, having a design of high rigor, having an outcome defined as time to clinical evolution, aiming at evaluating a treatment of COVID-19, evaluating a product of human origin, aiming at evaluating the efficacy of an intervention, not comparing a treatment to another active treatment, enrolling over the median number of patients, and having a primary completion year in 2023.
Cluster 3 (22% of the trials) over-represents trials not evaluating the efficacy of a treatment, evaluating an intervention for the prevention of COVID-19, having a primary outcome related to safety, having a design of low rigor, having a primary outcome of feasibility, being registered in the US, and not comparing a treatment to another active treatment.
Logistic Regression
The first logistic regression based on single variables shows that, at a statistical significance threshold of p<0.05, trials with efficacy as outcome (OR: 4.5), size of enrollment larger than the median (OR: 3.2), including a non-approved drug (OR: 2.8), and registered in France (OR: 2.5) were associated with the probability of the trial design being classified with a high rigor. With a statistical significance threshold of p<0.10, trials not including an active comparator (OR: 2.7) were also significantly associated with the highest level of rigor. Fit statistics show that the model explains 15% to 26% of the variance depending on the fit criteria (see Supplemental Digital Content 2, text that details the logistic regressions).
As seen in Table 3, a second logistic regression analysis, taking the clusters as explanatory variables, shows that trials belonging to cluster #2 were strongly associated with a high level of design compared to cluster #3 (OR: 3.4). Trials in cluster #1 were less likely to be classified with a design of high rigor than cluster #3, although the difference did not reach statistical significance. Fit statistics show that the model explains 6 to 9% of the variance depending on the fit criteria (see Supplemental Digital Content 2, text that details the logistic regression).
Note: the cluster significantly correlated at the 5% threshold is in bold.
Discussion
The large number of clinical trials registered in the early weeks of the pandemic illustrates substantial effort from investigators and regulatory authorities. However, only a fraction of the trials had a design with a high level of rigor that would be most likely to provide robust evidence to help clinicians and patients manage COVID-19.
The quantitative exploratory framework clusters trials by commonalities and differences, identifies and describes the characteristics that bring them in the same cluster, and analyzes the correlation between the rigor of the design and the clusters. Our analysis shows that trials most likely to be associated with a design of high rigor, as described in cluster #2, tend to evaluate the efficacy of a treatment, with a focus on unapproved drugs, compared to placebo or standard of care, and have outcome measured as a time to clinical evolution. These trials are more often funded by the industry compared to other trials of treatments for COVID-19. While these trials are designed to answer important questions, their profile will not provide evidence on critical dimensions. For example, the multiplication of rigorous trials would raise the chance to obtain reliable results about effective treatments. However, if several drugs show some efficacy, clinicians and policymakers would still require comparative data to determine which are the best. Optimally useful trials could therefore include several drugs, either in a direct comparative or adaptive design in which patients can switch enrollment from one arm to another based on the first observed results.Reference Bhatt and Mehta 14 However, among trials registered in ClinicalTrials.gov, only a small fraction — 29 among the 178 trials evaluating efficacy — include several therapeutic options simultaneously. Only one trial was registered as cost-effectiveness analysis. Our results may suggest different broad patterns in the research strategy in the two countries: a publicly funded repurpose of several readily available drugs where double-blind designs are difficult and costly to run in the short-term, in France, vs. a commercial development of drugs already under development, in which doubleblind designs are crucial to obtain the regulatory endorsement of drugs not yet marketed, in the US. An analysis of the relative benefits and drawbacks of the different types of trials may support choices in a research strategy to address future pandemics.
Four characteristics of trials likely to provide relevant evidence were underrepresented in trials associated with the highest level of rigor. They include the comparison to another active treatment, an outcome defined as either reducing mortality or achieving clinical endpoint rather than reducing the time to achieve it, the repurposing of already-approved drugs, and independent funding. Trials including these features were overrepresented in cluster #1, which was not associated with high levels of rigor in trial design.
Our descriptive findings confirm previous work related to initial COVID-19 clinical trials. 15 The weakness of design in many trials has turned out to be even more problematic as clinicians, and health systems have started making decisions on the basis of the fast-track publication process before peer-reviewReference Kubota 16 and emergency use authorizations based on the limited available evidence.Reference Rome and Avorn 17 The multiplication of trials also raises the question of the cost-effectiveness of the research itself. Although producing evidence from different independent investigators is worthwhile to ensure the reproducibility of the results, too much multiplication of similar studies risks becoming wasteful, especially if the trials do not reach a high level of rigor. The studies of hydroxychloroquine — and the even higher share of patients included in a single trial evaluating this drug — illustrate this risk, as a large number of trials, and competition among investigators undermines trial enrollment. Indeed, a French initiative to lead a European trial failed to include patients beyond France when other countries developed their own clinical trials. 18 Better cooperation in organizing the early research effort and incentives to link public funding to rigorous design might support a more cost-effective use of limited resources.
While the drugs under evaluation were private goods distributed by private companies, we found that a surprisingly low number of trials were funded by industry, while several studies were directly funded by the government or other public funds. This may be because the most commonly studied drugs — hydroxychlo-roquine and chloroquine — are available generically or at low cost in many countries. Over four-fifths (83%) of subjects studied were included in trials funded outside the industry in the US and France. Moreover, the funding labeled as industry might include public money through the fiscal support of private research.Reference Cleary, Beierlein, Khanuja, McNamee and Ledley 19 However, if a drug proves to be effective and safe enough to be used during the pandemic, there is no guarantee that the government or health insurance would be able to access the drug at a fair price. This is a particular issue for patent-protected drugs like remdesivir, which at the time of this writing remains limited to Emergency Use Authorizations and expanded access programs implemented by the firm itself in a limited number of countries. 20 Meanwhile, several other countries in the European Union have requested more extensive access.Reference Francisco 21 Even in the case of low-price drugs, like hydroxychloroquine, the company selling a branded version of the drug in France without generic competition receives the benefit of the largely publicly funded research programs. Private companies should not be the only beneficiaries of the research they do not fund, and public support to a research program should be associated with the guarantee of fair access and price for drugs proved to be sufficiently effective and safe.
The large number of clinical trials registered in the early weeks of the pandemic illustrates substantial effort from investigators and regulatory authorities. However, only a fraction of the trials had a design with a high level of rigor that would be most likely to provide robust evidence to help clinicians and patients manage COVID-19.
An over-arching question relates to the production and diffusion of the results of the trials. The widespread availability of results will help to address the potential next waves of the current pandemic or future pandemics with other corona-viruses. Indeed, the repositioning of drugs has been facilitated by previous research on the previous outbreaks of Middle East Respiratory Syndrome (MERS) and severe acute respiratory syndrome (SARS). While results are supposed to be listed in ClinicalTrials.gov within a year after the primary completion date, the database has reported deficiencies in filing results,Reference DeVito, Bacon and Goldacre 22 and there is no guarantee that final results will become available for each trial, for several reasons, including difficulties in enrolling patients as planned given the progression of the pandemic.Reference Wang, Zhang, Du, Du, Zhao and Jin 23 We found that 33% of the studies were not recruiting in early May while the pandemic was showing signs of slowing down in France. It also includes the risk of the interruption of trials following results from other studies, as illustrated by the decision by the French Medicines Agency on May 26, 2020 to suspend enrollment in trials involving hydroxychloroquine, 24 or a possible lower publication rate of inconclusive or negative trials.Reference Fanelli 25 Legislators in the US and France should consider special laws to ensure that all de-identified data relating to COVID-19 trials be posted in a special registry.
Some inherent limits of our data should be considered. The SARS-CoV-2 outbreak is an evolving situation, and new trials are registered almost daily. Thus, our study focused only on the trials registered during the early months of the pandemic; later trials might show different patterns. Many trials were registered as not yet recruiting, but their status may have changed since the registration of the study. The selection of the studies does not include the trials registered as expanded access programs (the database displays 17 such studies on May 8, 2020) nor pharmacoepidemiology studies based on the use of drugs in routine practice that may give complementary answers about effectiveness and safety.Reference Boyer, Auquier and Fond 26 ClinicalTrials.gov database may not register all clinical trials, especially in France. It is however widely used in France, including by regulatory authorities. 27 Only a limited set of variables were documented in the database variables that were not included, among which the effective inclusion of patients and the outcomes of the study, would be worth to explore in future research.
Our analysis considered only protocols registered in the database. Thus, our classification of primary outcomes could not integrate any insight on the level of effect, and the meaning of a result could depend on how the outcome is measured. An example of this risk relates to the use of the 7-level ordinal scale promoted by the World Health Organization, for which the change of one level may have a different clinical significance depending on the initial status of the patient.
Conclusion
In summary, we found a dynamic research landscape with numerous clinical trials registered in the two most active countries. However, the data collected raise concerns about the extent to which the trials will address unmet needs, as only a minority have rigorous designs likely to provide a high level of evidence. The most useful studies for comparative effectiveness, i.e., that evaluated several interventional strategies were all funded by non-industry-related sponsors, as were the studies with the largest enrollment. By contrast, studies of non-approved drugs were more likely to be funded by industry. Incentives for cooperation could help promote a more efficient research landscape with high-quality standards that would avoid unnecessary repetition, ensure the publication of the results, and guarantee fair access and price for drugs proved effective and safe based on publicly funded research.
Acknowledgments
The authors want to thank the member of the PORTAL team whose comments contributed to improving the first presentation of the research project.
