In 1948, the Medical Research Council’s trial of streptomycin for the treatment of pulmonary tuberculosis became the first published randomized controlled trial (RCT) in medicine.1 The RCT design is now considered to be the gold standard for generating rigorous and reliable data to inform clinical decision making.Reference Bhatt2 Randomization allows for causal conclusions because — assuming the trial is sufficiently large — both measured and unmeasured baseline covariates are balanced.Reference Kendall3 Randomization can also ensure that patients have a consistently defined study start date, which prevents immortal time bias.Reference Suissa4 When investigators conducting RCTs blind participants, clinicians, and researchers, the risk of ascertainment bias and performance bias are minimized, which helps to prevent time-varying confounding.5 Maintaining the rigor of a well-designed and well-conducted RCT requires considerable time and financial investment. The average cost of an RCT varies depending on the disease area, but in the United States the cost of Phase 3 RCTs range, on average, from $12 million to $53 million.Reference Sertkaya6 These costs are often prohibitive for academic investigators.
The potential for bias in industry-sponsored trials has been well documented, and has led to important questions about whether corporate entities with a financial interest in the outcome of trials should be so closely involved in the research.Reference Chopra7 Concerns have been raised that industry-sponsored trials may be terminated early for financial reasons rather than scientific or ethical reasons, and that academic investigators who receive corporate funding may be incentivized to bias the analysis and reporting of trial results. Further, there have been reports of industry choosing to selectively present positive results and withhold negative results. Among physicians, these documented instances have fostered mistrust in industry-sponsored trials.Reference Bodenheimer8 One well-cited example is GlaxoSmithKline’s marketing of Paxil (paroxetine) for adolescent major depression. The initial study publication in 2001 indicated that paroxetine for treatment of adolescent major depression was generally well tolerated, and demonstrated significantly greater improvement compared with placebo.Reference Keller9 However, a 2015 post-publication analysis of the original raw data revealed that GlaxoSmithKline had manipulated the data and selectively downplayed the harms of the drug: in fact, paroxetine is no better than placebo for any prespecified primary or secondary efficacy outcome, and use could result in serious side effects including self-injury and suicide.Reference Le Noury10
There is consequently — and understandably — a perception among clinicians that RCTs funded by industry may be of lower quality. In a 2012 study, Kesselheim et al.Reference Kesselheim11 created a series of abstracts for hypothetical clinical trials and randomly assigned to each the designation of either pharmaceutical company funding, federal funding, or no funding. Board-certified internists were then asked to evaluate the quality of the hypothetical trials. Despite the content of the abstract being otherwise identical, physicians who received the version indicating funding by a pharmaceutical company perceived the methodologic quality more negatively than physicians evaluating the exact same abstract with an indication of either federal funding or no funding. Systematic analysis of how trial characteristics differ between industry-funded and non-industry-funded trials may offer insight into the extent to which preconceived notions of study quality based on funder are accurate. Our objective was to compare study characteristics of RCTs funded by industry to those of RCTs not funded by industry.
Systematic analysis of how trial characteristics differ between industry-funded and non-industry-funded trials may offer insight into the extent to which preconceived notions of study quality based on funder are accurate. Our objective was to compare study characteristics of RCTs funded by industry to those of RCTs not funded by industry.
Methods
Study Population
We reviewed all RCTs published in the New England Journal of Medicine (NEJM), the Journal of the American Medical Association (JAMA), and Lancet between January 1, 2015 and December 31, 2019. These journals were selected because they have the highest impact factors among general medicine peer-reviewed journals, and they commonly publish RCTs. We chose to stop data collection at the year 2019 (inclusive) because the advent of the COVID-19 pandemic may have changed the landscape of RCT publishing. Excluding trials after 2019 also afforded at least two years of follow-up in the evaluation of post-publication metrics for each RCT. Using MEDLINE, we identified all articles published in these journals during the study period and then excluded review articles, research letters, letters to the editors, and editorials. The title and abstract of the remaining articles were reviewed independently by three study members (MF, UP, AR) to identify and exclude any remaining non-randomized trials and duplicate publications. Disagreements were resolved through consensus. A detailed outline of the study selection process is provided in Figure 1.
Study Flow Diagram
Legend: NEJM = New England Journal of Medicine, JAMA = Journal of the American Medical Association, RCT = randomized controlled trial
RCT Data Collection
For each RCT, data were manually extracted from the study abstract. These datapoints included sample size, blinding, comparator type (i.e., placebo, active comparator, other), disease area, outcome type (i.e., surrogate outcome or notReference Fralick12), the study’s conclusion as written in the abstract, the primary outcome result (i.e., positive, neutral, or negative), and funder type. Funder type was determined through manual review of the manuscript and classified as one of “industry-funded,” “non-industry-funded,” or “combination funded.” Combination funded was defined as trials that included both industry and non-industry funders. We then combined “industry-funded” and “combination funded” into one group, classifying the resultant merged group as “industry-funded.” When pharmaceutical companies donated the trial intervention (e.g., drug, device, placebo) but gave no monetary funding, the RCT was considered non-industry-funded. The study outcome was considered positive if the point estimate for the primary outcome identified a benefit and the 95% confidence intervals excluded the null. The study outcome was considered negative if the point estimate for the primary outcome was below the null and the 95% confidence intervals excluded the null. The study outcome was considered neutral if the point estimate included the null. For non-inferiority trials, the study outcome was considered positive if the study was superior or non-inferior. To determine whether results were posted online on ClinicalTrials.gov, we used the site’s API.
Knowledge Dissemination Data Collection
Altmetric scores were obtained for each RCT. Almetric scores quantify an aggregate of an article’s online attention by including data on volume of mentions, author of online mention (e.g., reporter), and mention source (e.g., Twitter, blog, news report). Citation counts were obtained for each RCT and used as a further measure of knowledge dissemination. Altmetric data, including Altmetric Scores and their breakdown, were obtained using the Altmetric Details Page API via the rAltmetric R package. The number of citations was obtained through the CrossRef API via the rcrossref R package. Using the Digital Object Identifier (DOI) of the peer-reviewed article, data were merged from various sources.
Study Outcomes & Statistical Analysis
We compared the frequency of double blinding, inclusion of placebo, sample size, and posting of trial results on ClinicalTrials.gov between industry-funded trials and non-industry-funded trials. We also analyzed the sentiment of each study’s conclusion using two different transformer-based models (a neural network-based technique for natural language processing). The models we used have been pre-trained on a large corpus of English text in a self-supervised fashion and routinely applied in sentiment analysis.Reference Yang13 Models 1 and 2 both used BERT, a bidirectional transformer model pre-trained using masked language modeling and next sentence prediction.14 Model 1 classified responses into either positive or negative, while Model 2 indicated the sentiment of the response as a number of stars (between 1 and 5, where 1 is highly negative and 5 is highly positive). Descriptive statistics were used to compare industry-funded RCTs to non-industry-funded RCTs. All analyses were conducted using R version 3.1.2.5.
Results
We identified 1533 RCTs published in NEJM, JAMA and Lancet between January 1, 2015 and December 31, 2019. Most of the RCTs were trials of medications. Approximately one-third were double-blind, approximately 40% had a surrogate outcome as their primary endpoint, and approximately two-thirds had a positive primary outcome (Table 1). Of all the RCTs, 697 were funded by industry while 835 were not; one study lacked funding details altogether.
Baseline characteristics
Legend: SMD = standardized mean difference; JAMA = Journal of the American Medical Association; NEJM = New England Journal of Medicine; IQR = Interquartile range. *Overall count includes one study that lacked funding details altogether. †Remaining studies had a comparator that was neither active nor placebo. Overall outcomes that were negative or neutral were grouped as “Negative.”
Trials funded by industry were more likely to be related to cardiovascular disease or oncology, and less likely to be related to infectious disease (Table 1). The median sample size was 557 (interquartile range [IQR]: 230, 1369) for industry-funded trials and 648 (IQR: 301, 1916) for non-industry-funded trials. Compared to non-industry-funded trials, industry-funded trials were more likely to be double-blind, more likely to include a placebo, and more likely to have a surrogate primary outcome (Table 1). The primary outcome was positive for 82% of industry-funded RCTs compared to 54% of non-industry-funded RCTs.
We found that the concluding statements of trials funded by industry were more likely to have greater positive sentiment than those not funded by industry (n=246, 35% vs. n=208, 25%), which aligns with our finding that industry-funded trials were more likely to have positive findings. Our findings were robust across two different natural language processing (NLP) methods (Table 2).
Primary outcome of the study compared to the sentiment of the abstract.
Legend: NLP = natural language processing. Overall outcomes that were negative or neutral were grouped as “Negative.”
Industry-funded trials accrued twice as many citations as non-industry-funded trials (n=285 [IQR: 140, 562] vs. n=145 [IQR: 76, 266], p < 0.01). AltMetric scores were similar for all included studies, regardless of their funding source (n=229 [IQR: 122, 468] for industry-funded; n=226 [IQR: 119, 441] for non-industry-funded, p=0.2). Industry-funded trials were also more likely to post their results on ClinicalTrials.gov (n=443 of 570, 78% vs. n=207 of 501, 41%) compared to non-industry-funded trials.
Discussion
In this study of RCTs published in NEJM, JAMA, and Lancet between 2015 and 2019, we identified that trials funded by industry were more likely to be blinded, more likely to be placebo controlled, and more likely to post trial results on ClinicalTrials.gov. These data suggest that on certain key metrics of trial quality, industry-funded trials may perform better than non-industry-funded trials, adding important and often overlooked nuance to the broader discussion about industry-funded research.
The greatest strength of a large RCT is that both measured and unmeasured confounders are balanced. However, without blinding it is almost impossible to prevent time varying confounding or ascertainment bias, both of which can directly affect the internal validity of a study, especially when the primary outcome is subjective or prone to measurement error. Trials funded by industry may have been more likely to include blinding and a placebo for several reasons. First, industry is less likely to perform pragmatic trials that test an intervention in real-world practice; this absence of real-world conditions makes it more feasible to include double-blinding and a placebo control. Second, industry entities manufacture not only the drug but also the matching placebo, facilitating easy access. Third, placebos are expensive, and industry-funded trials are typically better resourced than non-industry-funded trials.Reference Gelijns and Thier15 Fourth, there is often little incentive for a pharmaceutical company to use an available medication as their comparator, as opposed to a placebo; importantly, this can be problematic in cases where the use of an active comparator instead of placebo would have generated more clinically relevant data.
Taken in sum, our findings suggest that industry-funded RCTs published in high-impact journals (i.e., NEJM, JAMA, Lancet) are more likely to be blinded, more likely to include a placebo, and more likely to post trial results on ClinicalTrials.gov. Industry funding has been instrumental to major trials with clinical importance, so a culture of systematically undervaluing such trials is concerning. Our findings emphasize the importance of evaluating the quality of an RCT based on its methodological rigour, not its funder type.
Another important consideration is the primary outcome itself. We observed that industry-sponsored trials were more likely to utilize surrogate outcomes, which are arguably less clinically relevant to patients and clinicians. And while we did observe that industry-funded trials were smaller than non-industry-funded trials, the absolute difference was modest.
We found that industry-funded trials were more likely to make their results publicly available on ClinicalTrials.gov. Our results align with a recent study of over 4,000 trials registered on ClinicalTrials.gov which identified that while two-thirds of the registered trials posted their results, the odds of this occurring were threefold higher for industry-sponsored trials compared with non-industry sponsored trials.Reference DeVito, Bacon and Goldacre16 Similarly, Anderson et al. (2015) found that adherence to legal obligations (outlined by the Food and Drug Administration Amendments Act) for timely reporting of results within 12 months of trial completion was higher among industry-sponsored trials than NIH funded or academic institution-funded trials.Reference Anderson17 This observation might reflect that industry-funded RCTs are more likely to be for new molecules, which legally obligates the investigators to adhere to stricter regulatory reporting.
While reporting compliance appears to be higher among industry-funded trials, it is important to highlight prior literature documenting publication bias among this group. There are instances of industry-funded trials not publishing at all if the trial was negative, or choosing to present positive results while withholding negative results.Reference Ross18 Consider the previously cited example of Paxil by GlaxoSmithKline, which was prescribed to adolescents for years before it was confirmed not only to be ineffective, but to cause serious side effects including self-injury and suicide.19 Or the example of Roche’s Tamiflu (oseltamivir): the initial results from industry-funded trials indicated it was effective in reducing hospitalizations and serious complications from influenza, which resulted in governments stockpiling Tamiflu in advance of potential influenza outbreaks.Reference Gupta, Meenu and Mohan20 However, a Cochrane review subsequently identified that 60% of patient data from the randomized placebo controlled phase 3 treatment trials were never published, and that the exclusion of these data significantly changed the findings; in reality, oseltamivir was less effective in reducing complications of influenza than previously stated.Reference Jefferson21
While our study found that trials funded by industry were more likely to have a positive outcome, there are multiple reasons why this might be the case. Publication bias may explain why there was a higher number of positive industry-funded trials; if results aren’t positive, researchers are less likely to pursue publication. Industry-funded trials might be more likely to be positive because such studies focus their resources on drugs and devices that are more likely to work based on earlier phase trials, or involve comparison against placebo rather than an established, effective medication. We also found that the concluding statements of trials funded by industry were more likely to have greater positive sentiment than those not funded by industry, which aligns with the greater likelihood of industry-funded trials having positive outcomes. Our study did not find evidence to either refute or support the claim that industry-sponsored research chooses to withhold negative results.
One limitation of our study is that we lacked data on publication bias because we only included trials that were published. Another limitation is that we focused on three of the highest-impact medical journals, and thus our results may not apply to RCTs published in other medical journals. Finally, we lacked data on other aspects of trial design that can directly affect internal validity, including concealment of allocation, number of study sites (e.g., single-center compared to multi-center), adherence, statistical analysis (e.g., intention to treat), and loss to follow-up.
Taken in sum, our findings suggest that industry-funded RCTs published in high-impact journals (i.e., NEJM, JAMA, Lancet) are more likely to be blinded, more likely to include a placebo, and more likely to post trial results on ClinicalTrials.gov. Industry funding has been instrumental to major trials with clinical importance, so a culture of systematically undervaluing such trials is concerning. Our findings emphasize the importance of evaluating the quality of an RCT based on its methodological rigor, not its funder type. Our observations also raise questions about how non-industry-funded research needs to improve reporting of results to ClinicalTrials.gov. Further research is needed to determine trial characteristics of industry-funded research that is not published and industry-funded research that is published in other journals, as the findings from this current study may not be generalizable to these other areas.
Note
This study was funded by Mount Sinai Hospital Department of Medicine Research Fund. MF has received multiple grants from CIHR for investigator initiated clinical trials. MF was a consultant for ProofDx, a start-up company creating a point-of-care diagnostic test for COVID-19 using CRISPR. MF is an advisor for SIGNAL1, a start-up company deploying machine learned models to improve inpatient care.
