Peer review is essential to the normative well-functioning of science. It calls upon domain experts to scrutinize the soundness, significance, and novelty of research for journal publication, conference presentations, and grants. The rigor of this “system of institutionalized vigilance” (Merton Reference Merton and Storer1973: 339) bestows credibility upon successful submissions and, in so doing, bestows credibility upon the community and its mechanisms for self-governance.
In addition to policing the content of science, peer review provides a system of incentives motivating scientists to disclose findings in exchange for the prestige that comes with successful peer-reviewed publication and grant awards. It adjudicates the reward system in science by providing a mechanism by which the prestige of journals, editorial bodies, and granting agencies can be conferred upon scientists who successfully meet their stringent standards. The more exclusive and refined the gate-keeping, the more incentive scientists have to pass through the gate.
The beauty of the traditional system of peer-reviewed publication, from the perspective of the social structure of science, is that the same mechanism – peer review – serves normative and motivational functions simultaneously: ambitious scientists do well professionally by doing well epistemically. This suggests a mechanism by which scientific communities should be able to align author behavior with community-wide epistemic goals: we should expect journal editors, in their dual role as epistemic evaluators and prestige-allocators, to have the capacity to motivate improved author behavior (Zuckerman and Merton Reference Zuckerman and Merton1971: 66).Footnote 1
This paper will evaluate the success of this strategy in the context of academic medicine. Medical journal editors have periodically updated publication standards as a direct response to emerging empirical research on varieties of publication bias discovered in prestigious, peer-reviewed medical journals. The publication bias literature has diagnosed two major ways in which author behavior fails to align with the disciplinary goal of improving clinical practice. First, the disproportionate publication of positive outcomes leads to false and exaggerated effect sizes in meta-analyses (Simes Reference Simes1986). Second, authors have adopted dodgy strategies to craft positive outcomes that fail to answer the most clinically relevant questions (Smith Reference Smith2003: 1204). Unfortunately this latter phenomenon belongs to a broader trend of publication gamesmanship by authors across the sciences.
In the first part of the paper, I will discuss the co-emergence of the motivational and epistemic functions of peer-reviewed publication, motivate the hypothesis that stricter publication standards should improve author behavior, and introduce the institutions (created, constituted, and endorsed by medical journal editors) to leverage this proposed mechanism for change. Second, I will articulate the discipline-wide epistemic goal that medical journal editors hope to achieve through their interventions and explain why publication bias is prima facie normatively appropriate in relation to this goal. Third, I will evaluate editor efforts to debias trial data available for meta-analysis. Fourth, I will evaluate editor efforts to stop more pernicious forms of publication gamesmanship. To close, I will identify further actions – by members of the scientific community and federal regulators – that might help to better align author behavior with academic medicine's goal of improving clinical practice.
1. CHANGING AUTHOR BEHAVIOR BY CHANGING PUBLICATION STANDARDS
Peer-reviewed publication has long incentivized individual scientists in ways that have structured scholarly communication as we know it today. We can see how these incentives work by tracing peer review's history all the way to its first implementation in The Philosophical Transactions of the Royal Society back in the 1750s.Footnote 2 Before that time, scientists had set “a premium upon secrecy” as a way to safeguard their intellectual property from competing scientists (Zuckerman and Merton Reference Zuckerman and Merton1971: 69) – at least, until they had “amass[ed] a vast quantity of material, or. . . conceive[d] a complete new ‘world system’” sufficient for publishing a monograph-length treatise (Ziman Reference Ziman1969: 318). In contrast, the Philosophical Transactions provided scientists the opportunity to archive and establish priority for ideas that fit into article-length papers (Zuckerman and Merton Reference Zuckerman and Merton1971: 70). As the functioning editor Henry Oldenberg wrote to Robert Boyle, the journal “inviolably preserved to all posterity” a scientist's intellectual ownership of particular discoveries and experiments (Oldenberg Reference Oldenberg, Rupert Hall and Hall1966: 70).
However, archiving and establishing priority for smaller pieces of work were not the sole motivations for scientists. The Royal Society, concerned to protect the reputation of their Society and journal, established a new form of social technology: namely, peer review. From a sociological point of view, this turned the scientists at the Royal Society – in their capacity as reviewers and editors – into “status judges” charged with evaluating and allocating rewards for successful role-performance (Merton and Zuckerman Reference Merton, Zuckerman and Storer1971: 467–8). The reward was not simply publication, but prestige: recognition in a forum valued by other scientists whose opinions were valued by submitting authors.
In contemporary times, peer-reviewed publication continues to provide a system of incentives motivating scientists to formulate and disclose experiments and methods in exchange for peer recognition. Editors and editorial board members are selected for characteristics associated with prestige – such as educational background, scholarly contributions, and publication experience (Weller Reference Weller2001: 111). Selecting editors for these qualities burnishes the reputation of the journal and the vetted contents therein. Selectivity is sometimes measured through proxies such as rejection rates.Footnote 3 The rough idea behind this metric remains the same: the more selective the peer-run gate-keeping process, the more prestigious the venue, and the more incentive for individual authors to publish there.Footnote 4
From a purely sociological point of view, we should expect journal editors to have the epistemic authority and political power to motivate authors in ways that realign their behavior with community-wide epistemic goals.Footnote 5 Academic medicine is a good context for evaluating whether this expectation pans out. Prominent medical journal editors have explicitly sought to improve author behavior through the articulation and endorsement of revised publication norms. And, a thriving community of publication bias researchers has carefully tracked the impact of these interventions on the health of the publication record.
Publication bias research, as a genre of empirical study, received an institutional home with the inauguration of the International Congress on Peer Review and Biomedical Research, first started in 1986 and sponsored by the Journal of the American Medical Association. Soon after, in the early 1990s, editors for top general medical journals organized to articulate improved guidelines for the reporting of randomized controlled trials – guidelines that became the Consolidated Standards for Reporting Trials (CONSORT) statement of 1996.Footnote 6 This document has been regularly updated and extended in response to emerging evidence on harmful trial reporting behavior. The CONSORT statements are published by the most prestigious medical journals, endorsed by hundreds more, and explicitly referred to in journal instructions to authors (Altman Reference Altman2005). A second institution, the International Committee of Medical Journal Editors (ICMJE), constituted by editors at top general medical journals across the world, publishes and periodically updates a statement on Uniform Requirements for Manuscripts Submitted to Biomedical Journals that is endorsed by thousands of medical journals. The statement first emerged in 1979, primarily as a style guide, but has since shifted its focus towards methodological concerns (ICMJE 2004).
The strategy of improving author behavior through changes in publication standards had an auspicious start. Large reviews demonstrated significant improvements in reporting for those journals adopting the first CONSORT statement compared with journals that did not (Plint et al. Reference Plint, Moher and Morrison2006; Kane et al. Reference Kane, Wang and Garrard2007); and, among CONSORT-adopting journals, a review found improvements in reporting post-CONSORT (Moher et al. Reference Moher, Jones and Lepage2001). The magnitude of the percentage changes suggest that the 1996 statement was successful in shifting author behavior: for example, failing to explain subject attrition (Kane et al. Reference Kane, Wang and Garrard2007) or report allocation concealment (Moher et al. Reference Moher, Jones and Lepage2001) decreased from the 60%+ range down to 16% and 31% respectively.Footnote 7 This result provides a kind of proof of concept that this strategy for realigning author behavior with the community's goals can work in the real world, at least in certain conditions.
2. MEDICINE'S GOALS, PUBLICATION NORMS, AND PUBLICATION BIAS
Unlike many scientific disciplines (Kitcher Reference Kitcher2001: 145), academic medicine has explicitly articulated its primary epistemic goal. Textbooks tell us that the “best research” evidence “invalidates previously accepted diagnostic tests and treatments and replaces them with new ones that are more powerful, more accurate, more efficacious, and safer” (Sackett et al. Reference Sackett, Straus, Richardson, Rosenberg and Haynes2000: 1): the most scientifically significant research is that which improves clinical practice. This idea gets echoed in medical journal instructions to authors. The Lancet “prioritises reports of original research that are likely to change clinical practice or thinking about a disease” (The Lancet 2011). The Journal of the American Medical Association's “highest priority for publication is given to reports of original research that present important, potentially practice-changing findings” (Fontanarosa and DeAngelis Reference Fontanarosa and DeAngelis2004). And, the British Medical Journal takes its mission to be “to engage, inform, and stimulate doctors, researchers and other health professionals in ways that will improve outcomes for patients” (British Medical Journal 2011).
How can we fulfill this lofty goal? First off, authors must, when evaluating new treatments, provide studies demonstrating improved efficacy or safety – that is, they must demonstrate a positive direction in favor of the experimental hypothesis, where larger effects are more valued than smaller effects (Sackett et al. Reference Sackett, Straus, Richardson, Rosenberg and Haynes2000: 4). This is not to say that negative results are not relevant to changing clinical practice. Demonstrating the inefficaciousness of currently adopted therapies can improve clinical practice: important examples include trials on the relative inefficaciousness of standard drugs used to treat depression (Kirsch et al. Reference Kirsch, Moore, Scoboria and Nicholls2002) and the inefficaciousness of hormone replacement therapy to prevent heart disease in postmenopausal women (Hulley et al. Reference Hulley, Grady and Bush1998; Grady et al. Reference Grady, Herrington and Bittner2002). However, the assumption here is that most currently adopted clinical practices are efficacious to some degree, which turns the primary task to be one of improving upon standard treatment.
Secondly, in order for the conclusions of their studies to be sufficiently robust to ground recommendations for changing current practice, researchers must provide statistical measures of significance, power, or confidence to demonstrate that the findings were unlikely to result from chance alone. Statistical significance measures provide an estimate of the strength of the evidence against the null hypothesis (Sackett et al. Reference Sackett, Straus, Richardson, Rosenberg and Haynes2000: 233). In general, a publication preference for positive, statistically significant outcomes favoring a new experimental treatment serves medicine's mission of improving and transforming current clinical practice by adding better treatments.
So, it shouldn't be surprising that top general medical journals have a preference to publish positive outcomes as opposed to negative outcomes. “Positive outcomes” include results that have a positive direction (Bardy Reference Bardy1998: 147), are statistically significant (Dickersin et al. Reference Dickersin, Min and Meinert1992; Dickersin and Min Reference Dickersin and Min1993; Stern Reference Stern1997), or both (Ioannidis Reference Ioannidis1998: 282). To be consistent with the literature, I'll use “positive outcomes” to refer to this disjunction of result types. Likewise, “negative outcomes” include a disjunction of result types, including results that have a negative direction, are not statistically significant, or both.
Publication bias is the tendency for journals to publish studies reporting positive outcomes. A large study revealed that, of the studies submitted to the Journal of the American Medical Association, 20.4% of those with statistically significant results on the primary outcome were accepted. This is slightly higher than the base rate for publication at the journal, which was 17.9%. In comparison, 15.0% of studies with negative results were accepted, which lies below the base rate for publication (Olson et al. Reference Olson, Rennie and Cook2002). Reporting positive outcomes raises author chances of publication above the base rate, while reporting negative outcomes lowers author chances of publication below the base rate.
We can see much more dramatic publication bias comparisons when we shift our focus from the population of studies submitted for journal review to the larger population of studies approved by ethics boards – including papers that will never get submitted for journal review. Table 1 suggests that, among studies approved by ethics boards, those with positive outcomes are much more likely to be published than those with negative outcomes.
Percentage of trials approved by ethics boards that were later published
Why does the publication bias differential shift so dramatically when we look at the larger population of studies that are and are not submitted for journal review? It turns out that authors are selective in their choices about which trials to submit for journal review. A large study at Johns Hopkins found that over 90% of trials approved by its institutional review boards remained unpublished due to the inaction of investigators (Dickersin et al. Reference Dickersin, Min and Meinert1992: 377).Footnote 9
Self-reported reasons for non-submission reveal both epistemically virtuous and sullied considerations: authors report non-submission due to insufficient importance or interest of negative results as well as unfavorable likelihood of journal acceptance (Easterbrook et al. Reference Easterbrook, Berlin, Gopalan and Matthews1991; Dickersin et al. Reference Dickersin, Min and Meinert1992; Dickersin and Min Reference Dickersin and Min1993; Weber et al. Reference Weber, Callaham, Wears, Barton and Young1998). Further considerations include positive outcomes' higher citation counts (Ioannidis Reference Ioannidis2005a) and the strength they lend to grant applications (Ross et al. Reference Ross, Mulvey, Hines, Nissen and Krumholz2009: 6).
3. EDITORIAL EFFORTS TO DEBIAS META-ANALYSIS MEASURES
Unfortunately, the systematic publication of positive outcomes skews the sample of published results towards positive outcomes, which makes it difficult – if not impossible – to measure true treatment effects. Publication bias for results favoring the experimental hypothesis can create effects where there are none: for example, meta-analysis of published trials suggested a survival advantage when using combination chemotherapy over an initial alkylating agent in the treatment of ovarian cancer; but, when results from all registered trials were pooled, the survival advantage disappeared (Simes Reference Simes1986).
More subtly, a publication preference for statistically significant results can lead to exaggerated effect size measurements via meta-analysis. Hypothetically speaking, if the published record provided a representative sample of all conducted trials, by the law of large numbers, we would expect variance in effect magnitude to decrease as sample size increases; and, we would expect the mean effect magnitude to remain constant across sample size. However, if there is a publication bias for statistically significant results, then in cases where there is a moderate true effect, the vast majority of smaller studies reaching statistical significance will have extreme magnitudes in the direction of the true effect (Palmer Reference Palmer2000), while very few small studies with results in the opposite direction of the true effect will reach statistical significance (Jennions and Møller Reference Jennions and Møller2002a: 212). Within the publication record, this leads to the increase in mean effect sizes as sample size decreases (Jennions and Møller Reference Jennions and Møller2002b: 47) – something we would not expect to see had studies been sampled in an unbiased manner. Literature reviews demonstrate that small studies published in medical journals do report larger effects than large studies do in the prevention of heart attacks (Egger and Smith Reference Egger and Smith1995) as well as the treatment of obesity (Allison et al. Reference Allison, Faith and Gorman1996), preeclampsia (Pipkin et al. Reference Pipkin, Crowther and de Swiet1996), and cancer (Berlin et al. Reference Berlin, Begg and Louis1989).
Inaccurate meta-analysis measures prevent the community from being able to evaluate the true clinical costs and benefits of new treatments. To address this problem, the ICMJE instituted a requirement, effective in 2005, that trials be registered in a public database with information about the study's intervention(s), comparison groups, primary and secondary outcome measures (including harms), and contact information. That way, researchers conducting meta-analyses could refer to the database to identify relevant trials and contact individuals with access to and responsibility for the data (Mathieu et al. Reference Mathieu, Boutron, Moher, Altman and Ravaud2009: 982).
At first, it appeared that the ICMJE requirement successfully shifted author behavior. After the institution of the requirement, the average number of new trials registered per week at ClinicalTrials.gov jumped from 30 to 220 – a pace that has continued (Zarin et al. Reference Zarin, Ide and Tse2007: 2114). However, a closer look reveals that the quality of the registration information compromises the purpose of the registration requirement. Since the institution of the 2005 ICMJE requirement, the vast majority of registered trials provided at least one primary outcome; however, 61.8% of trials failed to provide specific/meaningful measures for those outcomes (Viergever and Ghersi Reference Viergever and Ghersi2011). Among published studies subject to the 2005 ICJME requirement, fewer than half (45.5%) conformed to it: 27.6% of published studies failed to register altogether, while 12.9% of those registered had no or an unclear description of the primary outcome (Mathieu et al. Reference Mathieu, Boutron, Moher, Altman and Ravaud2009: 982).
Likewise, reporting of mandatory contact information has been inadequate. Although 74–83% of registered trials provided a contact name as required, 60–62% of trials failed to report the requisite email address (Sekeres et al. Reference Sekeres, Gold and Chan2008; Viergever and Ghersi Reference Viergever and Ghersi2011). These findings perpetuate worries that trial data available for and recognizably relevant to answering specific questions continue to skew meta-analysis measures.
Medical journal editor efforts were not the only ones that failed to improve trial registration. The FDA Amendments (FDA) Act of 2007 required that trial results be registered within a year of trial completion. However, a large study of trials subject to mandatory reporting found that 78% had failed to comply (Prayle et al. Reference Prayle, Hurley and Smyth2012). Why do authors flout both legal and publication requirements? Unfortunately, further research on publication bias suggests that more pernicious motives and forms of gamesmanship are afoot.
4. EDITORIAL EFFORTS TO CURB DYSFUNCTIONAL GAMESMANSHIP
The incentive to publish in peer-reviewed journals has increasingly motivated dysfunctional forms of publication gamesmanship in science. Editors complain of practices – such as self-plagiarism, duplicate publication, and the carving up of large studies into the smallest publishable unit – that boost the number of authored publications by diluting or obscuring their intellectual contribution (Kravitz and Feldman Reference Kravitz and Feldman2011). Retractions have also been on the rise, even among elite journals, due to misconduct and questionable practices (Fang and Casadevall Reference Fang and Casadevall2011; Grieneisen and Zhang Reference Grieneisen and Zhang2012). Medicine currently accounts for half the retractions in publications across the physical, social, and engineering sciences (Grieneisen and Zhang Reference Grieneisen and Zhang2012: 6).
In medicine, the publication bias literature has catalogued many types of gaming behavior in which authors meet the letter of the norm of publishing positive outcomes (for individual gain) while violating the norm's function of improving clinical practice (at community-wide cost). Gaming strategies include engaging in covert duplicate publication,Footnote 10 conducting weaker study designs,Footnote 11 undertaking improper dosing strategies,Footnote 12 choosing and gerrymandering endpoints post hoc, and omitting mention of harms.Footnote 13 In what follows, I will focus on the last two “significance questing” (Rothman Reference Rothman1986: 445) strategies because of the availability of systematic literature reviews evaluating the effectiveness of CONSORT and ICMJE interventions designed to stop them.
Post hoc selection of endpoints
In their “chase of statistical significance” (Ioannidis Reference Ioannidis2005b: 696), authors select endpoints post hoc. A large study comparing the protocols of trials approved by ethics committees discovered that 62% of trials changed, introduced, or omitted at least one primary outcome in the published paper in favor of statistically significant ones and that these changes were not submitted as protocol amendments or mentioned in the published study (Chan et al. Reference Chan, Hróbjartsson, Haahr, Gøtzsche and Altman2004). Authors obscure a new treatment's weaknesses by lumping together strong and weak endpoints into a single one (CAPRIE Steering Committee 1996; Born and Collins Reference Born and Collins1997; Garattini et al. Reference Garattini, Bertele and Bassi2003: 1200–1). In light of these strategies, it should not be surprising that a review of 74 studies approved by the FDA found that 33% of results deemed to be negative by the regulatory agency were nonetheless published by authors “in a way that conveyed a positive outcome” (Turner et al. Reference Turner, Matthews, Linardators, Tell and Rosenthal2008: 256).
Can changes to publication standards incentivize improved author behavior? An important feature of the aforementioned 2005 ICMJE requirement was that it demanded that trials be registered before the enrollment of the first patient with information about the study's intervention(s), comparison groups, and primary and secondary outcome measures (including harms). Prospective registration was designed to curb the practice of choosing endpoints post hoc by enhancing transparency and accountability in the planning and reporting of clinical trials (Mathieu et al. Reference Mathieu, Boutron, Moher, Altman and Ravaud2009: 982).
However, subsequent research has demonstrated that among papers published in high-impact medical journals in 2008, more than half (54.5%) had registered improperly by failing to describe clearly a primary outcome or by failing to register prospectively. Among studies that were registered properly, 31.3% had, in the published manuscript, changed the primary outcome post hoc in favor of one that turned out to be statistically significant (Mathieu et al. Reference Mathieu, Boutron, Moher, Altman and Ravaud2009: 982). A study of registered trials (published and unpublished) at around the same time period shows even higher rates of failure when it comes to prospective registration and the adequate description of primary outcomes (Viergever and Ghersi Reference Viergever and Ghersi2011). Despite the ICMJE requirements, authors continue to game publication by leaving themselves the flexibility to frame results in ways that report positive outcomes.Footnote 14
Authors have an incentive not to comply with the ICMJE requirements. Prospective registration of trial protocol would obstruct significance questing practices such as: failing to report all of a study's dependent measures, failing to report all of a study's conditions, and selectively reporting studies that “worked” (John et al. Reference John, Loewenstein and Prelec2012: 527). Significance questing is not only a problem for academic medicine, but a problem plaguing any scientific discipline that prefers to publish statistically significant effects, including psychology (Neuliep and Crandall Reference Neuliep and Crandall1990, Reference Neuliep and Crandall1993) and ecology (Palmer Reference Palmer2000; Jennions and Møller Reference Jennions and Møller2002a: 212). One meta-analysis suggests that 33.7% of scientific researchers admit to using at least one questionable research practice (Fanelli Reference Fanelli2009); a recent survey in psychology suggests that 94% have done so (John et al. Reference John, Loewenstein and Prelec2012: 527). These practices are now “the steroids of scientific competition, artificially enhancing performance and producing a kind of arms race in which researchers who strictly play by the rules are at a competitive disadvantage” (John et al. Reference John, Loewenstein and Prelec2012: 524).
Omission of Harms
The adequate reporting of harms is essential in order for clinicians to compare the benefit versus harm of a treatment. Despite the fact that large trials “amass an enormous amount of information about safety and adverse effects during their conduct” (Ioannidis and Lau Reference Ioannidis and Lau2001: 443), published reports fail to report adverse events and toxicity adequately: one large review of drug trials from 1967–1999 shows that 71% failed to report toxicity adequately while 61% failed to report clinically adverse events adequately (Ioannidis and Lau Reference Ioannidis and Lau2001).Footnote 15 Literature reviews have also demonstrated the inadequate reporting of complications in the surgical literature (Martin II et al. Reference Martin II, Brennan and Jaques2002).
In response to such findings, CONSORT created the 2004 harms extension with requirements on how harms should be reported (Ioannidis et al. Reference Ioannidis, Evans and Gøtzsche2004).Footnote 16 Unfortunately, a study of applicable trials published in top medical journals two years after the harms extension demonstrates that although most reported the existence and rate of adverse events per experimental arm (as required), 78.4% failed to use severity grades for harms and 47.4% failed to provide information about patient withdrawal owing to adverse events (Pitrou et al. Reference Pitrou, Boutron, Ahmad and Ravaud2009: 1759, table 2). These omissions obscure clinically pressing questions about the severity of harms, including the worst cases (Ioannidis Reference Ioannidis2009).
Why was CONSORT 2004 unsuccessful in improving the reporting of harms? The encroaching power of industry sponsors in medicine has provided new incentives and mechanisms for publication bias. A 2007 funding analysis shows that industry accounted for about 58% of biomedical research funding, while NIH accounted for about 27% (Dorsey et al. Reference Dorsey, de Roulet and Thompson2010). Authors have a large incentive to cooperate with sponsor terms, which can include relinquishing full access to data, analyses, and rights to publish (Horton Reference Horton2004: 8). The fact that sponsors have been known to seek legal action against authors seeking to publish negative or harm-related outcomes creates further incentive for author cooperation (Thompson et al. Reference Thompson, Baird and Downie2001). Indeed, the role of academic researchers is sometimes so marginalized that they are placed in first and second author positions only after full manuscripts have been prepared by ghostwriters (Ross et al. Reference Ross, Mulvey, Hines, Nissen and Krumholz2009).
All of these practices enable industry sponsors to hide harm-related results that would block FDA marketing approval or decrease the profitability of their drug or device (Psaty and Kornmal Reference Psaty and Kornmal2008; Elliott Reference Elliott2010). And, hide they do: industry-sponsored studies that make it to print are more likely to report better harms outcomes (Ioannidis and Lau Reference Ioannidis and Lau2001: 441; Lundh et al. Reference Lundh, Sismondo, Busuioc and Bero2012) and efficacy (Lexchin et al. Reference Lexchin, Bero, Djulbegovic and Clark2003; Lundh et al. Reference Lundh, Sismondo, Busuioc and Bero2012) than studies not funded by industry. The suggestion that industry sponsors publish more selectively is not meant to imply that they do not care for the prestige associated with peer-reviewed publication. It is simply that prestige is only instrumental towards financial ends: the imprimatur of prominent journals brings an unparalleled form of visibility and credibility to their drugs and devices (Smith Reference Smith2003: 1203).Footnote 17
5. CONCLUSION
The beauty of the traditional system of incentives in science was that one mechanism – peer review – served normative and motivational functions simultaneously: the slogan was that ambitious scientists could do well professionally by doing well epistemically. Under this regime, we would expect that editors should be able to motivate others to raise their standards of performance (Zuckerman and Merton Reference Zuckerman and Merton1971: 66).
However, this has not been the case in academic medicine. ICMJE 2005 tried to debias trial data available for meta-analysis by requiring trial registration. While this has increased the rate of trial registration, registration information is too often incomplete and inadequately specified (Mathieu et al. Reference Mathieu, Boutron, Moher, Altman and Ravaud2009; Viergever and Ghersi Reference Viergever and Ghersi2011) – practices that undermine the purpose of the registration requirement. This demonstrates that the mere articulation of a stricter publication standard is not sufficient. Editors and reviewers, when evaluating manuscripts, must check that trial registration information is complete and adequate. Furthermore, the FDA and National Library of Medicine (NLM) should verify and enforce the completeness and adequacy of registration information (Ross et al. Reference Ross, Mulvey, Hines, Nissen and Krumholz2009: 7). Proper assessment of the completeness and adequacy of trial protocol might require adding a separate peer review process to the domain of data registration – a practice that has been recommended in other scientific disciplines (Parsons et al. Reference Parsons, Duerr and Minster2010; Lawrence et al. Reference Lawrence, Jones, Matthews, Pepler and Callaghan2011).
ICMJE 2005 required the prospective registration of trial protocol in an effort to increase transparency and deter the post hoc selection and gerrymandering of endpoints. However, more than half of published studies flouted the purpose of the requirement by failing to register prospectively or failing to articulate a specific primary outcome (Mathieu et al. Reference Mathieu, Boutron, Moher, Altman and Ravaud2009; Viergever and Ghersi Reference Viergever and Ghersi2011). This result underscores the suggestion that editors and reviewers should compare protocols identified in submitted manuscripts to the original registration record. Furthermore, the FDA, NLM, and reviewers for registration databases (if needed) should verify and enforce the completeness and adequacy of registration data.
Finally, CONSORT 2004 was designed to improve the reporting of harms and toxicity in published reports. However, subsequent literature reviews demonstrate the continued inadequacy of harms reporting (Pitrou et al. Reference Pitrou, Boutron, Ahmad and Ravaud2009). Industry sponsorship has provided a new system of incentives and mechanisms for publication bias; and editorial interventions thus far have failed to countenance them. Editors must state much stricter publication requirements that make compulsory author control over trial design, data, analyses, and interpretation as well as rights to publish. Federal agencies should create and enforce regulations designed to grant such control to authors and protect them from litigation by industry sponsors. Federal protections are also needed for medical journals and editors who publish unfavorable trial results (protections are also needed for reviewers who recommend their publication). We may even need to uproot the financial pull of industry more directly by getting rid of medical patents (Brown Reference Brown2004, Reference Brown2008; Reiss and Kitcher Reference Reiss and Kitcher2009; Reiss Reference Reiss2010) and providing full public funding for medical research (Brown Reference Brown2004, Reference Brown2008). To combat publication bias in medicine, we must go well beyond the prestige of its gate-keepers, the content of its evaluative standards, and the expertise of its members by appealing to increased federal legislation, enforcement, and funding.Footnote 18
