What is the problem? Defining a problematic study

There are many reasons that a clinical study can be considered untrustworthy. Cochrane, a major international producer of systematic reviews and standards for systematic review methods, defines problematic studies as “any published or unpublished study where there are serious questions about the trustworthiness of the data or findings, regardless of whether the study has been formally retracted.”Reference Boughton ³ I will adhere to this broad definition of a problematic study throughout this paper.

Importantly, the Cochrane definition does not consider intent. Systematic reviewers need to go beyond the legal definition for fraud as this varies by jurisdiction, and systematic reviews include studies conducted globally and that may be published in any country. Fraud, deliberate fabrication of data and scientific misconduct are not the only behaviors that can undermine confidence and trust in studies included in systematic reviews. Questions about the untrustworthiness of data could arise from poor research practices or honest errors. There is an urgent need for systematic reviewers to identify problematic studies and exclude them from reviews, without regard for the reason that a problematic study exists. Some criteria for defining problematic studies raise ethical concerns by, for example, suggesting that research from particular countries or research groups is at risk of being fraudulent. The Cochrane definition leads to identifying characteristics of studies that indicate untrustworthiness, not characteristics of individual researchers.

There are many types of problematic studies. Parker and colleagues conducted a qualitative interview study to explore indicators of problematic research, including fraud and fabrication, that could be included in a screening tool.Reference Parker ⁴ The 30 participants from 12 countries described methods for detecting different manifestations of problematic studies. The study described a spectrum of problems that can lead to untrustworthy results, including: falsified data, image manipulation, selective deletion of data, fabricated analysis using real data, inappropriate data analysis, nonadherence to ethical standards and research regulations, and plagiarism. All of these types of problematic studies could potentially be included in systematic reviews, thus affecting clinical decision-making and risking harm to patients.

How big is the problem? Prevalence of problematic studies

Increasing numbers of reports of falsified or fabricated clinical studies have raised alarms about the trustworthiness of data that are included in systematic reviews of clinical trials. However, it is impossible to establish the overall prevalence of problematic clinical trials because studies attempting to estimate prevalence have focused on specific clinical areas, used different sampling methods to select trials for analysis, and have used different methods to identify problematic studies.

A study focusing on identification of problematic studies in a specific clinical area found an alarming number of untrustworthy trials. The study examined 526 patient trials in anesthesiology that were submitted to a single journal. The trials were analyzed for anomalies or improbable similarities in the summary baseline data. The study found that 8% had fake data and 26% had critically flawed data.Reference Carlisle ⁵

Some research integrity investigators approach their analyses by focusing on all studies conducted by an author who has published retracted studies or studies with documented research integrity problems. For example, an investigation of 10 trials on treatment of persistent or subacute spinal pain published by a single research group found multiple research integrity problems with each trial.Reference O’Connell ⁶ These problems included authors not providing requested data or ethics approvals, randomization concerns, similarities in baseline data, and implausible results. When the effect sizes of these 10 trials were compared to the effect sizes of comparable trials published by other research groups, the included trials had larger effect sizes for pain, disability and quality of life. This means that the trials with research integrity problems exaggerated effectiveness of the interventions being tested compared to trials with no detected research integrity problems.

Weeks and colleagues applied a research integrity screening tool to studies included in 18 Cochrane reviews focusing on antenatal and postnatal nutritional interventions.Reference Weeks ⁷ Ninety out of 374 studies included in these reviews (24%) were classified as having at least one research integrity problem. The research integrity issues identified were mainly related to study governance and feasibility. No study had problems identified with the results. Excluding studies with research integrity problems from the systematic reviews had some impact on the review results. First, the number of included trials was reduced in 14 out of 18 reviews. Second, six reviews (33%) were judged to have important differences in the direction and size of effects and/or conclusions.

Although the available prevalence data on problematic clinical trials is highly variable, the potential impact on systematic reviews cannot be denied. The existence of problematic clinical trials is a threat to the trustworthiness of systematic reviews. The identification of studies with serious research integrity issues is important to patients and their caregivers because systematic reviews inform decisions about health care interventions. Inclusion of seriously flawed, fake, or untrustworthy studies in systematic reviews could influence the review outcomes, even leading to subsequent recommendations of ineffective or harmful treatment. For reviews where data are synthesized quantitatively (e.g., using meta-analysis), inclusion of problematic studies could change the magnitude, certainty, or direction of results.

There are many examples of how the conclusions of systematic reviews change when problematic studies are removed, such as reviews of Vitamin K for bone fracture and tranexamic acid for preventing postpartum hemorrhage.Reference Torgerson ⁸ The policy and clinical consequences of including problematic studies in systematic reviews were highlighted by reviews of ivermectin as a treatment for COVID-19. Multiple reviews of randomized controlled trials testing ivermectin as a treatment for COVID that included problematic studies (later retracted) claimed a large beneficial effect of ivermectin, in contrast to a Cochrane review that found no benefit of ivermectin.Reference Bryant ⁹ The authors of the Cochrane ivermectin review developed a research integrity assessment tool that assessed study retraction, prospective trial registration, adequate ethics approval, author group, plausibility of methods (e.g., randomization), and plausibility of study results.Reference Weibel ¹⁰ The tool was applied during screening of potentially eligible studies. When ivermectin trials with research integrity problems were excluded from the Cochrane review, the 11 remaining trials showed no beneficial effect of ivermectin.

Problematic studies likely exist in other scholarly areas where systematic reviews are common, but data on the prevalence of such studies outside of clinical trials is even more rare. Paper mills and image falsification have been well described.Reference Byrne and Christopher ¹¹ Problematic studies in other areas are also likely to occur as many problems stem from researcher behaviors that include making up data or results, fabricating and/or falsifying data, and other “questionable research practices.”Reference Gopalakrishna ¹²

Whose problem is it? The role of systematic reviewers

Preventing and eliminating problematic studies is a complicated task.Reference Bero ¹³ Changing incentives for publication among academics, publishers, and funders is needed to stamp out problematic studies before they are published. ¹⁴ When published problematic studies are detected, it can take years before a study is retracted, and, even then, retracted studies continue to be cited.Reference Wager and Williams ¹⁵ Thus, the scientific literature is already populated with problematic studies. Thus, systematic reviewers can play a critical role in detecting such studies and decreasing their use.

First, systematic reviewers look across an entire body of evidence related to a particular question. This allows them to detect patterns across studies that might be indicative of problematic studies, such as repeated passages of text, duplicative numerical data or repetitive errors or “tortured phrases.”Reference Cabanac ¹⁶ By comparing studies on the same topic, systematic reviewers can also detect outliers that have improbably favorable results or excessively uniform samples. Systematic reviewers can play a similar role to publishers who have looked across papers submitted to all their journals in order to detect submissions from paper mills.Reference Byrne ¹⁷ Individual researchers are not able to detect such patterns when they are scrutinizing a single study.

Second, assessment of risk of bias or some form of study quality evaluation is a key step that is built into the systematic review process. Commonly used risk of bias tools are designed to detect study design features, or internal validity issues, that are likely to bias the outcome of the studies.Reference Higgins, Thomas and Chandler ¹⁸ Some items in risk of bias tools detect features that suggest a study may be problematic. For example, risk of bias tools may detect that a clinical trial was not pre-registered as part of detecting reporting bias — the failure to report the results of all outcomes measured in the study (with a tendency to only report the outcomes that are favorable to the sponsor, for example). Or a risk of bias assessment may find that the authors will not make the data available. While these study design issues may raise suspicion about the integrity of the data, risk of bias tools are not designed to detect false or fabricated data or data manipulation. As described below, further checks would be needed to detect problematic data. In addition, methods to detect research integrity problems should be applied before the risk of bias assessment so that problematic studies are excluded before the risk of bias assessment step. Many journals would be unable to apply research integrity screening tools due to resource and expertise limitations, so an additional screening tool is unlikely to be widely adopted at the journal level. In contrast, systematic review teams likely have the expertise to apply a research integrity screening tool, although assessing problematic studies would add additional time to conducting the systematic review.

Third, excluding problematic studies from systematic reviews has the potential to improve the scientific literature. The exclusion of a problematic study from a systematic review might reduce its future citations because studies that are rated as high risk of bias in systematic reviews subsequently receive less attention in the literature. By comparing the citations of studies that were rated high risk of bias to those that were rated low risk of bias in Cochrane review, Salandra found that the citations of high risk of bias studies decreased by 12% over four years compared to the low risk of bias publications.Reference Salandra ¹⁹ She suggests that Cochrane reviews send a “quality signal” to scientists about which studies they should use and cite. Thus, if systematic reviewers begin to identify and exclude problematic studies, this could also serve as a signal to decrease use of problematic studies by other researchers. Ideally, problematic studies should be retracted from the scientific literature. When systematic reviewers identify problematic studies, they can initiate the process to request a retraction. ²⁰

How can systematic reviewers identify problematic studies?

Many different methods exist for identifying problematic studies, and these methods vary depending on the type of study and nature of the problem. There is a lack of consensus about an effective and time-efficient standard method for declaring studies untrustworthy. Interestingly, some researchers who routinely identify research integrity problems are opposed to publicly sharing their methods due to concerns that those interested in producing fraudulent data could devise ways to escape detection. ²¹ A scoping review identified 57 papers published up until 2020 that reported 27 methods for identifying “misconduct” in health research.Reference Bordewijk ²² The scoping review used the US Office of Research Integrity definition of research misconduct that includes fabrication, falsification, and plagiarism. The 27 published methods focused on concerns about text (e.g., duplicate text), images (e.g., reuse of images, image manipulation) and data (baseline p value distribution, statistical checks). Most of the tools were not developed in a transparent way, adequately tested or structurally implemented by systematic review organizations. ²³

Systematic, validated methods for detecting problematic studies are needed because use of an unvalidated method risks doing more harm than good. There are consequences of incorrectly labeling genuine study results as problematic. Excluding real data from a review could produce the wrong conclusion. Additionally, study participants have contributed their time to research, so there is an ethical requirement to use their data for patient benefit. Corporate interests sometimes use (false) accusations of fraud to discredit legitimate research that puts them in a bad light, as do many ideological factions. Accusations of fraud can put both the accused and whistle-blowers at risk of losing their jobs and damaging their reputations.

Cochrane is currently collaborating on a research project to develop a tool — INveStigating ProblEmatic Clinical Trials in Systematic Reviews (INSPECT-SR) — to identify problematic randomized controlled trials in systematic reviews.Reference Wilkinson ²⁴ The aim is that, following development and testing of INSPECT-SR, it will be recommended as the method for Cochrane reviewers to use in identifying problematic studies. It is important to use a standardized tool so that all eligible studies are scrutinized in the same way, just as all randomized controlled trials are assessed for risk of bias using the same Cochrane risk of bias tool.

As a global organization, Cochrane is sensitive to inequalities in research resources and infrastructure around the world. Some existing suggestions for identifying problematic studies introduce inequities when applied to studies by, for example, focusing on an investigator’s country of origin or where the paper was published rather than the study itself. Identification of a problematic study can have serious consequences including notification of the institution where the author works, or the editor of the journal where the study was published. These notifications could be followed by investigations of research fraud or misconduct, retraction of papers, stopping of academic advancement or dismissal. Thus, the method used to identify the problematic study should be uniformly applied across all studies eligible for a review.

The INSPECT-SR project, funded by the National Institute for Health and Care Research, will develop a tool to assess the trustworthiness of randomized controlled trials (RCTs) in systematic reviews of healthcare-related interventions. The project is informed by a scoping review of existing tools and a qualitative interview study of researchers involved in research integrity investigations.Reference Parker ²⁵ The INSPECT-SR project has five stages: (1) a survey of experts to assemble a comprehensive list of checks for detecting problematic RCTs, (2) an evaluation of the feasibility and impact of applying the checks to systematic reviews, (3) a Delphi survey to determine which of the checks are supported by expert consensus, culminating in (4) a consensus meeting to select checks to be included in a draft tool and to determine its format and (5) prospective testing of the draft tool in the production of new health systematic reviews, to allow refinement based on user feedback. Details have been published in a protocol. ²⁶ An international survey of 71 individuals with experience assessing potentially problematic studies presented participants with an initial list of 102 possible checks for problematic RCTs grouped into five domains: Inspecting results in the paper; Inspecting the research team; Inspecting conduct, governance, and transparency; Inspecting text and publication details; and Inspecting the individual participant data. Based on extensive feedback, a draft list of 116 checks was formulated.Reference Wilkinson ²⁷

The second step in developing the INSPECT-SR tool was to apply the list of 116 checks to RCTs included in systematic reviews in order to assess the feasibility and usefulness of each check for identifying problematic studies. A total of 95 trials from 50 reviews were evaluated. When the checks flagged studies that were considered problematic, the results of the reviews changed when these studies were deleted. For example, when trials that were flagged as possibly failing or failing a check were removed from the meta-analysis, the number of trials included in each review decreased, with 33% of continuous outcome meta-analysis and 12% of binary outcome analyses having no remaining trials. For meta-analyses where some trials remained, the effect estimates and standard errors changed. Some checks were not considered feasible or failed for almost all the trials assessed, thus making them not useful. These checks were not carried forward to the next stages of INSPECT-SR development. ²⁸

Ongoing work will further develop the INSPECT-SR method as outlined in steps three through five of the INSPECT-SR project above.

What should a systematic reviewer do when a problematic study is identified?

The Cochrane Policy for Managing Potentially Problematic Studies and its accompanying implementation guide provide detail on what Cochrane Review authors and editors should do when serious concerns are raised about the trustworthiness of studies eligible for inclusion in Cochrane reviews. ²⁹ These were launched in 2021, following years of internal and external consultation. During the consultation, there was very little disagreement on how Cochrane reviewers should handle a problematic study once it is identified. The bottom line is that the study should not be included in the Cochrane review, either descriptively as an included study, or in any quantitative or qualitative analysis.

The primary debates were around how to define problematic studies and the methods used to identify them. To preserve the integrity of their reviews, systematic reviewers only want to include studies they can trust. Thus, Cochrane’s definition of a problematic study is broad enough to encompass all types of problems that could result in untrustworthy data. A second area of contention was how to identify problematic studies. Given the lack of consensus around a method for identifying problematic studies, Cochrane launched research into developing a method, as described for the INSPECT-SR project above. The Implementation Guide also recommends several ways to learn if problematic studies have been identified by other investigators. For example, reviewers are advised to check for any post-publication amendments to a study and to confirm that the study has not been retracted.Reference Bakker ³⁰

Systematic reviewers have the obligation not only to exclude problematic studies from their reviews, but also to take action to purge these studies from the scientific literature. It is not within a systematic reviewer’s authority to conduct a formal investigation, to draw formal conclusions regarding the misconduct of an individual or group of individuals, or to determine whether a published article should be retracted. That role belongs to the employer, university, granting agency, regulatory body, or journal editor concerned. Systematic reviewers do have a responsibility to protect the integrity of the public scientific record by sharing reasonable concerns with authorities that can conduct such an investigation. Cochrane, for example, provides templates for contacting authors and editors with concerns about a problematic study.

The implementation guide recommends that if a systematic reviewer suspects that a study is problematic, they should first contact the authors of the study for clarification or additional information or data. ³¹ If the authors cannot be contacted or their response does not resolve the concern, Cochrane reviewers are asked to contact the editor(s) of the journal(s) where the study was published. As parts of studies can be legitimately published in multiple journals, or illegitimately duplicated between journals, this could require contacting multiple journals. It is the responsibility of the journal editor to initiate further investigation when a paper they have published is questioned. Cochrane’s recommendations for contacting journal editors have received pushback from some Cochrane reviewers who have found journal editors to be unresponsive. They consider contacting journals “a waste of time.”

Future Directions

As I have written previously, everyone involved in the scientific publication pipeline — researchers, universities, journal editors, publishers and funders — shares responsibility for publication of problematic studies and needs to coordinate efforts to prevent their publication. ³² While systematic reviewers can help draw attention to the problem and may reduce citation of problematic studies, they cannot prevent publication of a study or retract it from the literature. Systematic reviewers do have an obligation to share their methods for identifying problematic research and to help other organizations in the publication pipeline implement these methods.

One obstacle to preventing publication of problematic studies has been that organizations involved in research integrity investigations often do not share information.Reference Gunsalus and Piece ³³ This creates duplication of effort and means that an investigation can end without all the necessary information about a study being available for analysis. Systematic reviewers can minimize this problem by sharing the research integrity investigations they conduct. For example, Cochrane envisions a publicly available, structured, searchable archive of all research integrity investigations conducted using the INSPECT-SR method. This will be a valuable contribution to investigations of the same studies being conducted by journals, universities or publishers.

The development of INSPECT-SR will provide an evidence-based, standardized method for identifying problematic studies that can be applied equitably across all trials relevant for Cochrane reviews. However, implementation challenges remain. Although development of the method prioritized assessment items that are feasible to apply, it will still take time. Systematic review is already a laborious and time-consuming process, so widespread adoption of a research integrity assessment tool will add to the burden. One solution is to automate and use artificial intelligence to operationalize as many items in INSPECT-SR as possible. Several automated methods to interrogate numerical data anomalies, statistical tests, and duplicate text exist, for example. Items related to plausibility of the study are specific to the topic, such as an abnormally high proportion of females in a study of a disease affecting primarily males. These items may be more challenging to automate. Some automated tools are currently not freely available or have restricted availability.