Critical appraisal within sources of evidence

In comparison to the Cochrane review, which included only prospective empirical studies, this review includes a much broader range of evidence from retrospective empirical studies, other reviews, and opinion papers. Though we did not explicitly assign a strength of evidence or risk of bias rating, we fully acknowledge that the retrospective, descriptive, and qualitative papers provide inherently weaker evidence due to threats to internal validity and risk of bias. For example, with one exception, there were no measures of SDV quality reported prior to broadened interest in RBM. However, the two-dimensional framework for evaluating SDV quality (Table 3) would not have emerged without considering the outcomes measured in the retrospective studies. As a scoping review, these sources were included to provide a comprehensive compilation of the available literature and findings across the spectrum of research designs while at the same time recognizing the difference in evidence strength.

The recent systematic literature reviews for Good Clinical Data Management Practices also found scant Cochrane-strength evidence for practice recommendations [Reference Hills, Bartlett, Leconte and Zozus65–Reference Pestronk, Johnson and Muthanna69]. In these reviews, randomized, controlled experiments of operational processes were few and far between. It may be that this standard is not a feasible expectation for evidence-supporting practice in clinical research operations. On the other hand, it still seems ironic that the methods from which strong evidence is generated are not themselves held to that same standard. Without such a standard, some may interpret the literature regarding SDV accuracy as sufficient to recommend against SDV, while others looking at the same literature will refrain from such recommendations.

Results of individual sources of evidence and synthesis of results

The included quantitative studies exhibited significant heterogeneity along multiple dimensions. We identified four reported aspects of SDV quality (rows in Table 4): (1) source data availability and access, (2) data quality, (3) study process fidelity, and (4) SDV process fidelity. Furthermore, reports for the SDV quality domains naturally fell into three main categories (columns in Table 4): (1) SDV quality measures, such as rates of findings, missing data, or process problems, (2) the impact of SDV on decreasing future problems, and (3) the impact of SDV on study results. One of the included articles assessed SDV quality on all three levels [Reference Giganti, Shepherd and Caro-Vega53]. Three studies reported a measure of source data availability and access [Reference Mealer, Kittelson and Thompson17,Reference Embleton-Thirsk, Deane and Townsend51,Reference Yamada, Chiu and Takata56] (Table 4). The data quality aspect (domain) of SDV quality (second row in Table 4) was reported as data discrepancies and errors. Six articles reported counts or rates of data discrepancies [Reference Mealer, Kittelson and Thompson17,18,Reference Andersen, Byrjalsen and Bihlet31,Reference Maruszewski, Lacour-Gayet, Monro, Keogh, Tobota and Kansy47,Reference Tudur Smith, Stocken and Dunn48,Reference Fougerou-Leurent, Laviolle and Tual52], while seven reported measures of errors [Reference Mitchel, Cho and Gittleman33,Reference Maruszewski, Lacour-Gayet, Monro, Keogh, Tobota and Kansy47,Reference Sheetz, Wilson and Benedict49,Reference Embleton-Thirsk, Deane and Townsend51,Reference Giganti, Shepherd and Caro-Vega53,Reference Kondo, Kamiyoshihara and Fujisawa55,Reference Yamada, Chiu and Takata56]. An error is a discrepancy with the truth, i.e., an incorrect data value. A discrepancy, on the other hand, is a difference between two data values where either or both could be in error. Discrepancies are often reported when the correct value cannot be determined or was not determined. In the included articles, some reported database changes were made after investigating and resolving discrepancies; we counted these as reports of errors.

Table 4.

Aspects of source data verificaiton (SDV) quality and the level at which they were reported

* These assessments were done relative to another method rather than as a quantification of absolute errors. i.e., the studies have not measured SDV actual accuracy, instead, surrogates for accuracy were measured, such as a number of data discrepancies or errors missed relative to some other method.

^† Includes delivery of the intervention, endpoint Assessment, regulations, guidance, and other requirements to which the protocol must comply.

The third and most frequently reported aspect of SDV quality was the ability to detect or propensity to miss study process errors; we refer to this as study process fidelity (Table 4). The included articles varied greatly concerning the study processes for which SDV quality was reported. Eight articles reported SDV quality with respect to the detection of safety issues [Reference Brosteanu, Schwarz and Houben37,Reference Kim, Kim and Hong38,Reference Maruszewski, Lacour-Gayet, Monro, Keogh, Tobota and Kansy47–Reference Sheetz, Wilson and Benedict49,Reference Embleton-Thirsk, Deane and Townsend51,Reference Wyman Engen, Huppler Hullsiek and Belloso54,Reference Kondo, Kamiyoshihara and Fujisawa55]. One reported that SDV improved safety process fidelity over time [Reference Brosteanu, Schwarz and Houben37]. And four reported the impact of SDV on safety-related study results [Reference Maruszewski, Lacour-Gayet, Monro, Keogh, Tobota and Kansy47,Reference Tudur Smith, Stocken and Dunn48,Reference Embleton-Thirsk, Deane and Townsend51,Reference Wyman Engen, Huppler Hullsiek and Belloso54]. Eligibility, informed consent, and protocol adherence were tied as the second most frequently reported aspects of study process fidelity (Table 4). These aspects encompass the EMA (2017) integrated inspection report categories used to summarize and evaluate the potential implications of major or critical findings, “the impact on the integrity of the trial data, the rights, wellbeing, and safety of the subjects, the compliance of the trial with GCP (including ethical principles) …” [70]. Within each SDV quality aspect, the included articles varied in whether and, if so, how the impact of SDV was evaluated. All sixteen included quantitative articles reported counts or rates of items detected or missed by SDV (Table 4, Fig. 3). Eleven [Reference Mealer, Kittelson and Thompson17,18,Reference Andersen, Byrjalsen and Bihlet31,Reference Mitchel, Cho and Gittleman33,Reference Maruszewski, Lacour-Gayet, Monro, Keogh, Tobota and Kansy47–Reference Sheetz, Wilson and Benedict49,Reference Fougerou-Leurent, Laviolle and Tual52,Reference Giganti, Shepherd and Caro-Vega53,Reference Kondo, Kamiyoshihara and Fujisawa55,Reference Yamada, Chiu and Takata56] of the sixteen quantitative articles reported measures of data discrepancies or errors. Two articles [Reference Brosteanu, Schwarz and Houben37,Reference Giganti, Shepherd and Caro-Vega53] evaluated the impact of SDV on quality improvement over time within a study, whereas nine [Reference Andersen, Byrjalsen and Bihlet31,Reference Mitchel, Cho and Gittleman33,Reference Maruszewski, Lacour-Gayet, Monro, Keogh, Tobota and Kansy47,Reference Tudur Smith, Stocken and Dunn48,Reference Embleton-Thirsk, Deane and Townsend51–Reference Wyman Engen, Huppler Hullsiek and Belloso54,Reference Yamada, Chiu and Takata56] evaluated the impact of SDV findings on study results (Table 4, Fig. 3). However, five [Reference Andersen, Byrjalsen and Bihlet31,Reference Mitchel, Cho and Gittleman33,Reference Fougerou-Leurent, Laviolle and Tual52,Reference Wyman Engen, Huppler Hullsiek and Belloso54,Reference Yamada, Chiu and Takata56] of the articles reporting the impact of SDV findings on study results did so qualitatively, for example, stating that the errors occurred evenly across treatment groups, that the errors occurred in non-critical variables, or that the frequency or extent of the errors was too small to have impacted the analysis. The remaining four [Reference Maruszewski, Lacour-Gayet, Monro, Keogh, Tobota and Kansy47,Reference Tudur Smith, Stocken and Dunn48,Reference Embleton-Thirsk, Deane and Townsend51,Reference Giganti, Shepherd and Caro-Vega53] articles reporting the impact of SDV findings on study results did so by comparing analyses before and after error correction.

Figure 3.

Heterogeneity in SDV* quality assessment. Each 3x4 grid in the figure represents SDV quality assessments reported in one included, quantitative article. The SDV methods compared are listed at the bottom of each grid, with NR signifying not reported, extensive signifying high amounts of data values undergone up to 100% SDV, and mixed signifying a combination of two or more SDV methods. *Source Data Verification (SDV): the comparison of study data to their original recording to ensure that, “the reported trial data are accurate, complete, and verifiable from source documents [16].”

In addition to variability in the aspects of the SDV quality measured and the level at which they were reported (Table 4), the included articles exhibited just as much variability in the context in which they were measured. For example, seven assessments were done in industry clinical trials versus nine in investigator-initiated studies (Fig. 3). The included articles were heterogeneous with respect to the SDV process for which the quality was measured. The most common SDV process variants assessed included remote, targeted, and varied extents of reduced SDV (Fig. 3, Supplementary Material 3). All reported SDV quality assessments were relative, for example, comparing the number of SDV findings missed by one method that was subsequently detected by another, or reporting the portion of SDV findings that resulted in database changes. The reported assessments differed in how discrepancies or errors were identified, i.e., which two SDV processes were compared to identify items missed by one of the methods but detected by the other. The assessments differed in whether these discrepancies were verified. Multiple included studies reported the number of data discrepancies detected by SDV, while others reported those missed by SDV. One study [Reference Brosteanu, Schwarz and Houben37] comprehensively reported the number of discrepancies detected and missed by SDV but did so for only one parameter, informed consent (Supplementary Material 3). Further limiting the quantitative synthesis, the unit of analysis was inconsistent across included studies and included counts of individual findings, rates of findings per patient, rates of findings per site, or proportions of patients or sites with one or more findings. Additionally, some studies reported only major findings, while others reported all findings. These differences are detailed in Supplementary Material 3. Although multiple studies quantified one or more aspects of SDV accuracy, no article reported a comprehensive measure of SDV accuracy. No study reported the absolute rate of errors, i.e., detected against the gold standard of truth. One study [Reference Fougerou-Leurent, Laviolle and Tual52] declared 100% SDV to be the gold standard but did not compute accuracy measures (sensitivity and specificity) against it.

The search strategy broadly encompassed articles focused on SDV as well as those focused on clinical study monitoring; relevant reports of SDV quality were found in both types of articles. For example, a study comparing 100% SDV versus reduced SDV that quantified items missed by 100% SDV provided an assessment of SDV quality. Similarly, a study comparing triggered versus non-triggered on-site visits that quantified items missed by SDV provided an assessment of SDV quality.

The majority of included articles reported SDV quality in the context of comparing RBM (including targeted, remote, or reduced SDV) to traditional monitoring approaches usually characterized by more extensive SDV. The heterogeneity, such as differences in the amount, timing, or frequency of SDV, in articles included in the quantitative synthesis (Table 4 and Supplementary Material 3) means that the assessments of SDV quality are not comparable; the methodological heterogeneity limited this synthesis to a scoping review.

The evaluative studies identified and included in this review are varied in terms of which domains of SDV quality are reported for SDV (rows in Table 4). Reported SDV quality domains included accessibility of source data, data error rates, GCP or protocol deviations (often called monitoring findings), and audit-identified deviations in the SDV process. Further, these SDV quality domains were reported at multiple levels, including accessibility of source data needed to identify errors, rates of identified errors, effectiveness at preventing future errors, and impact of the identified errors on study outcomes (columns in Table 4). SDV likely has utility on each level, however, reports at the prevention and study results levels were few compared to counts or rates of findings. A comprehensive evaluation of SDV quality would include the rows and columns in Table 4 as well as items both identified and missed by SDV.

SDV accuracy cannot be calculated without a gold standard and enumeration of true positives, false positives, true negatives, and false negatives. One study [Reference Fougerou-Leurent, Laviolle and Tual52] declared 100% on-site SDV as the gold standard; however, the authors acknowledge that errors remain in 100% SDV’d data and did not report accuracy measures of sensitivity and specificity. Thus, we did not find a quantitative report of absolute SDV accuracy (sensitivity and specificity).

Related findings

There is a good number of reviews and surveys. Despite the fact that they didn’t meet the inclusion criteria, they reported related beneficial results, and we opted to present them. Seven relevant reviews [Reference Olsen, Bihlet, Kalakou and Andersen11,Reference Houston, Probst and Martin26,Reference Klatte, Pauli-Magnus and Love27,Reference Hurley, Shiely and Power71–Reference Macefield, Beswick, Blazeby and Lane73] were identified by the search (Fig. 4). Three of the identified reviews met the inclusion criteria (bottom row in Fig. 4). The reviews differed in their scope, with the three included reviews specifically addressing SDV and the four excluded reviews focusing more broadly on clinical trial monitoring or other aspects thereof (Fig. 4). Four of the reviews focused on methods; two included only empirical assessment of monitoring or SDV outcomes, and one review, Olsen et al. (2016), included both methods and empirical results (Fig. 4). While the Ward (2013) survey did not meet inclusion criteria, the systematic review portion of this work did and is included (Fig. 4). Only the three reviews, including Ward (2013), specifically addressed SDV were included in this review.

Figure 4.

Categorization of review articles. ¹Source Data Verification: the comparison of study data to their original recording to ensure that, “the reported trial data are accurate, complete, and verifiable from source documents [16].”

The Cochrane review by Klatte et al. (2021) included eight prospective empirical studies that compared different monitoring strategies [Reference Klatte, Pauli-Magnus and Love27]. Five of them are also included in our review. Overall, the Cochrane review concluded with moderate certainty that “risk-based monitoring is not inferior to extensive on-site monitoring with respect to critical and major monitoring findings in clinical trials” and noted that “more high-quality monitoring studies that measure effects on all outcomes specified in this review are necessary to draw more reliable conclusions.” The Cochrane review did not directly address the accuracy of the SDV process itself with respect to the identification of errant data.

Eight surveys [Reference Funning, Grahnén, Eriksson and Kettis-Linblad13,Reference Ward28,Reference Odette Jochems and Mountain74–79] were identified through the search, all but one [Reference Ward28] were excluded due to minimal focus on SDV quality. However, because each survey contained one or more SDV-relevant questions, we have listed them in Supplementary Material 4 for completeness. Across the surveys, perceptions varied widely with respect to the impact of SDV or monitoring on data quality. Collectively, the survey work indicates variability in SDV frequency, amount, and methods similar to that seen in the quantitative, included articles and in the included reviews. Multiple articles reporting survey results called for additional research on SDV methods, the impact of SDV, and more specific guidelines for methods, including the amount of SDV. However, their message is weakened by the limited generalizability of each survey and the significant changes in context and practice over the almost 30-year span over which survey results were reported.

Evidence throughout the literature, from opinion to experimental studies, supports the ability of SDV to identify unreported events. While reports of SDV-identified significant or systematic findings certainly exist [Reference Maruszewski, Lacour-Gayet, Monro, Keogh, Tobota and Kansy47,Reference Tudur Smith, Stocken and Dunn48,Reference Embleton-Thirsk, Deane and Townsend51,Reference Fougerou-Leurent, Laviolle and Tual52], the included articles reporting the impact of missed-event-type findings on study results indicated no significant impact [Reference Klatte, Pauli-Magnus and Love27,Reference Brosteanu, Schwarz and Houben37,Reference Embleton-Thirsk, Deane and Townsend51]. Similarly, multiple included studies concluded a lack of impact of SDV-identified data errors on study results[Reference Andersen, Byrjalsen and Bihlet31,Reference Tudur Smith, Stocken and Dunn48,Reference Embleton-Thirsk, Deane and Townsend51]. However, others found SDV-identified data discrepancies or errors impactful on one or more study analyses [Reference Mealer, Kittelson and Thompson17,Reference Giganti, Shepherd and Caro-Vega53]. These apparent differences may reflect differences in the SDV quality domains assessed, differences in measurement methods, differences in the SDV processes themselves, or differences in other aspects of data collection and processing. For example, one of the included studies assessed the type, frequency, and impact of data errors on an observational study and concluded that data errors identified through SDV would have otherwise impacted the study results [Reference Giganti, Shepherd and Caro-Vega53]. However, few upstream data quality control measures were in place. Similarly, two articles concluding no impact of SDV-identified data error measured SDV error for query-clean data and data collected on structured site worksheets (rather than abstracted from medical records) – the upstream data quality control measures described would have likely significantly decreased the number of errors remaining to be detected by SDV. The many-faceted heterogeneity precludes drawing conclusions.

With rising cost pressure, a wide variety of options continue to be explored. The Cochrane review concludes, albeit on what the authors deem to be moderate to low-quality evidence, that there is likely no difference between the different monitoring (and SDV) strategies tested in the five comparisons assessed by the review. We posit that this could be due to the likely high error rate of SDV itself, i.e., less of an error-prone inspection may not yield markedly worse overall quality. One of the most convincing studies comparing the outgoing error rate from RBM to traditional monitoring concluded the same, noninferiority [Reference Andersen, von Sehested, Byrjalsen, Popik, B. and Bihlet80].

Remote access to Electronic Health Records (EHRs) for monitoring and the ability to extract data directly from them greatly impact study monitoring and SDV processes. The justification for pursuing such EHR-to-eCRF data collection includes increasing data quality by decreasing manual medical record abstraction and decreasing the data collection burden at sites and Sponsors. In this case, SDV can be performed by computationally confirming that study data match the EHR source or may obviated altogether as mentioned in the FDA eSource guidance [81]. Eliminating manual SDV for EHR-to-eCRF data would likely reduce burden and cost. Given advances in technology and permissible regulatory guidance, available innovation will likely be applied to SDV, such as (1) establishing traceability back to the source that can be computationally traversed to demonstrate that the final data exactly reflects the medical record source and (2) defining a certified copy in the context of data extracted from EHRs such that sponsors and regulators are guaranteed that the final data are a replica of the source. These cannot be accomplished with manual SDV.

Limitations

Though performed systematically, our search could have missed an important article. The database searches were conducted over a 5-month period which may result in differences (though minimal) of returned articles. The methodological variation observed in the included papers was huge, making it difficult to draw conclusions from the body of the literature. For example, no two reported instances of reduced SDV were exactly alike. Similarly, the data processing and quality control applied to data prior to SDV were often not completely described. Significant differences were observed in available descriptions of upstream data processing, such as performing SDV against the medical record versus against structured site source worksheets. The contributions of SDV versus SDR (defined in Table 1) often could not be distinguished from one another. SDV includes reading the source record to ensure that there were no omissions in the study data which includes some amount of SDR. In fact, multiple included articles remarked that SDV was conducted by reading the full source. For this reason, findings attributed to SDV may have also been found or may be equally findable through SDR. Additionally, the variability in the terminology used in the literature could have easily led to inadvertent misclassification of studies with respect to the SDV process measured, the measures used, and the methods by which they were obtained.