Introduction
Source data verification (SDV) is the manual comparison of transcribed study data to their original recording to ensure that reported data are accurate, complete, and verifiable from source documents [1]. As the predominant, historical data quality control activity, the SDV process itself was largely unquestioned. For many years, 100% SDV was widely regarded as the primary means of ensuring that data reported in case report forms accurately reflected the original source documents and were suitable for regulatory submission [Reference Manasco and Bhatt2–Reference Andersen, Byrjalsen and Bihlet4]. At the same time, well- established knowledge in the human reliability field has shown that manual inspection, such as SDV, is inherently error-prone and that inspection error should be considered a priori when estimating process capability and outgoing quality [Reference Megaw5]. The average error rate reported for human inspection is 10–20% (ref,5) .This is greater than the average error rate of 657 errors per 10,000 fields, or 6.57% (95% CI 5.51, 7.72) reported for data abstracted from medical records [Reference Garza, Williams and Ounpraseuth6]. This is yet again an order of magnitude above error rate reported for clinical trial data processing methods reported in the same large literature review and pooled analysis [Reference Garza, Williams and Ounpraseuth6].
Clinical trial monitoring has been reported to comprise 25–50% of a study budget, with a major portion attributed to SDV [Reference Olsen, Bihlet, Kalakou and Andersen7–Reference Funning, Grahnén, Eriksson and Kettis-Linblad9]. Because of its high cost and uncertain value, particularly for non-critical data, regulatory guidance has shifted toward risk-based monitoring approaches that emphasize critical data, critical processes, and quality by design rather than routine 100% SDV [Reference Andersen, Byrjalsen and Bihlet4,Reference Tantsyura, Dunn, Fendt, Kim, Waters and Mitchel10–Reference Tantsyura, Dunn and Waters12]. FDA guidance has similarly promoted risk-based monitoring, electronic source data, and electronic data capture technologies to improve data integrity while reducing unnecessary manual verification. Regulatory guidance on risk-based monitoring and Good Clinical Practice supports reducing monitoring burden and costs when sufficient data quality is maintained.
Despite decades of reliance on SDV and growing interest in risk-based monitoring, remarkably little evidence exists regarding the accuracy of the SDV process itself. Existing studies have compared monitoring strategies or evaluated study database error rates but have not measured the ability of SDV to detect existing data errors. The most recent and comprehensive literature review found no reports of SDV accuracy [Reference Hamidi, Eisenstein and Garza13]. Quantitative measures of SDV accuracy are needed to support the design and evaluation of targeted or reduced SDV approaches capable of achieving the level of data quality required for clinical research. Although the occurrences and consequences of poor data quality are infrequently reported, documented examples illustrate its potential scientific and clinical impact. Notable examples include retractions due to data error [Reference Levett, Elkaim, Alotaibi, Weber, Dea and Abd-El-Barr14,Reference de Granda-Orive, Jimenez-Ruiz and Aleixandre-Benavent15], perpetuated health inequities [Reference Steed, Liu, Wu and Acquisti16,Reference Getzen, Ungar, Mowery, Jiang and Long17],inability to use EHR data [Reference Zozus, Walden and Pieper18], and randomization error [Reference Hamidi, Abbaas and Bhatraju19].
A process error occurred during the NIRVANA clinical trial (NCT04818216) conducted in ICU-hospitalized COVID-19 patients created a unique opportunity to evaluate SDV process accuracy [Reference Hamidi, Abbaas and Bhatraju19]. Three sequential rounds of SDV were performed on the same study dataset, with two rounds occurring after data cleaning had been completed. This rare circumstance enabled measurement of the SDV process itself by determining the number of errors detected in data that had already undergone full SDV. The objective of this study was to quantify the accuracy of the SDV process by measuring its ability to identify data errors.
Methods
The source of all data at the two study sites was the local Electronic Health Record (EHR). The study used a traditional clinical trial data collection process – data needed for the study were identified in each patient’s chart and entered into the study electronic Case Report Form, provided through the study Research Electronic Data Capture (REDCap) system. REDCap is a secure, web-based, electronic data capture system for research studies [Reference Harris, Taylor and Minor20]. In the NIRVANA trial discrepant data values identified through data quality rules programmed in REDCap were flagged and resolved by the sites during data entry. The initial study monitoring plan included SDV by the study monitor, i.e., checking critical data, including safety labs, adverse events, protocol fidelity (eligibility, study drug administration), and efficacy measures, in the study database against the EHR to confirm that the data recorded in the study database were correct.
SDV was conducted during three separate, sequential, time periods. During period 1, the initial study monitoring plan was followed including SDV for the first patient at each site, followed by SDV for all patient visits completed by the time of the three-month monitoring visit. Only critical data were assessed during period 1. In period 2, full (100%) SDV was performed for all EHR-based data using the same procedures applied in period 1. In period 3, an independent external monitoring team, blinded to the period 2 findings, conducted an additional 100% SDV of all EHR-based data. Otherwise, SDV procedures in all periods were identical. Errors identified in period 3 were data errors that were either missed or introduced during period 2.
Any difference between the EHR and the study database not explained by trial documentation was considered a discrepancy, marked, and classified by the individual performing SDV. The site together with the monitor decided the disposition of each discrepancy, i.e., whether the transcribed data value in the study database or the EHR was errant, or whether the monitor was in error. When an error in the study database was found, the value in the database was corrected by the site and documented in the database audit trail. Thus, errors were easily detected computationally through the REDCap audit trail as data changes following the period’s SDV. By the start of period 2, all study data processing was complete. Thus, during the period 2 SDV, there were no reasons other than correction of errors detected by the period 2 SDV for data to change; similar for period 3. To facilitate tracking and reporting of the period 3 SDV, an exhaustive and mutually exclusive classification scheme was implemented for the identified data discrepancies and their disposition. Period 3 discrepancies were documented and classified according to the type of discrepancy (missing data in the study database, extra data in the study database, or discrepant data in the study database) by the external monitor using REDCap Data Resolution Workflow. Discrepancies were also classified according to the action taken to affect their resolution. Discrepancies resulting in changes to study data were counted as errors.
The point estimate of the source-to-database error rate obtained through SDV was calculated according to the Good Clinical Data Management Practices [Reference Zozus, Kahn and Weiskopf21] as the number of errors identified divided by the number of data values inspected. The error count is the numerator for the error rate. The number of fields SDV’d is the denominator. The field count estimation method used here accounted for erroneous omission and commission while avoiding denominator inflation by (1) excluding data values copied onto subsequent visits or forms, such as study, site, patient, and visit identifiers, (2) using the actual number of repeating events such as adverse events, and (3) adding errantly omitted data values in the field count. The 95% Confidence Intervals (CIs) were calculated using the method in Fleiss, Levin, and M. Paik [Reference Fleiss, Paik and Levin22].
Results
There were 14,394 data values inspected in period 1 and 55,455 data values inspected for periods 2 and 3 (Table 1). Period 3 SDV identified 484 errors across both sites, for an overall study database error rate of 0.87% (Table 1) with a 95% CI of (0.80, 0.95). No systematic data errors were identified. No errors were identified in data related to the study’s primary endpoint. Seven errors were identified in the study’s secondary endpoint data.
Error rates by period (SDV rounds)

Table 1. Long description
The table has four rows and five columns. The columns are labeled Number of data values inspected, Number of data errors identified, Data error rate (percent), 95% Confidence interval (percent), and Data error per 10,000 data values. The rows are labeled Period 1, Period 2, and Period 3. Row 1: Number of data values inspected, 14,394; Number of data errors identified, 401; Data error rate (percent), 2.79; 95% Confidence interval (percent), 2.53, 3.07; Data error per 10,000 data values, 279. Row 2: Number of data values inspected, 55,455; Number of data errors identified, 2,958; Data error rate (percent), 5.91; 95% Confidence interval (percent), 5.71, 6.11; Data error per 10,000 data values, 591. Row 3: Number of data values inspected, 55,455; Number of data errors identified, 484; Data error rate (percent), 0.87; 95% Confidence interval (percent), 0.80, 0.95; Data error per 10,000 data values, 87.
*Critical variables only were SDV’d in period 1. Periods 2 and 3 performed SDV for all data fields.
In period 2, there were at least 3,442 errors (instances where the database did not reflect the source) in the database – we call these errors “known” because they were detected in either the period 2 or period 3 SDV. There were likely some errors that escaped detection by period 2 or period 3 SDV and remained unknown. The period 2 SDV found 85.9% of the known errors. The period 3 SDV found and corrected 87.1% of the known errors existing at the start of period 3. However, both partial (period 1) and complete (periods 2, and 3) manual SDV missed data errors. Seventy-two period 2 findings were noted and counted in the period 2 error rate but inadvertently not corrected in period 2. These 72 errors were missed (not identified as discrepancies) in period 3, i.e., at least 72 (12.9%) of the known errors were missed by the period 3 SDV.
An additional 68 discrepancies identified in period 3 were miss-labeled by the monitor in the system Data Resolution Workflow and, as such, not visible to the sites for adjudication. Though these were identified prior to the database lock and corrected, they were missed by the period 3 SDV process. Both cases demonstrate fallibility in the SDV process itself. Overall, there were 484 (14.1%) of the extant data errors missed in period 2 but detected in period 3, i.e., the period 2 100% SDV missed 14.1% of the extant errors. Similarly, the period 3 SDV also missed 12.9% of the extant data errors. Extant data errors are the data errors existing in the data at the time of the SDV. Missed data errors are the extant data errors that were not detected by the SDV. For example, because the data were not changing at the time of the period 2 and 3 SDV, a data error detected in period 3 SDV is a data error missed by the period 2 SDV.
Discussion
The overall study database error rate estimated through (and prior to) corrections made based on the period 3 SDV, was 0.87% (95% CI 0.80%, 0.95%) and well within the acceptance criterion of 5% set for the study. The error rate is also well below the average error rate of 657 errors per 10,000 fields, or 6.57% (95% CI 5.51, 7.72) for similarly processed data reported in a large literature review and pooled analysis [Reference Garza, Williams and Ounpraseuth6].
The high error rate observed in the SDV process itself, 14%, is consistent with human inspection error rates reported in other industries [Reference Megaw23]. Accounts of errors in the SDV process with one exception, were not published until the cost of SDV was openly questioned [Reference Tantsyura, Dunn, Fendt, Kim, Waters and Mitchel10]. These accounts reported errors found or missed by one method relative to a comparator method, and as such, did not provide measures of SDV accuracy. The results presented here are the first reported measure of SDV accuracy, i.e., the number of errors found in data previously subjected to SDV [Reference Hamidi, Eisenstein and Garza13]. Measures of SDV process accuracy are important because the targeted and reduced SDV alternatives pursued today are expected to be capable of producing the data accuracy required for the planned study analysis [1]. Measures of accuracy are needed to support these a priori process capability claims such as by taking into account the error rate of the SDV process itself when planning partial and reduced SDV processes for data quality control in clinical trials. This work provides one SDV accuracy data point to inform quality by design. More are needed to cover capability prediction across contexts and data relevant to a specific clinical trial.
In the years preceding the 2013 FDA Guidance on Risk-based Monitoring the utility of the SDV increasingly came into question due the high cost compared to the limited value of applying SDV to non-critical variables. At the time, full SDV (100% of variables on 100% of the study participants) as was the predominant approach in the industry sponsored clinical trials. Since the 2013 FDA guidance, interest has focused on reducing or eliminating SDV of non-critical variables. Our findings do not contradict this approach.
As manual processes undertaken in complex healthcare records, neither Medical Record Abstraction (MRA) nor its close cousin, SDV have ever been on firm ground from a quality perspective [Reference Zozus, Pieper and Johnson24]. Historically, when most clinical study data came from the medical record, SDV was the only consistently available way to provide assurance that the study data reflected the source. SDV has not routinely been used in more sophisticated ways such as in process control for clinical studies, i.e., estimating of the error rate as feedback to control the accuracy of the MRA process. However, SDV has recently been demonstrated capable of controlling the MRA error rate [Reference Garza, Williams and Myneni25]. Our results here show that though costly and an example of Deming’s advice against inspecting quality into a product, SDV can provide a control mechanism to improve MRA accuracy and completeness. It is in this light that our results should be interpreted. As a fallible and manual process SDV, similar to human inspection in other industries, has a significant error rate that should be considered when designing processes that employ SDV.
Reliance on MRA and SDV should decline as more advanced electronic data generation and collection methods become widely available. Examples include device-based measurement and recording of data, electronic capture of patient-reported or provider-observed outcomes, and electronic extraction of study data from EHRs. Each of these approaches requires specific quality considerations [26], but obviates MRA and SDV by providing an exact copy of the source. However, these electronic approaches, can increase cost, often require manual integration with other study data, and are not applicable to all data needed for clinical studies. As such, advanced electronic data generation and collection methods have varying levels of adoption. Thus, MRA and SDV will be used for some time.
This study has important limitations regarding generalizability. The analysis was based on a single clinical trial conducted at two sites using a single EHR source environment, and the opportunity to perform repeated SDV under stable data conditions resulted from a unique procedural circumstance. Therefore, the observed 14% SDV error rate should not be interpreted as a universal estimate of SDV performance across clinical trials, sites, EHR environments, or data types. Rather, this study provides one quantitative demonstration that SDV is a fallible manual inspection process and illustrates an approach for measuring SDV process accuracy. Additional studies across diverse clinical research settings and data sources are needed to characterize the variability of SDV accuracy and inform context-specific quality-control strategies.
Conclusion
In this case study the SDV process missed 484 (14%) of the extant data errors. Even 100% SDV misses errors. As such, 100% SDV was never a guarantee of data accuracy. As advocated by the human reliability discipline, the error rate of manual inspection processes, here SDV, should be considered in process design, i.e., a priori estimates of process capability.
Acknowledgments
This work would not have been possible without the collective support and contributions of the entire Nirvana Clinical Trial team.
Author contributions
Muayad Hamidi: Conceptualization, Data curation, Formal analysis, Validation, Writing - original draft, Writing - review & editing; Subrata Debnath: Investigation, Resources, Validation, Writing - review & editing; Manju Bikkanuri: Data curation, Project administration, Validation, Writing - review & editing; Kumar Sharma: Conceptualization, Investigation, Resources, Supervision, Writing - review & editing; Meredith Zozus: Conceptualization, Data curation, Formal analysis, Investigation, Resources, Supervision, Validation, Writing - review & editing.
Funding statement
The NIRVANA trial (NCT04818216) was supported by the National Institutes of Health through grants from the National Institute of Diabetes and Digestive and Kidney Diseases (PI: KS, UH3DK114920); The SDV process accuracy report was supported by a National Center for Advancing Translational Science grant to the University of Texas at San Antonio Health Science Center (MH, MZ, UTSA-HSC UM1TR004538); Manuscript preparation was supported by a grant from the Burroughs Wellcome Fund (MZ, BWF G-1021753).
Competing interests
All authors declare that they have no conflicts of interest or competing interests related to this work.

