Clinical registries represent important sources of data on practice and outcomes across multiple clinical fields, but particularly so in pediatric and congenital cardiac care where administrative data often fails to accurately capture detailed anatomic and procedural information.Reference Pasquali, He and Jacobs 1 , Reference Jantzen, He and Jacobs 2 Clinical databases provide an ideal foundation for quality improvement initiatives, research activities, policy decisions, and public reporting.Reference Pasquali, He and Jacobs 1 – Reference Pasquali and Jacobs 3 In this context, it is important for these registries to maintain high standards of completeness and accuracy for the clinicians, researchers, and public who use the information.
The Pediatric Cardiac Critical Care Consortium (PC4) is a quality improvement collaborative of hospitals from North America who voluntarily participate and submit clinical data on patients with critical cardiovascular disease cared for in cardiac ICUs.Reference Gaies, Cooper and Tabbutt 4 The mission of PC4 is to provide benchmarks of quality to participants through clinical outcome reporting, to identify variation in performance between hospitals, and to use the data as a catalyst to initiate quality improvement efforts that will lead to better outcomes for these critically ill patients.
Data collection officially began at eight institutions between October 15, 2013 and January 1, 2014. A detailed audit was performed at each of these institutions during the 1st year of data collection. In this manuscript, we describe the audit methodology and report the findings from the audit to provide evidence regarding the integrity of the data in the PC4 clinical registry.
Materials and methods
The clinical registry and data collection processes
An overview of the variables in the PC4 registry and the database features were described previously.Reference Gaies, Cooper and Tabbutt 4 Before data submission, the data coordinating centre staff perform on-site training with the local data collection team members over 2 days, including reviewing data definitions and data entry processes and assistance with the submission of multiple cases. All data collectors must pass a certification exam testing knowledge of the database variables and coding rules initially and after every software version change. PC4 conducts weekly teleconferences with all data collection teams and hospital clinical champions to review definitional questions and ensure consistency of interpretation across sites; the data coordinating centre circulates content from these calls to all sites. The collaborative also hosts all data collection teams at an annual in-person meeting to further refine these processes.
The registry contains information on all encounters where a patient with cardiovascular disease is admitted to any critical care unit under the care of the attending cardiac intensivist. The database is relational in design and includes patient, hospitalisation, and intensive care encounter variables. Detailed information is collected and date-stamped related to critical care interventions and outcomes specific to cardiac critical care, including complications. More than 90% of the fields in the registry are mandatory for case submission. A robust system of internal data checks was built into the data entry software to prevent missing, illogical, or implausible data to pass from the site to the data warehouse; all errors are flagged and must be resolved by the data collector before the case can be successfully submitted.
Several variables with common terminology and definitions are shared between the PC4 registry, the Society of Thoracic Surgeons Congenital Heart Surgery Database, the Joint Congenital Cardiac Anesthesia Society–Society of Thoracic Surgeons Congenital Cardiac Anesthesia Database, and the American College of Cardiology IMPACT Registry®; the nomenclature of the International Pediatric and Congenital Cardiac Code is used in all of these databases.Reference Franklin, Jacobs and Krogmann 5 The data entry software allows sharing of these common fields at the local level to avoid duplicate data collection and to minimise the burden of data entry. For the purpose of maintaining consistency across databases, PC4 data collection teams are instructed to use the fundamental diagnosis, procedural diagnosis, and define all procedures according to how they are coded in the Society of Thoracic Surgeons Congenital Heart Surgery Database.
Audit content
The Data Coordinating Centre analytic team and the PC4 Database and Executive Committees reviewed the entire registry to determine the key fields for audit and definitions of major and minor discrepancies for the audited variables. The list of these fields and definitions of major and minor discrepancies are shown in Appendix A. Shared fields with other registries are not included in the audit as these fields are already subject to review by other entities – for example, The Society of Thoracic Surgeons. We did evaluate whether the dates of all cardiothoracic surgeries were coded correctly to ensure that surgical cases are not missing in the PC4 database and that they are appropriately timed in relation to therapies and complications.
Audit process
The overarching approach to the maintenance of data integrity focusses on two main domains: data accuracy and timely submission of completed cases. In-person audits began in January 2014; three members from the Data Coordinating Centre (M.G., J.E.D., and G.M.W.) audited seven sites. The clinical champion (S.T.) and data collector from the University of California-San Francisco performed the University of Michigan audit. The audit was carried out over 2 days. Auditors were granted access to the medical records by the participant institution for the purpose of quality improvement review. Before arrival, the data coordinating centre generated a randomly selected sample of intensive care encounters. A patient could have more than one encounter in the sample.
Data accuracy was assessed through source data verification at the participant site. The planned audit included review of 62 encounters at each site; this number was chosen because we predicted that this was the maximum number of cases that could be completed over a 1.5-day audit. This case number represents a 5–15% sample of yearly cases depending on the centre volume. The audit team aims to perform blind abstraction of the entire data set included in the audit for two encounters, and on the sample of key fields chosen by the Database Committee necessary for reporting outcomes and performing risk-adjustment on 20 encounters. In addition, the team performs source data verification on the same key fields with the data collector on up to 40 additional encounters. As the system was learnt over the first three audits, the team reviewed only 75% of the maximum number of planned encounters. Process improvements were enacted to increase efficiency, but specific circumstances prevented the team from reviewing some records in the sample at the remaining sites.
In addition, two auditors (M.G. and G.M.W.) completed the blind chart abstraction using the electronic medical record to obtain values for each data variable without knowledge of how those fields were originally coded by the participant data collection team. We then compared the results of this blind abstraction with the record in the data warehouse and identified discrepancies. Source data verification was performed one-on-one with the data collector at each site. The data collector opened the medical record for selected cases and reported a value for each field in the audit. A member of the audit team (J.E.D.) compared responses with the values in the data warehouse for that case. Discrepancies were recorded and ultimately resolved through a secondary review. The discrepancies were reported to the local data collection team, who subsequently verified whether there was a true discrepancy or an error in abstraction by the auditor. After completion of this adjudication process, a final quantitative report was released to the site.
PC4 assesses completeness and timeliness of case submission by comparing all the cases submitted to the data coordinating centre at the time of audit to a third-party source document. This is provided by the participant and includes all admissions to the cardiac ICU over the same time period as the audit cases. This list of admissions has to be from a source other than that maintained by the data collection team – for example, hospital billing or census records. Discrepancies between these lists were reviewed during the in-person audit. The expectation is that all cases are submitted to the data coordinating centre within 1 month of the hospital discharge date.
Quantitative analysis
All audited fields were included in the analysis; however, if an error was made on a data field with a “parent–child relationship” to other fields, we removed the “child fields” from calculations in order to avoid double counting discrepancies – for example, if a central venous line was not recorded, we did not count errors for all the other fields that would have been related to that catheter had it been recorded: start/stop time, site of insertion, location of insertion, etc. In the extremely rare instance where a software data check allowed an implausible value to enter the data warehouse – for example, illogical dates – we did not count the discrepancy, and the denominator of adjudicated fields was adjusted to reflect these situations. If such an error was discovered, the software vendor immediately updated the data check and re-submitted cases. Finally, if the audit team discovered a systematic error in data entry, such as misinterpretation of a variable definition, this field was removed from the entire audit accuracy calculation and the cases were corrected and re-submitted with the correct data by the local data collection team.
We calculated two different accuracy metrics. Overall accuracy was calculated as follows:
$${{{\rm Fields}\,{\rm without}\,{\rm a}\,{\rm major}\,{\rm or}\,{\rm minor}\,{\rm discrepancy} } \over {{\rm Total}\,{\rm fields}\,{\rm audited} }}{\times} 100$$
We then calculated a major discrepancy rate; as not all fields could have an associated major discrepancy, we only included fields in the denominator where there was a potential major discrepancy:
$${{{\rm Fields}\,{\rm with}\,{\rm major}\,{\rm discrepancy}} \over {{\rm Fields}\,{\rm audited}\,{\rm with}\,{\rm potential}\,{\rm for}\,{\rm a}\,{\rm major}\,{\rm discrepancy}}}{\times}100$$
In addition to calculating aggregate overall accuracy and major discrepancy rate for the entire group, we determined these rates for each hospital. We then calculated overall accuracy and major discrepancy rate within specific domains of the registry in aggregate and by site: patient demographics, hospital-level and intensive care encounter-level information, cardiac surgical data – number and dates of procedures – devices and therapies and complications.
We also calculated the timeliness of case submission by comparing the number of cases in the data warehouse with the document provided at the time of audit and reported the percentage of encounters submitted within 1 month of the hospital discharge date. We also performed a simple count of patients not appearing in the data warehouse whose hospitalisation ended in mortality.
Results
Across eight hospitals, we audited 434 encounters and 29,476 individual data fields. At the first three centres, 41, 45, and 44 encounters were reviewed, respectively. As the efficiency of the process improved, no less than 58 encounters were reviewed at each of the five subsequent sites.
Accuracy
Results of the audit are shown in aggregate in Table 1 and broken down according to the registry domains in Table 2. The aggregate overall accuracy was 99.1%, 255 total discrepancies. By centre, rates of overall accuracy ranged from 98.9 to 99.5%. The overall accuracy within each domain of the registry ranged from 97.5% to 99.7%.
Aggregate scores (n=8 sites).
Accuracy scores by domain (n=8 sites, %).
The aggregate major discrepancy rate across hospitals was 0.62%, 121 major discrepancies, with 19,639 fields eligible for a major discrepancy. The major discrepancy rates within the individual registry domains ranged from 0.33% for complications to 1.1% for patient demographics. The major discrepancy rates for the individual hospitals ranged from 0.3 to 0.92%.
Timeliness and completeness of case submission
The percentage of cases submitted to the data warehouse within 1 month of hospital discharge ranged from 80.2 to 98.8% across centres. Of the eight hospitals, seven submitted >90% of their cases within 1 month of discharge. The one centre with 80% submission had recently lost their data manager; an analysis of missing cases revealed that all of them were in the local database but had not yet been submitted to the data warehouse. When analysing cases that did not appear in the data warehouse at each site, five records were those of patients whose hospital stays ended in mortality. These patients’ records existed in the local database but had not yet been submitted due to incomplete data fields. Since the audit, the data coordinating centre has confirmed the submission of all cases that were outstanding at the time of the audit.
Discussion
Results from the 1st year audit at the initial eight hospitals submitting data to the PC4 clinical registry demonstrate excellent accuracy with no evidence of purposeful omission of cases. These data compare favourably with those reported by investigators from congenital cardiac registries and others.Reference Clarke, Breen and Jacobs 6 – Reference Jacobs 10 The integrity of the data evident in this report should instill confidence in those who would use the data for outcome reporting and benchmarking, clinical research, and future dissemination to the public.
Data accuracy is particularly important within PC4 due to the relatively unique process of data flow from and back to the participants. Unlike many other registries in congenital cardiac care where cases are submitted in batches, PC4 data collection teams are encouraged to submit cases individually as soon as the hospitalisation ends and all data fields are complete. This process exists to support the real-time nature of outcome reporting within the collaborative. Cases that are submitted become incorporated into the online reporting platform within 24 hours, and participants can view their aggregate results and perform local drill downs to identify individual patients in the registry with the most current information. This process and the online platform were created to give participants the tools for near real-time, ongoing assessment of performance in order to stimulate local quality improvement initiatives.
Maintaining such a process requires a robust set of data integrity checks built in to the data entry and reporting platforms. In collaboration with the software partners supporting these solutions, PC4 has created a system where clean records can be submitted without missing or illogical data, and these records are ready for use immediately by clinicians and researchers. Records with missing data do not enter the data warehouse. In addition to these electronic data checks, PC4 has invested a significant amount of time to promote and maintain consistent data collection across sites that conforms to the rigorously defined data definitions manual. These include dedicated in-person installations and training, regular contact with data collection teams through scheduled and ad hoc meetings, and maintenance of certification examinations for data collectors. The results of the initial audit confirm that these efforts have been successful.
In addition, in April 2015, PC4 participants began sharing data internally in an unblinded manner; clinical champions at each site are able to see aggregate centre-level data and identify the other participants. Sites that have submitted data for at least 1 year and passed the initial audit become eligible for this function using an opt-in opt-out approach; seven of the eight initially eligible sites opted in. This degree of transparency demands the highest standard of data quality, and various processes maintained by PC4, including the audit method presented here, have given participants confidence to use the data in this manner.
PC4 committed to auditing every hospital within the 1st year of data submission in order to ensure that every data collection team was given the opportunity to identify potential barriers to accurate and timely data submission as soon as possible. The audits were conducted in a completely non-punitive manner. Assessing and improving the accuracy and efficiency of data collection processes at the local level were the primary goals of the initial audit. The auditing team searches for evidence of systematic issues with data collection and case submission and provides education and support to the local teams to remedy any potential issues – for example, if an error in definition interpretation is uncovered, the audit team helps identify all cases – that is, even those cases that are not part of the audit sample – that may contain an error, and the participant data collection team edits and re-submits those cases as appropriate. At the conclusion of the audit analysis, the data coordinating centre personnel review the findings with the participants and cases are updated to reflect the most accurate data. Most data collection teams expressed great satisfaction with the process, were receptive to feedback on their performance, and embraced opportunities to improve their work. The data coordinating centre personnel learnt a great deal by observing data collection in situ at each hospital, and this knowledge continues to foster our ability to disseminate successful strategies for data collection across the collaborative and to new participants. The result of this shared experience is a path of continuous quality improvement in data collection methods.
The PC4 Executive Committee defined a passing audit score as a major discrepancy rate of <1.5%, >97% overall accuracy, and >90% of cases submitted within 1 month of hospital discharge. Participants achieving these standards receive and maintain privileges of data use within the collaborative, including access to unblinded data. Failure to meet standards of accuracy and timeliness trigger a remediation programme including re-education and close monitoring. The site would be re-audited within 1 year.
We found no significant differences in audit performance across sites, although the participating programmes varied in case volume and available resources for data collection. Based on the 1st year of experience, most hospitals dedicate approximately one full-time equivalent of data collector time per 800 CICU admissions; however, hospitals with fewer resources available for data collection have created systems to enter cases accurately and on time, as demonstrated in this report. Hospital case volume varied among the eight participants from 388 to 977 CICU admissions per year, and therefore a 62-case sample during the audit represents 6–16% of a hospital’s yearly volume. As described, the sample size for audit was chosen based on the maximum number of cases that can be completed in a one and a half-day on-site visit. The data coordinating centre is considering the feasibility of adjusting sample sizes based on hospital case volume in future audits.
Going forward, PC4 will continue to perform audits at each new participant site within the 1st year of data collection initiation. All hospitals will be audited at least once every 2 years thereafter. There will likely be changes to the audit process in both content and methods, as the registry evolves and the number of participating sites grows. As new data variables are incorporated into the database, some or all of these will need to be verified in future audits – for example, in August 2014, participants began collecting an additional set of variables that will be used for case-mix adjustment across hospitals in order to evaluate differences in mortality rates and other clinical outcomes. The next template of key fields used in the audit must include those that are included in the final risk model to verify accurate collection of data necessary for complexity-adjusted outcome reporting. Conversely, some fields were identified in the initial audit where errors are exceedingly rare, and/or external auditors are unlikely to abstract the medical record more accurately than the local collection teams. These fields will be dropped in future audits, allowing the auditing team to focus on higher-impact aspects of the database.
Although the Data Coordinating Centre personnel have conducted all but the University of Michigan audit, this may be an unsustainable model over time. Future iterations of the audit process might involve other PC4 data collection teams in a peer-to-peer audit, a third-party organisation, or some combination of these options. Each approach presents numerous potential advantages and disadvantages. Although peer-to-peer auditing could be advantageous by allowing bidirectional exchange of knowledge, this approach could be perceived as less rigorous with a potential concern that the auditors may “let mistakes slide”; however, given the degree to which hospitals in the collaborative share data transparently with one another, and the commitment to using the data for quality improvement purposes among participants, it seems highly unlikely that peers will allow inaccurate data collection to occur. Although a third-party group may mitigate some of these potential concerns, there can be limitations regarding content knowledge necessary to perform a detailed audit to the level currently required by PC4, and the cost could also be prohibitive.
Conclusion
Results from the 1st year audit of the PC4 registry demonstrate accurate, timely, and complete submission of data to a database designed to promote collaborative quality improvement for children with critical medical and surgical cardiovascular diseases. Regular source verification of data is the gold standard approach among congenital cardiac surgical databases such as the Society for Thoracic Surgeons Congenital Heart Surgery Database and the European Association for Cardio-Thoracic Surgery Congenital Heart Surgery Database, and it should be the expectation that critical-care databases for this population maintain the same standards. PC4 remains committed to the ongoing efforts to verify and report on the data integrity of the registry.
Acknowledgements
The authors recognise the work of all data collection team members at the participating Consortium sites whose efforts contributed to the data quality.
Financial Support
Dr Michael Gaies received support from the National Heart, Lung, and Blood Institute (K08HL116639, PI: Gaies) that indirectly supported this research.
Conflicts of Interest
None.
Supplementary Material
To view supplementary material for this article, please visit http://dx.doi.org/10.1017/S1047951115001833
