Background
Recent studies demonstrate the potential utility of artificial intelligence (AI) systems, including large language models, for infection and stewardship surveillance activities. Reference Morgan, Goodman and Branch-Elliman1 These studies have largely focused on text data available in the electronic health record (EHR) applied to a relatively small corpus of text note data. Reference Morgan, AlShanqeeti and Coffey2–Reference Goodman, Shams and Magder4
Clinical notes entered in the EHR comprise a large proportion of hospitalization data and are particularly rich for electronic surveillance and stewardship activities. Historical medical information and information from external health systems is often in the form of a scanned document stored as an image in the EHR. These files are not searchable using standard natural language processing (NLP) and text-based methods. Recent advancements in AI systems, including commercially available large language models that include Optical Character Recognition (OCR) capabilities, present an opportunity to use these AI tools to access scanned notes, including historical hand-written notes, for a variety of purposes, although little is known about the accuracy and feasibility of this approach for classifying unstructured information that is challenging under existing systems to process and analyze. Reference James, Maran and Rice5,Reference Damani, Hinton and Hunt6 Thus, the objective of this pilot project was to explore the accuracy of two publicly available AI models with OCR capabilities for extracting antibiotic administration data from scanned-in preoperative and anesthesia records as part of a larger effort to create an electronic system for monitoring compliance with perioperative antimicrobial stewardship recommendations.
Methods
This project was conducted as part of a larger effort in the Veterans Health Administration (VA) to electronically replicate the Surgical Care Improvement Project (SCIP) manual review process, which, until 2015, included three antimicrobial stewardship measures regarding drug names and timing of surgical prophylaxis: INF-1, INF-2, and INF-3. Reference Branch-Elliman, Elwy and Lamkin7 SCIP encompassed major inpatient cardiac, orthopedic, vascular, colorectal, and gynecological surgical procedures. Only procedures with documentation of any antimicrobial receipt were eligible for inclusion and evaluation in the program. Reference Bratzler and Hunt8
The VA Corporate Data Warehouse (CDW) was used to identify all surgical procedures that would have been eligible under historical SCIP rules. Then, structured fields in the CDW were used to identify antimicrobial use (eg, orders, administration data) to measure case inclusion and determine compliance with SCIP measures. Details of the creation of the electronic tools and the development and validation of the structured data elements are previously published. Reference Mull, Golenbock and Pendergast9 However, following application of structured variables, a portion of surgeries had no evidence of documented antimicrobial administration in the CDW, prompting concern that data were recorded but inaccessible with standard data extraction methods. Thus, we sought to identify additional strategies for measuring this unstructured data within the EHR.
A random sample of 10 orthopedic surgeries from each of four VA facilities with the highest rate of undocumented antimicrobial receipt between October 1, 2023 and September 30, 2025 was selected for manual review using a randomization program available in SAS version 8.3 (SAS Corporation, Cary, North Carolina). These cases were manually reviewed using the Joint Legacy Viewer (JLV), the graphical user interface with the VA EHR, which allows for review of scanned-in records, including PDF and hand-written documentation, not available in structured data fields or clinical notes directly entered into the EHR. Two trained reviewers, one with a pharmacy background (RO) and 2 years of chart review experience, and another infectious diseases physician (WBE), evaluated the medical records for (1) receipt of any antibiotic, (2) antibiotic type, (3) antibiotic dose, and (4) administration time. The location of the data (eg, in clinical notes, in pharmacy records, in scanned-in images) was also collected. Among cases where information was only contained in scanned-in image data, the relevant section of the administration documentation was saved as an image.
These images were then submitted to VA-approved large language models with OCR capability (Claude Sonnet 4.6 or OpenAI version 5.0) in June 2026. Prior to systematically evaluating each tool, one study member conducted pilot work in Claude beginning with the basic prompt “what does this say” with an image of physician handwriting. The AI response was an address in Indonesia. The next prompt added “this is physician’s handwriting of a drug name and drug dose” to the question “what does this say” and the results were more accurate; however, the final image search resulted in a description of the note data with the sentence “this confirms the earlier handwritten note” when the notes were separate cases and the AI tool had not been instructed to use prior information (see Supplementary Materials 1 for examples of the images, the prompt provided, and the AI generated calls). Following these early attempts at data extraction, we then used more directed and tailored prompts, similar to those that had been used in previous infection prevention surveillance efforts, Reference Morgan, AlShanqeeti and Coffey2,Reference Basnet, Mull and Morgan10 to assess feasibility of the AI-based approach for extracting the unstructured antibiotic data elements.
The main analysis of 10 cases was performed by a separate study team member to ensure no history in either model and used with a prompt requesting classification of antibiotic receipt and name, dose, and time: You are an infection surveillance nurse. I am going to provide you with a physician note. I want you to read all of it, including any hand written portions, and convert the information into structured data elements for standardized reporting. Use only the information I provide you and do not make anything up or supplement it (see Supplementary Materials 2 for examples of the images, the prompt provided, and all AI generated calls and output). Each tool’s response was collected only once. Accuracy of both of the AI tools was measured against the gold standard manual review results, with the goal of ascertaining whether an AI-driven process would be an accurate and feasible approach for extracting antimicrobial administration data when it is not accessible via other structured or text-based extraction mechanisms. A TRIPOD Statement is included as Supplementary Materials 3. Reference Gallifant, Afshar and Ameen11
This work was approved by the VA Boston Research and Development Service prior to data collection and analysis. All AI tools are VA-approved and HIPAA-compliant; per the federal licenses, these tools do not retain or learn from data provided to them within the VA closed ecosystem.
Results
Among 40 randomly selected orthopedic surgery cases without evidence of any antibiotic administration in standard data tables, any antibiotic administration was identified in 39 cases. Of these, antibiotics were identified in searchable text in 25, leaving 14 cases with documented antibiotic administration in a scanned note associated with the surgery (eg, anesthesia record) without searchable text or pharmacy record data.
Human determination, AI generated output, and accuracy of the antibiotic administration data documented in images are presented in Table 1 (See Supplementary Material 2 for images and original output). The complete accuracy of the AI tools, including accurate determination of drug name, dose, and administration time was overall low (3/14, 21.4% for Claude and 2/14, 14.3% for OpenAI), although performance for individual elements was substantially higher. The AI tools were able to identify any administration and type (Claude, 14/14, 100%; OpenAI, 13/14, 92.8%). The case that OpenAI was unable to read was a handwritten record and notably the AI tool responded that it was uncertain in its determination in its response.
Prompts and outputs
Table 1. Long description
The table presents a comparison of human determination and AI-generated output for antibiotic administration data. It includes columns for image ID, clade, drug, dose, time and date, dog, and various elements of AI determination and accuracy. Each row provides specific details for different images, including the date, time, medication, dose, and administration information. The table also indicates whether the drug, dose, and administration time were correctly identified by the AI tools, Claude and OpenAI. Notable trends include the overall low complete accuracy of the AI tools, with individual elements showing higher accuracy. The AI tools were generally able to identify the administration and type accurately.
a Images included in Supplementary materials and are available for manual review and validation.
b Partially correct responses categorized as incorrect.
Both AI tools had limited accuracy for measuring the total dose administered (7/14, 50% for Claude, 6/14, 42.9% for OpenAI). These errors were primarily due to an inability to distinguish between standard aliquots (ordered in 1-gram vials) versus the total dose administered (which typically included multiple vials of drug and/or may have included intraoperative re-dosing). While the AI tools could accurately extract the text from the record, they sometimes did not interpret the clinical documentation correctly. For example, in some cases, the tool interpreted 1-gram vials and 2 g administered as 3 total grams administered, rather than appreciating that the size of the aliquot was different than the information about the total dose administered. However, and reassuringly, in these cases, the tools did often call that they were unsure of the correct response.
Antibiotic administration time was accurately identified in most records unless graphic ranges were presented. Anesthesia record formats may include graphs with specific times delineated and then the antibiotic administered within a time range (eg, time stamps for 12:00 and 12:15 with the demarcation of antibiotic administration falling within this range). In the event a graph or range was used to document administration time, the errors in ascertainment of timing were due to the inconsistency with which the AI tools selected one end of the range or the other (ie, either the start or end of the range). How the AI selected one end of the range was inconsistent from case to case. Similar to dose errors, these were interpretation errors rather than extraction errors and the AI tool’s notes indicated that multiple date/time administrations were possible.
Discussion
As part of a larger effort to capture antimicrobial use metrics within the EHR, Reference Branch-Elliman, Elwy and Lamkin7 we tested the feasibility and accuracy of two commonly used and commercially available AI tools for measuring antibiotic name, dose, and timing from scanned document images. We found that accuracy for identifying any administration of an antibiotic was high, but that specific details, such as total dose administered and exact timing of administration were often incorrectly classified and depended, in part, on how the anesthesia record was organized. Thus, the feasibility of using these AI tools for extracting detailed antimicrobial use data depend, in part, on the specific goals of the program, including what information is needed and how precise measurements need to be to ascertain compliance. It is also possible that additional instruction about how to interpret the information would have led to improved performance.
Prior to testing accuracy, we first attempted to use a more general prompt, which resulted in an outlandish response from the AI tool, for example stating that a snip about antibiotic administration was an address in Indonesia. We thus used an approach of instructing the tool to behave as an infection surveillance expert, an instruction that has been used successfully in other studies and that grounds the context of the query for the LLM. Reference Morgan, AlShanqeeti and Coffey2,Reference Basnet, Mull and Morgan10 We explicitly prompted the AI to avoid interpretation and it is possible that a more complex approach, such as creating a labeled data set and providing it to the model to demonstrate how to interpret the images, would have led to additional improvements. While additional gains may be made by utilizing LLMs built specifically for, or fine-tuned to, patient chart data, issues may arise from task-alignment. However, these more specific models would not benefit from all of the data available to frontier laboratories. Leveraging an agentic-architecture is also a potential approach; however, issues arise given that most health systems restrict automated access to patient data and the potential tool-calling abilities of any system would be limited by institutional privacy and security standards and is unlikely to be available to most providers interested in using AI tools for stewardship quality measurement.
Although both tools demonstrated acceptable accuracy in extracting information from scanned records that cannot be accessed with traditional NLP tools, accuracy depended largely on the characteristics of the record image itself. The AI tools demonstrated near-perfect performance when the scanned-in records were in typed font; in the few records with hand-writing and/or with graphic representations of time, the accuracy was lower. However, it is important to note that the human was able to use the larger context of the handwriting to decipher the meaning and this larger context may not be processed by an AI system in the same way. Additionally, in several cases, the quality of the handwriting was poor, potentially contributing to the AI’s difficulty in correctly characterizing the output. The AI tools were also unable to decipher data presented in a graphical format, whereas a human can comprehend the meaning of the graphic time lines.
The addition of scanned records into the EHR ensures clinicians can access these files as images to deliver comprehensive patient care. For large scale surveillance activities, the lack of electronic search functionality in some of the VA CDW’s scanned-in clinical notes has stymied NLP efforts. This exploratory study is to our knowledge the first to evaluate the utility of AI to extract data from scanned non-searchable images that are otherwise inaccessible electronically. Although the AI tools were inaccurate on precise details about antibiotic use perioperatively, this study provides evidence that electronic data extraction is feasible, and the AI tool’s identification of ambiguous information suggests an opportunity for AI training and prompt engineering to improve results. While there is not currently an option in the VA CDW for bulk extraction of non-searchable clinical notes for mass AI-driven data labeling, large language models like those from Claude and OpenAI have the potential to contribute to surveillance activities if this functionality were available. These AI systems can be successfully used to extract data from images, particularly if the image data is structured and clean. Output may be less reliable when human hand-writing, which is inherently highly variable itself, is the target of the surveillance activities, however, the relative use of hand-written notes is falling over time and thus this limitation is likely relatively less impactful as the EHR is increasingly adopted. 12
There are several limitations to this study. First, this study was conducted in the VA EHR with VA specific systems and search functionality may be different in different health systems. Second, the LLMs were applied to a relatively small corpus of data. We did not assess how these tools can be used at scale for data extraction. Third, the images provided to the AI tools were small and limited to the section of the clinical note with the relevant information. We did not assess the accuracy of these tools when the full note text was available. The major implication of this is that we did not assess how well these tools work when antibiotics are used for other purposes, such as for local site irrigation, rather than systemically, which may introduce noise and potentially worsen accuracy. Fourth, the goal of this study was not to train AI tools to do this task optimally with robust prompt engineering, rather it was to see how well general LLMs can do on this task with a simple prompt. Future studies should explore how prompt engineering can be used to improve this process, including improving interpretation of graphical information on timing and medication dose.
In summary, AI tools were able to extract some data elements from scanned electronic clinical records with high levels of accuracy, but the overall performance for comprehensive extraction of all clinical data elements was poor when general prompts without a data labeling and training step were used. The structure and quality of the clinical note itself drove AI performance suggesting that the utility of AI tools for image data extraction depends on both the reliability of the AI tool and the suitability of the clinical image for AI review. This has implications not only for the current study but also for the use of AI tools for extracting data from clinical notes more broadly. More work is needed to verify accuracy and utility as AI tools are increasingly used to summarize non-searchable electronic data.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/ash.2026.10919
Data availability statement
All relevant data are included in the supplementary materials and the table in the manuscript.
Acknowledgments
We would like to thank Sunny Singh, MHI for his insights into AI tools applied to electronic health records and specifically about the use of agentic approaches in medical contexts.
Financial support
This work was supported by Veterans Affairs Health Systems Research Investigator-Initiated Research award IIR 20-101 (MPI, Branch-Elliman and Mull).
Competing interests
All authors report no conflicts of interest.
The views expressed are those of the authors and do not necessarily represent those of the US Federal Government or the Department of Veterans Affairs.
Ethical standard
Given the retrospective nature of the study, a waiver of informed consent was provided.
Consent for publication
Not applicable.
Open access
