Introduction
Patient information leaflets are a cornerstone of modern healthcare, providing patients with essential information to make informed decisions about their care.Reference Sustersic, Gauchet, Foote and Bosson1, Reference Nedbal, Naik, Davis, Madaan, Tokas and Cacciamani2 Communication of information in a way that is understandable to patients is central to enabling shared decision making.
Despite guidance on best practice for producing patient information literature,3 the items produced are often not standardised and the process of developing and updating patient information literature is cumbersome and time-consuming. Patient information leaflets require regular updating to reflect new research and clinical guidelines. The burden of producing and updating patient information leaflets falls on clinicians, who are already under pressure as a result of the increasing demands of healthcare services and the current medical workforce crisis. These pressures limit the time available for non-clinical tasks such as creating and maintaining patient information literature.
Artificial intelligence (AI) offers a promising solution to address some of these challenges. Artificial intelligence is one of the most transformative advances in technological history, capable of quickly processing large data sets and performing tasks traditionally carried out by humans. Clinicians are recognising its potential to support patient care, with significant advancements in targeted prognostic, diagnostic and therapeutic interventions over the past 30 years.Reference Tama, Kim, Kim, Kim and Lee4, Reference Bassani, Santonicco, Eccher, Scarpa, Vianini and Brunelli5
Artificial intelligence is already being utilised in healthcare to analyse histopathology slides, radiographic images and other medical imaging modalities.Reference Petsiou, Martinos and Spinos6 In recent years, AI technologies, such as machine learning and deep learning, have found application in diverse facets of ENT surgery, from use in automated audiometry and clinical prediction in audiology,Reference Frosolini, Franz, Caragli, Genovese, de Filippis and Marioni7 to interpretation of computed tomography, magnetic resonance and endoscopic imaging.Reference Wu, Wang, Liang, Luo, Zhou and Deng8
By automating and improving the efficiency of producing patient information leaflets, AI could reduce the time clinicians spend on these tasks and ensure patient information leaflets are kept up to date with current evidence and guidelines. Furthermore, AI offers the potential to personalise written information to individual patient needs, such as age, medical history, language, reading ability and visual impairment. This would make information more relevant, accessible and understandable to a diverse patient population, supporting progression towards more individualised patient-centred care.
Whilst there are significant potential benefits to utilising AI to produce patient information leaflets, there is a risk of harm to patients if the literature produced contains incorrect information, omits important information or is written in a way patients cannot understand.Reference Verran9 Delivering poor-quality AI-generated patient information leaflets to patients could influence patients’ decision making and pose a risk to patient safety, therefore clinician oversight to safeguard the quality of the literature produced is essential.
In the past few months, there has been a surge of interest in the potential to utilise AI in producing patient information leaflets. Several studies have explored the use of AI to produce patient information leaflets across numerous specialties, including dermatology,Reference Verran9 vascular surgery,Reference Haidar, Jaques, McCaughran and Metcalfe10 general surgeryReference Ghanem, Rouhi, Al-Houssan, Saleh, Moccia and Joshi11 and neurosurgery.Reference Temel, Erden and Bağcıer12 Within ENT, two recent studies explored the use of AI to generate patient information, but found that AI-generated information was of lower quality and more difficult to read than pre-existing literature written by clinicians.Reference Abou-Abdallah, Dar, Mahmudzade, Michaels, Talwar and Tornari13, Reference Shamil, Ko, Fan, Schuster-Bruce, Jaafar and Khwaja14 To our knowledge, there have been no investigations into the acceptability of AI-generated patient information literature to patients themselves.
We performed a single-blinded study to investigate the feasibility of using AI to adapt patient information literature, assess whether patients find AI-adapted leaflets readable and determine whether patients exhibit a preference for AI-adapted leaflets versus existing human-generated patient information literature.
Methods
We identified four examples of patient information leaflets given to patients in an ENT department in a district general hospital in England. These leaflets covered two common ENT procedures (tonsillectomy and myringotomy and grommet insertion) and two common ENT conditions that required further patient education (nasal fracture and epistaxis).
Open-source ChatGPT (https://chatgpt.com) is a large language model developed by OpenAI (San Francisco, CA, USA) that uses AI to produce human-like responses to prompts in a natural language, and it was used to modify the existing patient information leaflets. We inserted the content of our Trust’s existing human-generated patient information leaflets into ChatGPT version 3.5 (https://chatgpt.com) in August 2023, alongside the prompt ‘Please refine the patient information leaflet provided using up-to-date medical information and adjust the content for a UK reading age of 14’. This age was chosen as research into health literacy has shown that the majority of adults in the UK have a reading age of 11–14 years.15
The AI-adapted content was then manually formatted to have similar layout and presentation to the existing documents because ChatGPT version 3.5 (https://chatgpt.com) does not have the ability to preserve complex formatting such as headings, images or layout styles. This was performed to maximise patient blinding as to whether each leaflet was human-generated or AI-adapted. Examples of the pre-existing human-generated and AI-adapted epistaxis patient information leaflets are included in Appendices 1 and 2, respectively.
A survey was designed to ascertain whether there was a patient preference for the AI-adapted or human-generated patient information leaflets. The survey questionnaire (Appendix 3) consisted of five questions, asking the participants to grade their preference between leaflet A, leaflet B or no preference, and individually assess the quality of each leaflet on a 10-point Likert scale in 5 domains. The questions addressed leaflet presentation, explanation of condition, ease of understanding, when to seek medical attention and overall preference. Participants were also able to provide free-text comments.
Patients were recruited from ENT out-patient clinics, where they were invited to voluntarily participate in the survey. Participants received a pair of leaflets alongside the questionnaire, and were blinded to which leaflet was AI-generated and which was human-generated. Data were collected over two six-week periods in the ENT out-patient department. The first data collection period was conducted in August to September 2023, and the second period of data collection was held in January to February 2024. We excluded incomplete surveys that did not have the preference section completed.
The statistical significance of participant preferences was evaluated using a chi-square goodness-of-fit test.
The readability of each leaflet was assessed using the Flesch–Kincaid Readability Ease Score and Grade Level.Reference Kincaid16 These tools analyse sentence length and word complexity to grade pieces of writing according to their ease of reading and the required education level to understand them. The Flesch–Kincaid Readability Ease Score grades the readability of a text using a scale from 1 to 100, with higher scores indicating easier readability. A score between 60 and 80 is generally considered appropriate for a broad audience because it balances ease of reading with sufficient complexity for adult readers.Reference Flesch17 The Flesch–Kincaid Grade Level score estimates the US school grade level required to understand the piece of writing. Our targeted reading age of 14 years corresponds to the eighth or nineth grade in the US schooling system.
No ethical approval was sought, following advice from the National Health Service Health Research Authority (HRA) and Medical Research Council (MRC) Regulatory Support Centre tool.18 The survey was registered as a quality improvement project after departmental review.
Results
A total of 111 responses were deemed eligible for inclusion. One response was deemed ineligible as the preference section had not been completed.
In terms of appearance, on a Likert scale from 1 to 10, the AI-adapted leaflets were graded with a mean score of 7.03 (standard deviation (SD) = 2.17) and the existing human-generated leaflets were allocated a mean score of 6.97 (SD = 2.32). The difference in Likert rating was not statistically significant (p = 0.99).
Overall, 41.3 per cent of participants (N = 43) stated they did not have any preference in terms of the overall appearance of the leaflets, while 29.8 per cent (N = 31) preferred the human-generated leaflets and 28.8 per cent (N = 30) preferred the AI-adapted leaflets (p = 0.22).
Regarding the explanation of the condition or procedure, 53.3 per cent of participants (N = 56) had no preference on the information provided, 29.5 per cent (N = 31) preferred the human-generated leaflets and 17.1 per cent (N = 18) preferred the AI-adapted leaflets (p ≤ 0.0001).
Both categories of leaflet were deemed equally easy to read by 43.4 per cent of participants (N = 46), 31.1 per cent (N = 33) found the human-generated leaflets easier to read and 25.5 per cent (N = 27) preferred the AI-adapted leaflets (p = 0.07).
It was found that 55.0 per cent of participants (N = 60) indicated there was no difference in their understanding of when to seek medical attention between the two leaflets, 25.7 per cent (N = 28) had a clearer understanding of when to seek medical attention after reading the human-generated leaflets and 19.3 per cent (N = 21) indicated a preference for the AI-adapted leaflets (p = ≤0.0001).
When asked to indicate their overall preference between the two leaflets, 39.6 per cent of participants (N = 44) had no preference between either leaflet, 32.4 per cent (N = 36) preferred the human-generated leaflet and 27.9 per cent (N = 31) preferred the AI-adapted leaflet (p = 0.31).
The readability of each leaflet was assessed using the Flesch–Kincaid Readability Ease Score and Grade Level (Table 1). The mean average Flesch–Kincaid Readability Ease Score for the human-generated leaflets was 65.2, compared to a mean average of 63.1 for the AI-generated leaflets. This indicates a slight decrease in the readability of the patient information leaflets when adapted by AI, indicating they became slightly more difficult to read. This result was not statistically significant (p = 0.429). The AI-adapted patient information leaflet was easier to read than the human-generated counterpart for epistaxis, had similar ease of reading to the human-generated leaflet for nasal trauma, but was more difficult to read for tonsillectomy and myringotomy and grommets (Figure 1).
Readability assessment of human-generated and AI-adapted patient information leaflets
AI = artificial intelligence
Flesch–Kincaid Readability Ease Score of human-generated and artificial intelligence-adapted patient information leaflets.
Discussion
This pilot study has demonstrated feasibility of utilising AI to refine existing human-generated patient information literature. To our knowledge, this study is one of the first to assess patient satisfaction with AI-adapted patient information leaflets and to evaluate patient preference for educational content adapted by AI. We selected four existing patient information leaflets, adapted them using AI, and assessed the patient preference and readability of these pairs of leaflets.
Our findings indicated no statistically significant difference in patient preference for human-generated patient information leaflets versus AI-adapted patient information leaflets. Overall, 40 per cent of participants expressed no preference between AI-adapted and human-generated patient information leaflets, 32 per cent preferred the human-generated leaflets and 28 per cent preferred the AI-adapted leaflets.
The majority of participants indicated they had no preference between the human-generated and AI-adapted leaflets regarding appearance, clarity of explanation of the condition or procedure, ease of reading or their understanding of when to seek medical attention. These results are encouraging because they suggest patients are open to the use of AI as a tool to support patient education and improve understanding of their conditions. The results demonstrate the ability of AI to generate readable health literature based on an input of existing health literature that is acceptable to patients.
Whilst AI technologies can be powerful tools for producing and adapting health information, their output often requires human oversight for fine-tuning and adjustment of the process to optimise results. We observed an increase in complexity, indicated by a decrease in the Flesch–Kincaid Readability Ease Score, for three of the four leaflets after they were adapted by AI. This highlights the importance of human supervision in the use of AI to adapt or update patient information leaflets to ensure that content remains accessible and understandable to patients.
Recent studies show mixed results regarding the readability of AI-generated patient information, with some reporting improved readability of AI-generated contentReference Verran9 and others finding AI-generated content more difficult to read.Reference Shamil, Ko, Fan, Schuster-Bruce, Jaafar and Khwaja14 It is clear that the prompts entered into the AI tool will influence the output, therefore it may be necessarily to specifically instruct the tool to maintain a set standard of readability to address this issue. However, prioritising ease of readability of AI-generated patient information may compromise the quality of the information output,Reference Abou-Abdallah, Dar, Mahmudzade, Michaels, Talwar and Tornari13 and human oversight will be essential to achieving this balance.
Despite the decrease in readability observed for some of our patient information leaflets, there was no statistically significant difference in patient preference between the human-generated and AI-adapted leaflets. However, this study is limited by the small sample size of 111 participants and it is possible with a larger data set that there could be a difference in patient preferences.
Artificial intelligence technologies such as ChatGPT are trained on massive datasets and represent the cutting edge of natural language processing. This means they can be a powerful tool for generating or adapting health literature. However, these models are not without their limitations.
One of the key limitations of ChatGPT 3.5 is that the training data only include information available up to September 2021,Reference Kelly19 meaning that it lacks access to updates in the medical literature occurring after this time. Furthermore, the model was trained only on publicly available sources and not private data sets, therefore it does not have access to all articles published in medical journals. ChatGPT 3.5 may struggle to understand nuance and may not be adept at distinguishing between medical terminology that is understandable to health professionals versus what is appropriate for patients to understand, therefore it has the potential to use jargon that is too technical for the average patient.
As AI technology continues to advance, some of these challenges may be addressed. A newer model, ChatGPT 4.0 (https://chatgpt.com), was progressively rolled out during 2023. This is a larger, more capable model that can process both images and text, which has potential benefit in producing patient information leaflets. Claude (https://claude.ai) is an alternative publicly available AI model that has a strong focus on ethical AI usage and is trained to adopt a more cautious, conservative approach.Reference Ticong20 This could be beneficial in avoiding errors in medical information when producing patient information leaflets. The quality of AI-generated content is highly dependent on the prompts provided by the user, therefore further research is needed to evaluate what factors in the prompt lead to a higher quality output.
Whilst demonstrating the feasibility of utilising AI technology to adapt existing patient information leaflets, this study lacks the statistical power to draw definitive conclusions about patient preferences about AI-adapted content. Future research with larger sample sizes may reveal more meaningful differences in patient preferences and improve understanding of how AI-generated patient information leaflets compare to human-generated patient information leaflets in terms of patient understanding and overall satisfaction.
A further limitation of the study is that it focuses on feasibility, readability and patient satisfaction, but does not evaluate the quality and accuracy of the outputted material. The study evaluated the use of AI to adapt existing human-generated patient information leaflets and therefore further research is needed to evaluate the acceptability of content produced primarily by AI.
The potential advantages for utilising AI technology to produce and update patient information literature are vast. Whilst human oversight is essential, the ability of the technology to rapidly process large data sets and synthesise content within seconds offers a valuable opportunity to save clinicians considerable time. Amidst the current medical workforce crisis, this could help free up more clinician time for direct patient care, whilst streamlining the process of producing and updating patient information leaflets. This would improve patient access to medical information, supporting the shared decision-making process.
As the technology continues to advance and our understanding of the prompting process is refined, there is potential for scalability of AI-driven patient information leaflet production across a vast range of conditions and specialities. Furthermore, it is possible to use AI technology to produce literature appropriate for a specific reading age and language, thereby personalising the information provided to the individual patient and thus supporting patient-centred, personalised healthcare.
• This was a single-blinded study evaluating the patient preferences and readability of patient information leaflets refined and updated using artificial intelligence (AI)
• Participants indicated no significant overall preference between human-generated leaflets and leaflets adapted by AI, but there was a slight reduction in readability of the leaflets after adaptation by AI
• Human oversight is essential to the process and care must be taken to ensure leaflets remain easy to read
• There is potential for AI to make the process of producing and updating patient information more efficient and to personalise information to the individual patient
• This study demonstrates feasibility of using AI to adapt existing patient information leaflets in a way that is acceptable to patients
Conclusion
Artificial intelligence technology has found its way into clinical practice and is already used in audiology and imaging interpretation within ENT. This study provides valuable insights into the feasibility and acceptability of using AI technology to refine and adapt patient information literature. Artificial intelligence has the potential to revolutionise the process of producing and updating patient information literature in a way that is readable and acceptable to patients. Furthermore, AI offers the potential to further personalise the delivery of health information by tailoring the literature to the individual patient’s level of health literacy. However, human oversight and iterative refinement are essential in safeguarding the quality, accuracy and readability of the content produced, and further research is required to improve our understanding of the prompt inputs necessary to enable AI to produce high-quality patient information leaflets.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0022215125103757.
Competing interests
None declared.
