Introduction
Artificial intelligence (AI) applications are increasingly used across various fields and continue to develop rapidly. These advancements have brought the potential application of AI technologies in medicine and healthcare to the forefront. Numerous studies have explored the integration of AI in healthcare, covering topics such as patient education during surgical procedures, disease diagnosis through symptom analysis, patient triage and AI-assisted robotic surgery.Reference Savage, Seth, Angus and Rozen1–Reference Bir Yücel, Sutcuoglu, Yazıcı, Ozet and Ozdemir7
In terms of ENT, AI applications have been studied in various areas, including tonsillectomy, adenoidectomy, ventilation tube placement, rhinological pathologies, sleep apnea syndrome, sialendoscopy, laryngeal carcinomas and parotidectomy.Reference Abou-Abdallah, Dar, Mahmudzade, Michaels, Talwar and Tornari8–Reference Langlie, Kamrava, Pasick, Mei and Hoffer13
While the results are promising, the exact implementation and standardisation of AI in medical practice remain uncertain. Despite initial findings demonstrating the potential benefits of AI applications, several critical concerns must be addressed.Reference Marques, Almeida and Pereira14, Reference Alowais, Alghamdi, Alsuhebany, Alqahtani, Alshaya and Almohareb15 These include the inaccuracy or low quality of information provided, data privacy and security issues, the risk of over-reliance on AI, liability for AI-related errors and challenges in clinical validation. Therefore, assessing and verifying the quality of information obtained from AI applications are essential. Various standardised scales have been developed to evaluate the quality of medical information available on the Internet and are widely used today.Reference Silberg, Lundberg and Musacchio16–Reference Bernard, Langille, Hughes, Rose, Leddin and Veldhuyzen van Zanten18 However, no universally accepted scales specifically designed for assessing the quality of AI-generated medical information have yet been developed.Reference Boscolo-Rizzo, Marcuzzo, Lazzarin, Giudici, Polesel and Stellin19–Reference Saibene, Allevi, Calvo-Henriquez, Maniaci, Mayo-Yáñez and Paderno23 Vaira et al. introduced the Quality Analysis of Medical Artificial Intelligence (QAMAI) tool and proposed its use for evaluating AI-generated medical information. QAMAI consists of a concise six-item questionnaire, with each item rated on a 1–5 Likert scale, yielding a total score between 6 and 30. The tool assesses responses based on key criteria such as accuracy, clarity, relevance, completeness, provision of sources and references and usefulness (Table 1).Reference Vaira, Lechien, Abbate, Allevi, Audino and Beltramini24 Several studies have applied the QAMAI tool in different medical contexts: Frosolini et al. in maxillofacial trauma triage, Boscolo-Rizzo et al. in reconstructive surgery for head and neck carcinomas, Khaldi et al. in tracheostomy training for patients, Lim et al. in pre-operative education for stoma patients, and Saxena et al. in the prevention and management of peri-operative neurocognitive disorders.Reference Frosolini, Catarzi, Benedetti, Latini, Chisci and Franz5, Reference Boscolo-Rizzo, Marcuzzo, Lazzarin, Giudici, Polesel and Stellin19, Reference Khaldi, Machayekhi, Salvagno, Maniaci, Vaira and La Via25–Reference Saxena, Barreto Chang, Suppan, Meco, Vacas and Radtke27 As far as we know, this is the first study to evaluate five different AI applications and commonly performed 15 surgical procedures in ENT practice using the QAMAI tool.
The Quality Analysis of Medical Artificial Intelligence tool
6–11 points, Poor quality; 12–17 points, Fair quality; 18–23 points, Good quality; 24–29 points, Very good quality; 30 points, Excellent quality.
In this study, we aimed to assess the quality of information provided by AI applications on ENT surgical procedures and examine its usability for patient education.
Materials and methods
Study design
Chat Generative Pre-trained Transformer 4.0 (ChatGPT 4.0) (OpenAI, California, USA), GEMINI 1.5 Flash (Google Artificial Intelligence, California, USA), Copilot (Microsoft, New Mexico, USA), Claude 3.5 Sonnet (Anthropic, California, USA) and DeepSeek-R1 (DeepSeek, Zhejiang, PRC) AI applications were queried to provide patient-oriented information on 15 commonly performed surgical procedures in ENT practice (Table 2). The AI applications were instructed to generate detailed responses covering key aspects such as surgical techniques, procedure duration, pre-operative preparation, intra-operative steps and post-operative recovery. The responses were required to be comprehensible to patients and to include at least four to five medical references, with citations numbered accordingly in the text (Supplementary material). Each of the 15 surgical procedures was queried separately for each AI application. To assess whether AI applications could provide consistent and repeatable responses, the questions were asked a total of three times at 3-day intervals, with search histories cleared before each session (27 January 2025; 30 January 2025; and 2 February 2025). Responses were evaluated using the QAMAI tool by two otolaryngologists affiliated with different institutions. One possessed six years of clinical experience following completion of specialty training in a tertiary hospital and held certification from the European Board Examination of Otorhinolaryngology, whereas the other had four years of post-specialisation experience. Any discrepancies in scoring were discussed, and the agreed-upon score was considered final. Statistical analysis was conducted based on the three sets of evaluations.
ENT surgical procedures list
Statistical analysis
Statistical analyses were performed using the Statistical Package for the Social Sciences (SPSS) for Windows (version 30.0; IBM Corp., Armonk, NY, USA). The accuracy, clarity, relevance, completeness, referencing, usefulness and total QAMAI scores for AI-generated responses were reported as means and standard deviations. The Kruskal–Wallis test was used for non-parametric data or data that did not follow a normal distribution. Intraclass correlation coefficient (ICC) analysis was performed to evaluate the reliability of AI-generated scores across different time points. A 95 per cent confidence interval (CI) was applied. An ICC value greater than 0.75 indicated strong reliability, whereas a value between 0.50 and 0.75 was interpreted as moderate reliability. A significance level of p less than 0.05 was considered statistically significant.
Ethics statement
This study was conducted in accordance with the principles of the Declaration of Helsinki. Ethics committee approval was not required, as the study did not involve human or animal subjects.
Results and analysis
In this study, 15 commonly performed surgical procedures were queried in AI applications a total of three times at 3-day intervals, and the responses were evaluated using the QAMAI tool. The mean total scores from the three evaluations were as follows: 27.56 ± 1.20 for ChatGPT 4.0, 26.24 ± 1.26 for GEMINI 1.5 Flash, 26.84 ± 1.35 for Copilot, 28.24 ± 0.77 for Claude 3.5 Sonnet and 28.13 ± 0.84 for DeepSeek-R1. Among the evaluated criteria, ChatGPT 4.0, Copilot, Claude 3.5 Sonnet and DeepSeek-R1 received the lowest scores in the “Completeness” category, whereas GEMINI 1.5 Flash achieved the lowest scores in the “Provision of Sources and References” category. The highest scores across all AI applications were observed in the “Clarity” and “Relevance” categories (Table 3).
Mean scores of three different evaluations of AI applications based on QAMAI items
Abbreviation: QAMAI, Quality Analysis of Medical Artificial Intelligence.
Statistical analysis revealed a significant difference between the AI applications in terms of response quality (p < 0.001). The scores obtained from GEMINI 1.5 Flash were significantly lower than those from ChatGPT 4.0, Claude 3.5 Sonnet and DeepSeek-R1. Similarly, the scores from Copilot were significantly lower than those from ChatGPT 4.0, Claude 3.5 Sonnet and DeepSeek-R1. Conversely, the scores obtained from Claude 3.5 Sonnet were significantly higher than those from ChatGPT 4.0, Copilot and GEMINI 1.5 Flash (Table 4).
Surgical procedure-specific and total mean scores of three different evaluations of AI applications based on QAMAI items
a Kruskal–Wallis test, p < 0.05.
Abbreviation: AI, artificial intelligence; QAMAI, Quality Analysis of Medical Artificial Intelligence.
The consistency of responses across the three time points was assessed using ICC. The agreement of mean scores was found to be 0.955 (95 per cent CI: 0.893–0.984) for ChatGPT 4.0, 0.936 (95 per cent CI: 0.847–0.977) for GEMINI 1.5 Flash, 0.968 (95 per cent CI: 0.924–0.988) for Copilot, 0.977 (95 per cent CI: 0.942–0.993) for Claude 3.5 Sonnet and 0.970 (95 per cent CI: 0.928–0.990) for DeepSeek-R1 (p < 0.001 for all; Table 5).
Inter-rater reliability (intraclass correlation coefficient)
* Measurement of each surgical procedure.
** Measurement of the average of surgical procedures.
Discussion
In this study, we questioned five AI applications about 15 common ENT surgical procedures and assessed their responses using the QAMAI tool. The ranking of average total scores across the three evaluations, from lowest to highest, was as follows: GEMINI 1.5 Flash (26.24 ± 1.26), Copilot (26.84 ± 1.35), ChatGPT 4.0 (27.56 ± 1.20), DeepSeek-R1 (28.13 ± 0.84) and Claude 3.5 Sonnet (28.24 ± 0.77). The score difference between Claude 3.5 Sonnet, which received the highest score, and ChatGPT 4.0, GEMINI 1.5 Flash and Copilot was statistically significant. However, the score difference between Claude 3.5 Sonnet and DeepSeek-R1 was not statistically significant. Additionally, the scores obtained from GEMINI 1.5 Flash were significantly lower than those of ChatGPT 4.0, Claude 3.5 Sonnet and DeepSeek-R1.
There is no consensus in the literature on whether AI applications are reliable in providing medical information and which AI application is the most accurate or reliable. Ostrowska et al. in a study comparing ChatGPT 3.5, ChatGPT 4.0 and Bard AI for laryngeal cancer information, identified ChatGPT 3.5 as the most reliable AI application.Reference Ostrowska, Kacała, Onolememen, Vaughan-Lane, Sisily Joseph and Ostrowski11 Similarly, Rahsepar et al. in a study evaluating commonly used terminology in lung cancer prevention, screening and radiology reports, found that, although ChatGPT 3.5 had higher accuracy than other tools, none of the AI models including ChatGPT 3.5, Google Bard, Bing and Google Search could answer all questions correctly and with full consistency.Reference Rahsepar, Tavakoli, Kim, Hassani, Abtin and Bedayat21 In another study involving 600 ophthalmology-related questions, Bahir et al. reported correct response rates of 66 per cent for GEMINI Advanced, 62 per cent for ChatGPT 4.0, 58 per cent for GEMINI and 46 per cent for ChatGPT 3.5.Reference Bahir, Zur, Attal, Nujeidat, Knaanie and Pikkel28 Furthermore, Zhao et al. found that Claude 3 and ChatGPT 4.0 outperformed GEMINI in providing precise and comprehensive information on uveitis-related questions.Reference Zhao, He, Liang, Cen, Wang and Lin29 In their study evaluating the recommendations of ChatGPT 4.0 and GEMINI against published guidelines for the prevention and treatment of peri-operative neurocognitive disorders, Saxena et al. reported that the highest level of compliance was observed in the recommendations generated by ChatGPT 4.0.Reference Saxena, Barreto Chang, Suppan, Meco, Vacas and Radtke27 However, they noted that neither AI application achieved full compliance with the guidelines. Although ChatGPT-derived AI applications generally achieve the highest scores in the literature, we found that Claude 3.5 Sonnet had the highest score in this study, with a statistically significant difference from all other applications except DeepSeek-R1.
Several studies have assessed the quality of AI-generated medical information using the QAMAI tool. In their study on the management of vestibular schwannomas using the QAMAI tool, Borsetto et al. reported that ChatGPT 4.0 demonstrated statistically significantly better performance compared to Claude 2 in the categories of “Accuracy,” “Clarity,” “Completeness,” “Relevance,” and “Usefulness.”Reference Borsetto, Sia, Axon, Donnelly, Tysome and Anschuetz30 However, no statistically significant difference was found in the “Sources” category. Boscolo-Rizzo et al. found that ChatGPT 4.0 scored higher than Claude 2 overall in their study of the QAMAI tool for reconstructive surgery for head and neck cancer, but Claude 2 outperformed ChatGPT 4.0 in the “References.”Reference Boscolo-Rizzo, Marcuzzo, Lazzarin, Giudici, Polesel and Stellin19 In another study, Lim et al. evaluated the effectiveness of AI applications in providing information to peri-operative stoma patients and reported that QAMAI scores were 18 for ChatGPT 3.5, 20 for ChatGPT 4.0, 22 for Copilot and 19 for GEMINI.Reference Lim, Lirios, Sakalkale, Satheakeerthy, Hayes and Yeung26 AI applications received the lowest scores in the “Provision of Sources and References” category. Khaldi et al. in their study on tracheostomy-related AI responses, showed that ChatGPT 4.0 achieved total QAMAI scores of 22.85 ± 4.75 and 21.45 ± 3.95, with the highest scores in “Clarity” and “Completeness of Explanations”, while the lowest scores were observed in “Accuracy of Information” and “Referencing”.Reference Khaldi, Machayekhi, Salvagno, Maniaci, Vaira and La Via25 In this study, Claude 3.5 Sonnet’s scores were significantly higher than those of all other AI applications except DeepSeek-R1 (Table 4). AI applications achieved the highest scores in the “Clarity” and “Relevance” categories. ChatGPT 4.0, Copilot, Claude 3.5 Sonnet and DeepSeek-R1 received the lowest scores in “Completeness,” while GEMINI 1.5 Flash had the lowest score in “Provision of Sources and References.”
Many studies have examined the accuracy of references provided by AI applications. Vaira et al. reported that bibliographic references generated by ChatGPT 4.0 were of poor quality, with 46.4 per cent of responses lacking references.Reference Vaira, Lechien, Abbate, Allevi, Audino and Beltramini31 Furthermore, Lechien et al. noted that ChatGPT sometimes fabricated scientific references.Reference Lechien and Rameau32 However, Langlie et al. did not find significant errors in AI-generated references.Reference Langlie, Kamrava, Pasick, Mei and Hoffer13 Saxena et al. similarly found that AI applications scored the lowest in the “Provision of Sources” category according to the QAMAI tool.Reference Saxena, Barreto Chang, Suppan, Meco, Vacas and Radtke27 In this study, with the exception of GEMINI 1.5 Flash and Copilot, AI applications generally performed well in “Provision of Sources and References.” GEMINI 1.5 Flash did not number its references within the text, while Copilot cited general health websites rather than academic publications. Consequently, these two AI models received lower scores in this category. Notably, none of the AI applications hallucinated references. The high scores observed in “Provision of Sources and References” may be attributed to the explicit instruction requiring AI applications to include at least four to five numbered medical references in their responses. Additionally, AI applications referenced three to nine sources per response, which likely reduced the occurrence of fabricated citations.
The quality of AI-generated medical information regarding surgical procedures has been analysed in various studies. Abou-Abdallah et al. reported that ChatGPT-generated information on tonsillectomy, adenoidectomy and grommet tube insertion was of lower quality and readability than ENT United Kingdom patient information leaflets.Reference Abou-Abdallah, Dar, Mahmudzade, Michaels, Talwar and Tornari8 Conversely, Varia et al. found that ChatGPT 4.0–generated informed consent documents for 10 oral surgery procedures were significantly superior to those produced by Bard GEMINI Advanced and surgical residents.Reference Vaira, Lechien, Maniaci, Tanda, Abbate and Allevi33 Decker et al. stated that ChatGPT 3.5-generated consent forms for colectomy, coronary artery bypass grafting, laparoscopic cholecystectomy, inguinal hernia repair, knee arthroplasty and spinal fusion surgery were more readable, complete and accurate than those created by surgeons.Reference Decker, Trang, Ramirez, Colley, Pierce and Coleman34 Patel et al. noted that ChatGPT-generated informed consent forms for five plastic surgery procedures were significantly shorter and more readable than those of the American Society of Plastic Surgeons (ASPS), with no significant differences in completeness or accuracy.Reference Patel, Om, Cuzzone and Garcia Nores35 Langlie et al. reported that ChatGPT 3.5 provided accurate answers regarding intervention needs, management recommendations and treatment alternatives for procedures such as adenoidectomy, tonsillectomy, tympanoplasty, endoscopic sinus surgery, parotidectomy and total laryngectomy, but it lacked sufficient details on risks and recovery for complex procedures.Reference Langlie, Kamrava, Pasick, Mei and Hoffer13 Compared to previous studies, this study is more comprehensive, as it evaluates 15 ENT surgical procedures across five AI applications. AI-generated responses were generally accurate, reliable and comprehensive, but they lacked details on some current surgical techniques and devices (Supplementary material). Notably, AI applications did not mention endoscopic techniques in procedures 1–5, preservation rhinoplasty in procedure 6, or transoral/transaxillary thyroidectomy in procedure 13.
One of the key issues discussed is the stability and consistency of responses generated by AI applications. Beaulieu-Jones et al. identified inconsistencies in repeated responses from ChatGPT 4.0, while Liang et al. reported similar inconsistencies in responses from both ChatGPT 3.5 and 4.0. Lechien et al. classified the stability of ChatGPT 3.5 as mild and that of ChatGPT 4.0 as moderate in repeated responses.Reference Beaulieu-Jones, Berrigan, Shah, Marwaha, Lai and Brat36–Reference Lechien, Briganti and Vaira38 In contrast, Boscolo-Rizzo et al. and Khaldi et al. reported high response stability in their studies.Reference Boscolo-Rizzo, Marcuzzo, Lazzarin, Giudici, Polesel and Stellin19, Reference Khaldi, Machayekhi, Salvagno, Maniaci, Vaira and La Via25 In this study, we found that the stability of the average repeated responses for each surgical procedure across all AI applications was high (Table 5).
In the near future, AI applications may serve as supportive tools in the development of patient education materials, provided they are used under expert supervision. These tools have the potential to help bridge the health information gap, particularly for patients seeking accessible and understandable information outside of clinical visits. However, before recommending the routine use of AI applications in patient communication or decision-making support, further validation, regulation and integration with clinician oversight will be essential.
There are several limitations to this study. First, not all surgical procedures related to ENT were evaluated. Second, AI applications were queried regarding standard surgical procedures; however, in real life, a patient’s medical condition may vary, and the execution of surgical procedures and AI responses may differ accordingly. Third, the study was conducted using responses obtained from AI applications on 27 January 2025, 30 January 2025 and 2 February 2025. Given the rapid advancements in AI technology, the quality of responses may evolve over time. Fourth, the evaluation was limited to only five AI applications, despite the rapidly increasing number of such tools. Additionally, a limitation of this study is that the readability of AI-generated responses was not assessed. The strength of this study, compared to previous literature, is the evaluation of a greater number of surgical procedures using a larger set of AI applications, with queries conducted three times to ensure consistency.
What is known:
• Artificial intelligence (AI) applications are increasingly used in healthcare, including ENT surgery
• The Quality Analysis of Medical Artificial Intelligence (QAMAI) tool has been developed to assess the quality of AI-generated medical information based on six core criteria
• Existing studies using QAMAI have evaluated a limited number of procedures or AI tools, often reporting low scores in completeness and referencing
What this paper adds:
• This study is the first to evaluate five advanced AI tools (ChatGPT, GEMINI 1.5 Flash, Copilot, Claude 3.5 Sonnet and DeepSeek-R1) across 15 commonly performed ENT surgical procedures using the QAMAI scale
• Claude 3.5 Sonnet achieved the highest overall score, followed closely by DeepSeek-R1; both significantly outperformed others in most categories
• AI-generated content was generally accurate, relevant and stable across repeated responses, but lacked detailed information on newer surgical techniques
• Unlike previous studies, relatively high scores were observed in the “Provision of Sources and References” category, possibly due to specific prompting
• The findings support the potential use of AI in patient education under professional oversight, though further validation is still needed
Conclusion
AI has the potential to provide patients with accurate and consistent information about ENT surgeries, yet differences in QAMAI scores show that information quality varies between platforms.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0022215125103368.
Funding
The authors did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Competing interests
The authors declared no potential conflict of interest with the respect of the research, authorship, and/or publication of this article.
