Significant outcomes
-
• ChatGPT-4 provided responses of moderate-to-good overall quality to frequently asked patient questions about rTMS, indicating its potential as a supportive patient education tool when complemented by guidance from healthcare professionals, particularly rTMS specialists.
-
• Although content quality was acceptable, the reliability and evidence-based accuracy of the responses were poor, underscoring the need for clinical verification prior to use.
-
• The readability of the responses exceeded levels generally recommended for patient-oriented health information, which may limit comprehension among patients with lower health literacy.
Limitations
-
• The evaluation was based on 25 selected frequently asked questions, assessed only in English using a single ChatGPT version, without follow-up queries, real-world user interaction, or personalisation, which may not represent the real-world patient questions.
-
• Scoring relied on subjective expert judgement using the GQS and DISCERN tools, which may vary between raters, and although all reviewers were experienced psychiatrists, only one had specific expertise in rTMS.
-
• Although the latest version of ChatGPT available at the time was used, LLMs and their responses are constantly evolving, and the results reflect a specific period; response quality may also vary across languages and cultural contexts, limiting the generalisability of the findings.
Introduction
People often use common online resources and search engines to access medical information (Battineni et al., Reference Battineni, Baldoni, Chintalapudi, Sagaro, Pallotta, Nittari and Amenta2020), and the role of the internet in providing mental health information to patients is not a new topic (Powell & Clarke, Reference Powell and Clarke2006; Kalckreuth et al., Reference Kalckreuth, Trefflich and Rummel-Kluge2014; Montagni et al., Reference Montagni, Donisi, Tedeschi, Parizot, Motrico and Horgan2016). Already two decades ago, internet users were seeking mental health-related information online (Powell & Clarke, Reference Powell and Clarke2006). Kalckreuth et al. (Reference Kalckreuth, Trefflich and Rummel-Kluge2014) reported that 70.9% of internet-using psychiatric patients used the internet for mental health-related purposes, most commonly to seek information on mental disorders, followed by information on medications and searches for mental health services, while platforms involving other patients or mental health professionals were accessed less frequently. In the study by Montagni et al. (Reference Montagni, Donisi, Tedeschi, Parizot, Motrico and Horgan2016), 49.8% of the surveyed university students used the internet to seek mental health information and support, while 42.9% had searched for general information on mental health problems, most commonly related to stress, depression, and anxiety. Psychiatric patients can face long wait times before obtaining proper diagnosis and treatment (Sun et al., Reference Sun, Correll, Trestman, Lin, Xie, Hankey, Uymatiao, Patel, Metsutnan, McDaid, Saha, Kuo, Lewis, Bhatt, Lipphard and Kablinger2023), so over 20% of individuals with mental health problems (and over 10% of the general population) seek medical information themselves (Powell & Clarke, Reference Powell and Clarke2006). Furthermore, many patients with major depressive disorder (MDD) do not receive sufficient care (Thornicroft et al., Reference Thornicroft, Chatterji, Evans-Lacko, Gruber, Sampson, Aguilar-Gaxiola, Al-Hamzawi, Alonso, Andrade, Borges, Bruffaerts, Bunting, De Almeida, Florescu, De Girolamo, Gureje, Haro, He, Hinkov, Karam, Kawakami, Lee, Navarro-Mateu, Piazza, Posada-Villa, De Galvis and Kessler2017). Although this treatment gap has multifactorial causes, it is partly due to the shortage of mental health care professionals (Kohn et al., Reference Kohn, Saxena, Levav and Saraceno2004). Therefore, digital mental health interventions are presented as an approach to address the global mental health crisis (Rudd & Beidas, Reference Rudd and Beidas2020). Individuals awaiting mental health services have shown interest in digital mental health interventions providing information about therapies, mental illnesses, and access to treatment (Buck et al., Reference Buck, Kadakia, Larsen, Tauscher, Guler and Ben-Zeev2025), with such interventions available in different formats, including online web-based and chatbot-based approaches (Kruzan et al., Reference Kruzan, Fitzsimmons-Craft, Dobias, Schleider and Pratap2022).
Individuals using large language models (LLMs) for mental health purposes have reported barriers to traditional mental health care, including treatment costs, insurance coverage, and provider availability, while convenience and real-time availability were among the factors associated with their use (Stade et al., Reference Stade, Tait, Campione, Wiltsey Stirman and Eichstaedt2026). The potential relevance of LLMs to patient education lies in their accessibility as a source of psychoeducational information that may help to improve patient education and complement professional mental health care.
With the increasing popularity of artificial intelligence (AI), a new field of research has emerged that is studying the use of AI in medicine, including psychiatry (Pham et al., Reference Pham, Nabizadeh and Selek2022; Haug & Drazen, Reference Haug and Drazen2023), where LLMs may help to improve patient education and effective mental health education. The rapid expansion of LLMs such as ChatGPT has led to an increase in studies aiming to verify the accuracy of the information they provide (Ayers et al., Reference Ayers, Poliak, Dredze, Leas, Zhu, Kelley, Faix, Goodman, Longhurst, Hogarth and Smith2023). In particular, healthcare professionals need to understand how LLMs can be used for patient education and how they are used by patients. To date, studies have examined the quality of ChatGPT responses to questions about dementia (Aguirre et al., Reference Aguirre, Hilsabeck, Smith, Xie, He, Wang and Zou2024; Saeidnia et al., Reference Saeidnia, Kozak, Lund and Hassanzadeh2024), bipolar disorders (Parker & Spoelma, Reference Parker and Spoelma2024), insomnia (Alapati et al., Reference Alapati, Campbell, Molin, Creighton, Wei, Boon and Huntley2024), electroconvulsive therapy (Lundin et al., Reference Lundin, Berk and Østergaard2023), and autism spectrum disorders (Almulla & Khasawneh, Reference Almulla and Khasawneh2025), but to our knowledge, so far none has examined ChatGPT responses to questions about repetitive transcranial magnetic stimulation (rTMS).
First introduced in 1985, transcranial magnetic stimulation (TMS) is a noninvasive method for stimulating the brain, and its form, rTMS delivers repeated single magnetic pulses of the same intensity (Barker et al., Reference Barker, Jalinous and Freeston1985; Wassermann et al., Reference Wassermann, Grafman, Berry, Hollnagel, Wild, Clark and Hallett1996). Major evidence-based treatment guidelines and current literature on the therapeutic use of rTMS recommend it as a treatment option for MDD, including treatment-resistant depression (Bundesärztekammer et al., 2022; Trapp et al., Reference Trapp, Purgianto, Taylor, Singh, Oberman, Mickey, Youssef, Solzbacher, Zebley, Cabrera, Conroy, Cristancho, Richards, Flood, Barbour, Blumberger, Taylor, Feifel, Reti, McClintock, Lisanby and Husain2025). In the United States, rTMS is approved by the Food and Drug Administration for the treatment of MDD, obsessive–compulsive disorder (OCD), and deep TMS for the treatment of smoking cessation (US Food & Drug Administration 2008, 2018, 2020; Cohen et al., Reference Cohen, Bikson, Badran and George2022), while according to the German guidelines (Deutsche Gesellschaft für Psychiatrie und Psychotherapie, Psychosomatik und Nervenheilkunde [DGPPN], 2025), low-frequency rTMS (1 Hz) should be considered for persistent auditory hallucinations, and high-frequency rTMS (10/20 Hz) may be considered for negative symptoms of schizophrenia.
According to the German Guideline for Unipolar Depression (Bundesärztekammer et al., 2022), augmentation with rTMS may be considered in patients with MDD who show an inadequate response to antidepressant monotherapy and should be considered in patients with treatment-resistant depressive episodes. In the German Guideline for OCD (Voderholzer & Favreau, Reference Voderholzer and Favreau2022), rTMS may be considered in patients with an inadequate response to first-line treatments, with the aim of short-term symptom relief. According to the National Institute for Health and Care Excellence (NICE) Interventional Procedures Guidance, rTMS is indicated for major depression, including treatment-resistant depression, whereas its use for OCD should only be used in the context of research (NICE, 2015; NICE, 2020). Beyond guideline recommendations, studies also show possible effects of rTMS in post-traumatic stress disorder (PTSD) (Lefaucheur et al., Reference Lefaucheur, Aleman, Baeken, Benninger, Brunelin, Di Lazzaro, Filipović, Grefkes, Hasan, Hummel, Jääskeläinen, Langguth, Leocani, Londero, Nardone, Nguyen, Nyffeler, Oliveira-Maia, Oliviero, Padberg, Palm, Paulus, Poulet, Quartarone, Rachid, Rektorová, Rossi, Sahlsten, Schecklmann, Szekely and Ziemann2020; Harris & Reece, Reference Harris and Reece2021; Brown et al., Reference Brown, Kweon, Sharma, Siddiqi and Ressler2025).
Patient-centred shared decision-making is an essential component of modern medicine because it promotes patient involvement, improves treatment adherence, enhances treatment satisfaction, and leads to more individualised, effective care, improved affective cognitive outcomes, and less decisional conflict (Shay & Lafata, Reference Shay and Lafata2015; Williams et al., Reference Williams, Fleming and Doubleday2017; Birkeland et al., Reference Birkeland, Bismark, Barry and Möller2022). Therefore, providing patients with clear and accurate information is necessary to increase treatment acceptance, improve understanding of procedures, help them develop realistic expectations, reduce anxiety, and encourage active participation in the treatment process by helping patients to make informed decisions. Patient education is particularly necessary for those with the option to be treated by rTMS because despite increased awareness about the technique, significant knowledge gaps and false beliefs remain widespread (Morrison et al., Reference Morrison, Uusberg, Ryan, Goldenberg, Etkin and Gross2022). Barriers to timely access to mental health care, and growing tendency to seek health information online, together with the increasingly widespread use of ChatGPT to search for information (Loh, Reference Loh2024; Iqbal et al., Reference Iqbal, Tanweer, Rahmanti, Greenfield, Lee and Li2025), make it even more important to ensure the accuracy of psychoeducational responses so that patients are not misinformed.
This study aimed to evaluate the quality, reliability, accuracy, and readability of information provided by ChatGPT regarding frequently asked patient questions about rTMS.
Methods
A comprehensive review of frequently asked questions about rTMS was carried out using patient information pages from respected and well-known healthcare institutions, including institutions listed in the SCImago Institutions Rankings for Psychiatry and Mental Health (https://www.scimagoir.com/rankings.php), that provided patient information sections containing FAQs (frequently asked questions) about TMS, as well as websites of medical associations, online patient manuals, and information materials. Frequently asked questions about rTMS were compiled from online information available on numerous patient support pages of healthcare institutions and medical associations and in patient manuals, flyers, and patient information sources provided in guidelines (see Supplementary Material for a list of the websites containing the questions). First, three psychiatry specialists (C.T., G.G, M.K.), one of whom (M.K.) is a TMS expert, who is an experienced neuropsychiatrist, Deutsche Gesellschaft für Hirnstimulation in der Psychiatrie e.V. certified in non-invasive and non-convulsive transcranial brain stimulation methods, leads a brain stimulation unit, and has over 15 years of clinical and academic experience including high volume experience with TMS, performed a preliminary evaluation of the questions by drawing upon their clinical experience and knowledge, taking into account both clinical relevance of the questions and their recurrence in the FAQ sections of the sources and real-world patient physician interactions. Based on this evaluation, the reviewers reached a consensus, with the final decision made by M.K., an expert on TMS, and the final 25 most common questions about rTMS were curated. The questions were structured to reflect natural language used by patients in clinical practice, with each question referring to rTMS. These questions were queried separately on June 23, 2025, using the most recent freely accessible version of ChatGPT available at the time, GPT-4 (OpenAI, San Francisco, CA, USA). The public interface was accessed without signing into a user account, thereby excluding access to account-specific features such as prior conversations history, stored memory, or personalised setting. A new chat session was used for each individual question to minimise any potential influence of previous prompts or generated responses on subsequent outputs. Each question was submitted once, and no follow-up questions or prompts, repeated submissions, or response regenerations were used. This approach was intended to reflect a single real-world patient interaction rather than repeated testing across sessions. All interactions with ChatGPT were conducted in English. The 25 responses were independently evaluated by the three psychiatry specialists by using an evidence-based approach, in light of the current literature, including major relevant clinical guidelines, clinical recommendations, and consensus statements, and based on the specialists’ clinical experience. Each specialist was blinded to the others’ assessments. As in similar studies (Liu-Zarzuela et al., Reference Liu-Zarzuela, Mallya, Munoz and Grayson2023; Tirumala et al., Reference Tirumala, Mishra, Trivedi, Shivakumar, Singh and Shariff2024) for each response the quality and usefulness was independently evaluated with the Global Quality Scale (GQS) (Bernard et al., Reference Bernard, Langille, Hughes, Rose, Leddin and Veldhuyzen Van Zanten2007; Yeung, Reference Yeung2025) (a Likert scale ranging from 1 [poor quality] to 5 [excellent quality]; see Table 1) and the reliability, accuracy, and quality, with the commonly used, validated DISCERN (Charnock et al., Reference Charnock, Shepperd, Needham and Gann1999) tool (Rees et al., Reference Rees, Ford and Sheard2002; Charnock & Shepperd, Reference Charnock and Shepperd2004). The DISCERN is a validated questionnaire for assessing the accuracy and relevance of written medical information (Charnock et al., Reference Charnock, Shepperd, Needham and Gann1999). It was originally employed for written medical information, and now it is largely used to assess ChatGPT questions (Artioli et al., Reference Artioli, Veronesi, Mazzotti, Brogini, Zielli, Giavaresi and Faldini2025; Kalem et al., Reference Kalem, Balaban, Savran, Kocaoğlu and Şahin2026). The DISCERN tool comprises 16 questions divided into three sections: Introduction evaluates reliability; Methods, the quality of treatment options; and Results, overall quality. Each question is rated on a scale of 1 (poor) to 5 (excellent; see Table 2). The final score is divided into Excellent (63–75 points), Good (51–62 points), Fair (39–50 points), Poor (27–38 points), and Very Poor (16–26 points) (Artioli et al., Reference Artioli, Veronesi, Mazzotti, Brogini, Zielli, Giavaresi and Faldini2025) (Table 2).
Global quality score
Table 1. Long description
A table with two columns and five rows. The first column is labeled ‘Global score’ and the second column is labeled ‘Global score description’. Row 1: Global score 1, Global score description Poor quality, poor flow of the site, most information missing, not at all useful for patients. Row 2: Global score 2, Global score description Generally poor quality and poor flow, some information listed, but many important topics missing, of very limited use to patients. Row 3: Global score 3, Global score description Moderate quality, suboptimal flow, some important information is adequately discussed, but others poorly discussed, somewhat useful for patients. Row 4: Global score 4, Global score description Good quality and generally good flow, most of the relevant information is listed, but some topics not covered, useful for patients. Row 5: Global score 5, Global score description Excellent quality and excellent flow, very useful for patients.
DISCERN scores with corresponding values
Table 2. Long description
A table categorizing DISCERN scores into five quality levels. The table has two columns: Score Range and Quality Level. It contains five rows. Row 1: Score Range, 63-75; Quality Level, Excellent. Row 2: Score Range, 51-62; Quality Level, Good. Row 3: Score Range, 39-50; Quality Level, Fair. Row 4: Score Range, 28-38; Quality Level, Poor. Row 5: Score Range, 16-27; Quality Level, Very Poor.
We assessed information transparency with the four Journal of the American Medical Association (JAMA) Benchmark Criteria: authorship, attribution, disclosure, and currency as in similar studies (Strauss, Reference Strauss1997; Kalem et al., Reference Kalem, Balaban, Savran, Kocaoğlu and Şahin2026). Furthermore, we calculated the readability level of the answers with the Flesch-Kincaid Grade Level score and the scores on the Gunning Fog Index, Coleman-Liau Index, and Simple Measure of Gobbledygook Index. These assessment tools help to rate the readability of a text and the years of education needed to comprehend it.
This study did not involve human participants, animals, or protected data, so it did not require ethical approval.
Statistical analysis
Intraclass correlation coefficients (ICC) were calculated with Microsoft Excel (Microsoft Corp., Redmond, USA) to evaluate the inter-rater reliability of the three reviewers. The ICCs measure the reliability of ratings or measurements within groups, and values range from 0 to 1, with values closer to 1 indicating higher reliability (Koo & Li, Reference Koo and Li2016). Mean and median values were calculated for all reviewers’ evaluations.
Results
Table 3 presents the 25 frequently asked patient questions and corresponding reviewer evaluations. Supplementary Material presents the full responses. ChatGPT responses begin with a brief text, followed by a detailed explanation. Nine (36%) responses suggested contacting a doctor or other healthcare professional. The mean word count of the provided answers was 149.2 (range, 69–251 words).
Ratings of responses to common questions regarding rTMS, DISCERN scores, and GQS
Table 3. Long description
A table with 25 rows and 7 columns. The columns are labeled as Question no, DISCERN R1, DISCERN R2, DISCERN R3, GQS R1, GQS R2, and GQS R3. The rows are labeled with question numbers from Q1 to Q25. Each row contains numerical values corresponding to the ratings for each question. The table captures the evaluations of reviewer responses to common questions about repetitive Transcranial Magnetic Stimulation (rTMS).
GQS, Global Quality Scale; DISCERN, DISCERN score; R1, Reviewer-1; R2, Reviewer-2; R3, Reviewer-3; rTMS, repetitive transcranial magnetic stimulation.
The mean of the four readability scores (Flesch-Kincaid Grade Level, Gunning Fog Index, Coleman-Liau Index, and Simple Measure of Gobbledygook Index) found that a grade level of 10.98, that is, 11 years of education, was necessary to understand the ChatGPT responses. Details of the readability scores are shown in Table 4.
Readability assessment tool
Table 4. Long description
A table comparing readability scores for text. The table has four rows and two columns. The columns are labeled ‘Readability assessment tool’ and ‘Score’. The rows are labeled with different readability assessment tools and their corresponding scores. Row 1: Flesch-Kincaid grade level, 8.74. Row 2: Gunning Fog Index, 12.16. Row 3: Coleman-Liau Index, 11.83. Row 4: Simple Measure of Gobbledygook Index, 11.22.
The 25 ChatGPT responses were rated by Reviewers 1 [C.T.], 2 [G.G.], and 3 [M.K.], respectively, as shown in Table 3 (Reviewer 3 was an experienced clinician from our team as TMS expert). Questions 1, 9, and 13 were evaluated with equal GQS by all three reviewers. Median and mean GQS and DISCERN scores for all reviewers are shown in Table 5.
DISCERN and GQS values of reviewers
Table 5. Long description
A table comparing DISCERN and GQS values of reviewers. The table has 10 rows and 2 columns. Row 1: DISCERN Mean for R1, 37.88 (poor). Row 2: DISCERN Mean for R2, 31.56 (poor). Row 3: DISCERN Mean for R3 TMS expert, 36.56 (poor). Row 4: DISCERN Median Value R1, 38. Row 5: DISCERN Median Value R2, 34. Row 6: DISCERN Median Value R3 (TMS expert), 37. Row 7: GQS Mean for R1, 4.16. Row 8: GQS Mean for R2, 3.04. Row 9: GQS Mean for R3 (TMS expert), 3.88. Row 10: GQS Median Value R1, 4. Row 11: GQS Median Value R2, 3. Row 12: GQS Median Value R3 (TMS expert), 4.
GQS, Global Quality Scale; TMS, transcranial magnetic stimulation.
The mean GQS score of the ChatGPT responses was 3.69 (ICC, 0.17), indicating moderate-to-good quality, with some missing or insufficiently covered points, but mostly useful for patients; the mean DISCERN score was 35.33 (ICC, 0.29), indicating poor quality; and the JAMA Benchmark Criteria score was 0 in all cases because the responses lacked affiliations, references, funding disclosures, and currency.
The mean DISCERN score indicated a low overall level of accuracy and reliability of the responses. In Introduction of the DISCERN instrument, the responses were rated as mostly reliable, whereas lower scores were observed for items in Methods related to references and the declaration of information sources.
Analysis of the mean GQS scores for individual questions showed that 1 response was classified as Excellent, 16 as Good, and 8 as Moderate, with no responses classified as Generally Poor or Poor. Question 12 (Q12; ‘What are the side effects of rTMS?’) had the highest mean score (4.67; Excellent), whereas Questions 15 and 17 (Q15 and Q17; ‘Do I need to be hospitalised for a course of rTMS?’ and ‘How long will the benefit of rTMS last if I respond?’) had the lowest mean scores (2.67 each; Moderate). In the TMS expert reviewer’s assessment, Q12 also received the highest score (5).
Beyond the overall mean DISCERN rating of Poor, analysis of the mean scores for individual questions showed that none of the items were classified as Good or Excellent. Overall, 6 items were classified as Fair, 17 as Poor, and 1 as Very Poor. Question 20 (Q20; ‘Can I have rTMS if I am pregnant or breastfeeding?’) had the highest mean score (46.33; Fair), whereas Question 6 (Q6; ‘Who administers rTMS?’) had the lowest mean score (25.33; Very Poor). In the TMS expert reviewer’s assessment, Q20 also received one of the highest scores (46), while the lowest score was assigned to Question 7 (Q7; 21), followed by Q6 (23).
Apart from these interpretations, Reviewer 3, a TMS expert, rated 5 answers as ‘excellent quality and excellent flow, very useful for patient’, 12 answers as ‘Good quality and general good flow, most of the relevant information is listed, but some topics not covered, useful for patients’, and 8 answers as ‘Moderate quality, suboptimal flow, some important information is adequately discussed, but others poorly discussed, somewhat useful for patients’. The median value for GQS of 4 and mean score of 3.88 was given by Reviewer 3, and median value for DISCERN of 37 and mean score of 36.56 was given by Reviewer 3 (Table 5).
To assess agreement between the psychiatrists without expertise in TMS (Reviewers 1 and 2) and the psychiatrist who was a TMS expert (Reviewer 3), we compared the mean ICC of Reviewers 1 and 2 with that of Reviewer 3. The ICCs for GQS (ICC, 0.46) and DISCERN (ICC, 0.42) were higher for this comparison than that calculated when all three reviewers were compared. Although the DISCERN ICC of 0.42 was not in the moderate range, it still indicated that the reliability of the TMS expert’s ratings was comparatively higher than those of the two psychiatrists. The ICCs between the two psychiatrists (Reviewer 1 and 2) who were not rTMS experts were 0.35 for DISCERN and 0.16 for GQS.
Discussion
This study aimed to explore how ChatGPT responses to frequently asked questions about rTMS may affect patient education and to raise awareness about this topic. In the study, two psychiatrists and a psychiatrist who was also a TMS expert evaluated ChatGPT responses to 25 frequently asked questions about rTMS by using the GQS, DISCERN, and JAMA Benchmark Criteria. The main finding was that ChatGPT has the potential to educate patients about rTMS because it provided responses that had an acceptable overall quality and were mostly useful for patients, although some points were missing or not covered.
The low overall DISCERN score may partly reflect the lack of references and explicit declaration of information sources, particularly affecting ratings in Methods. Thus, the overall DISCERN score would be higher if responses included citations for and declarations about the sources of information.
Across different areas of TMS-related information, higher scores were observed for clinical topics, including adverse effects (Q12) and safety (Q20), whereas lower scores were found for questions involving prognosis (Q17) and logistical aspects of treatment delivery (Q6, Q15). These differences may indicate potential limitations of LLM-generated information when responses depend on individual clinical trajectories or local healthcare contexts. This pattern may partly relate to the probabilistic nature of LLMs, which generate responses based on patterns learned from large-scale text corpora (Zhao et al., Reference Zhao, Zhou, Li, Tang, Dong, Hou, Zhang, Min, Zhang, Liu, Wang, Du, Yang, Chen, Chen, Jiang, Ren, Li, Tang, Liu, Hu, Nie and Wen2023; Liu et al., Reference Liu, Cao, Liu, Ding and Jin2025).
The reviewers’ mean GQS score of 3.69 suggests an acceptable level of quality, flow, and usefulness of the responses for patients and indicates a satisfactory degree of consensus among the three reviewers in their evaluation of the quality of the responses. However, the low ICC (0.17) indicates poor reliability, potentially because of the small number of reviewers and because of their differing opinions of the quality of certain responses. Nevertheless, although the reviewers’ ratings showed poor reliability, their mean ratings were similar, and they reached a consensus about the quality (rated with the GQS) and reliability and accuracy (rated with DISCERN) of the responses. In a similar study by Gudimella Tirumala et al. (Tirumala et al., Reference Tirumala, Mishra, Trivedi, Shivakumar, Singh and Shariff2024), the evaluators used the GQS to assess the quality of responses generated by LLMs and agreed on the overall quality in terms of usefulness and comprehensiveness. Another study by Aguirre et al. (Aguirre et al., Reference Aguirre, Hilsabeck, Smith, Xie, He, Wang and Zou2024) used a 4-item rating scale to evaluate the quality of ChatGPT’s responses to dementia caregivers; although clinicians disagreed about the comprehensiveness and specificity of the responses, they found that the responses were of high quality.
One study on the use of AI in psychiatry evaluated the potential use of ChatGPT for psychodynamic formulations (Hwang et al., Reference Hwang, Lee, Seol, Jung, Choi, Her, An and Park2024). This study suggests that ChatGPT may have potential applications in psychiatry, given that it produced appropriate results. Lawrence et al. (Reference Lawrence, Schneider, Rubin, Mataric, McDuff and Bell2024) discuss the opportunities and risks of using LLMs in mental health and indicate that LLMs have strong potential to support mental health information, although caution is needed when people use LLMs to assess their own mental health (Lawrence et al., Reference Lawrence, Schneider, Rubin, Mataric, McDuff and Bell2024). To our knowledge, our study is the first to assess ChatGPT’s responses to patients’ most common questions about rTMS; however, several other studies have evaluated other mental health-related topics (Lundin et al., Reference Lundin, Berk and Østergaard2023; Spallek et al., Reference Spallek, Birrell, Kershaw, Devine and Thornton2023; Alapati et al., Reference Alapati, Campbell, Molin, Creighton, Wei, Boon and Huntley2024; Saeidnia et al., Reference Saeidnia, Kozak, Lund and Hassanzadeh2024). This area represents an emerging research topic because LLMs are widely used by patients in psychiatry and healthcare in general (Pham et al., Reference Pham, Nabizadeh and Selek2022; Iqbal et al., Reference Iqbal, Tanweer, Rahmanti, Greenfield, Lee and Li2025).
Despite recommendations, including from the American Medical Association (Weiss et al., Reference Weiss, Schwartzberg, Davis, Parker, Williams and Wang2003), that patient health information should be written at a 5th or 6th grade reading level, our analysis shows that ChatGPT responses require an 11th grade reading level. A previous study on the readability of ChatGPT responses found a readability level of 8 and highlighted their patient friendliness (Tirumala et al., Reference Tirumala, Mishra, Trivedi, Shivakumar, Singh and Shariff2024), but another study found that the reading level was poor and did not meet the recommended standard (Spallek et al., Reference Spallek, Birrell, Kershaw, Devine and Thornton2023). Thus, in some cases, ChatGPT responses may be difficult for people with a lower education level to understand.
Regarding the specific questions used in this study, ChatGPT’s response to ‘Who can benefit from rTMS?’ provided limited information on current studies and evidence-based guidelines (Slotema et al., Reference Slotema, Blom, Van Lutterveld, Hoek and Sommer2014; DGPPN, 2025). Also, it did not properly address the potential effects of rTMS on auditory hallucinations and negative symptoms (Lefaucheur et al., Reference Lefaucheur, Aleman, Baeken, Benninger, Brunelin, Di Lazzaro, Filipović, Grefkes, Hasan, Hummel, Jääskeläinen, Langguth, Leocani, Londero, Nardone, Nguyen, Nyffeler, Oliveira-Maia, Oliviero, Padberg, Palm, Paulus, Poulet, Quartarone, Rachid, Rektorová, Rossi, Sahlsten, Schecklmann, Szekely and Ziemann2020; DGPPN, 2025). Studies on the potential role of rTMS in treating auditory hallucinations remain limited, but nevertheless, more comprehensive information on these indications can be found in systematic reviews and expert guidelines, which provide some recommendations (Marzouk et al., Reference Marzouk, Winkelbeiner, Azizi, Malhotra and Homan2020; Cotovio et al., Reference Cotovio, Ventura, da Silva, Pereira and Oliveira-Maia2023).
Another response to the question ‘Who can benefit from rTMS’ and also the response to the question ‘What improvements can occur with rTMS therapy?’ provided information on OCD and PTSD. A systematic review of randomised controlled trials suggested that TMS shows promise as a treatment for OCD and PTSD, and research in this area has been shaped by neuroimaging and electrophysiological findings. However, the data may be insufficiently reliable given the complexity of mental health conditions and the human experience (Vicheva et al., Reference Vicheva, Osborne, Krieg, Ahmadi and Shotbolt2025). Although studies and guidelines have been published on the effects of rTMS in OCD and PTSD (Bandelow et al., Reference Bandelow, Allgulander, Baldwin, da Costa, Denys, Dilbaz, Domschke, Hollander, Kasper, Möller, Eriksson, Fineberg, Hättenschwiler, Kaiya, Karavaeva, Katzman, Kim, Inoue, Lim, Masdrakis, Menchón, Miguel, Nardi, Pallanti, Perna, Rujescu, Starcevic, Stein, Tsai, Van Ameringen, Vasileva, Wang and Zohar2023), guideline treatment recommendations vary (NICE, 2020; Voderholzer & Favreau, Reference Voderholzer and Favreau2022), and existing evidence shows inconsistencies and certain limitations (Freire et al., Reference Freire, Cabrera-Abreu and Milev2020). Differences between official recommendations and doctors’ advice may lead to discrepancies between patient expectations and treatment outcomes and may have potential negative effects on patients by negatively impacting therapeutic relationships, therapy outcomes, and adherence (Jackson and Kroenke, Reference Jackson and Kroenke2001; Bell et al., Reference Bell, Kravitz, Thom, Krupat and Azari2002; Brancazio et al., Reference Brancazio, Eskildsen, Abimbola, Olcott, Kamath and Del Gaizo2020; Pietrzykowski et al., Reference Pietrzykowski, Kasprzak, Michalski, Kosobucka, Fabiszak and Kubica2022).
Research has frequently addressed various aspects of the question ‘Who can benefit from rTMS?’ To date, studies have compared the effects of TMS in early treatment of depression with those of other first-line treatments, such as selective serotonin reuptake inhibitors, and its impact on various clusters of depressive symptoms (Becker-Sadzio et al., Reference Becker-Sadzio, Brendel, Weller, Bornheimer, Mehlig, Padberg, Vogelmann, Kammer, Strube, Martus, Fallgatter and Plewnia2025; Dalhuisen et al., Reference Dalhuisen, Biemans, Vos, ter Hark, van Oostrom, Spijker, Wijnen, van Exel, van Mierlo, de Waardt, Arns, Tendolkar, Janzing and van Eijndhoven2025). Overall, ChatGPT’s responses did not mention theta burst stimulation, a widely used rTMS technique that takes only a few minutes instead of the 20 to 40 minutes needed for other rTMS approaches and is therefore less time consuming for patients (Suppa et al., Reference Suppa, Huang, Funke, Ridding, Cheeran, Di Lazzaro, Ziemann and Rothwell2016).
We explored the potential and role of ChatGPT in the context of patient education on rTMS as a treatment option for psychiatric patients, and our results indicate that it can play a supportive role in providing accessible information and guidance. However, ChatGPT should not be viewed as a substitute for professional mental healthcare and expert advice. There are limitations to the use of ChatGPT in patient education and treatment planning regarding rTMS, where individualised decision-making is fundamental because of the nature of the treatment and the variability of treatment responses. Various rTMS methods are used that target different brain regions and differ in the frequency, intensity, and timing of the stimulation, which can make rTMS confusing for patients and decision-making more difficult (Klooster et al., Reference Klooster, Ferguson, Boon and Baeken2022; Benster et al., Reference Benster, Weissman, Suprani, Toney, Afshar, Stapper, Tello, Stolz, Poorganji, Daskalakis, Appelbaum and Kohn2025; DGPPN, 2025). Treatment decisions in psychiatry are complex and multifactorial, and medically trained health professionals who can integrate clinical evidence and patient preferences need to be involved when determining the most suitable treatment approach. ChatGPT lacks the clinical experience that healthcare professionals develop through years of medical training and direct patient interactions. Furthermore, it does not know a patient’s age, complete medical background (including results of physical examinations), laboratory findings, imaging results, symptoms, family history, treatment history, and comorbid psychiatric or medical conditions. Even when these data are available, interpreting them requires specialised training and clinical reasoning, and before considering rTMS as a treatment option, it is crucial to evaluate various patient-specific factors to ensure optimal outcomes.
This study finds that clinician involvement is needed to clarify the information provided by ChatGPT. For example, ChatGPT did not mention other TMS protocols or forms of TMS, such as intermittent theta burst stimulation or accelerated protocols like the Stanford protocol. This lack of information may result in patients missing important aspects of patient education. In addition, ChatGPT tended to attribute greater success to outcomes such as cognitive improvement and anxiety improvement, which may inaccurately increase patients’ expectations for improvement in these symptoms. Although there are studies examining conditions such as cognition, anxiety, and schizophrenia (including hallucinations and negative symptoms), these studies report mixed results with certain limitations. Thus, while ChatGPT can provide useful patient education, the information may at times be insufficient or too optimistic, indicating that a clinician should carefully review the information with the patient.
Our findings are consistent with previous research. For example, the assessment of ChatGPT’s responses in a study by Boßelmann et al. (Reference Boßelmann, Leu and Lal2023) supported the idea of using it as a supplementary tool for patient education but emphasised that responses should be reviewed by someone with expertise in the topic who can identify missing information and incorporate details from the patient’s medical history (Boßelmann et al., Reference Boßelmann, Leu and Lal2023). The quality of evidence on the efficacy of rTMS ranges from low to moderate, with the strongest data supporting high-frequency stimulation of the left dorsolateral prefrontal cortex. The nature of rTMS, the heterogeneity and rapid evolution of rTMS techniques, the need for personalised stimulation protocols, and the varying levels of recommendations in current guidelines indicate that treatment decisions regarding rTMS rely on an individualised decision-making process (NICE, 2015; NICE, 2020; Bundesärztekammer et al., 2022; Klooster et al., Reference Klooster, Ferguson, Boon and Baeken2022; Voderholzer and Favreau, Reference Voderholzer and Favreau2022; DGPPN, 2025).
The observed differences in inter-rater reliability may suggest that rTMS is a specialised field of psychiatry, where differences in clinical experience and knowledge and variations in individual scoring styles may affect the evaluation of LLM responses. The low inter-rater reliability of the responses of the two psychiatry specialists highlights the importance of guidance from an rTMS specialist in patient-centred decision-making. This observation may support the APA Position Statement 2024, which highlights that most psychiatrists lack sufficient education and training in TMS and emphasises the need for evidence-based TMS training for residents (Carpenter et al., Reference Carpenter, Cochran, Alpert, Krystal, McDonald and Widge2024).
Although the information provided by ChatGPT may be of acceptable quality, transparency remains limited, with all responses receiving a score of 0 on the JAMA Benchmark Criteria. This may lead to uncertainty among users regarding the sources of the information provided. Professional societies and clinical institutions may therefore consider providing evidence-based and comprehensible answers to frequently asked questions on their own platforms, written at recommended readability levels. Such resources could facilitate patients’ access to reliable health information and potentially reduce the need to rely on LLMs as a primary source of information.
In clinical practice, ChatGPT-generated information could be incorporated into rTMS counselling by psychiatry or neuropsychiatry specialists, with guidance from an rTMS expert, as expert-reviewed educational materials, such as information sheets or patient information booklets. Before being provided to patients, ChatGPT-generated content should be reviewed by the treating psychiatrist for accuracy and tailored to the individual patient’s clinical condition, expectations, and reading level. Such materials should be considered a supplementary source of information rather than a replacement for expert counselling and shared decision-making.
This study has some limitations. Besides using a small number of reviewers, it evaluated responses to a limited number of questions. The questions were based on information available online and filtered by the authors and therefore may not represent the full spectrum of patient questions. Furthermore, no individual follow-up inquiries were used. Because, to our knowledge, this was the first study on the usefulness of LLMs for patient education on rTMS, further studies are needed to strengthen the evidence. Ongoing studies on rTMS, regional differences, for example, in healthcare systems or hospitalisation requirements, may influence the way that reviewers evaluate responses to certain questions. Another limitation is that although we used the latest version of ChatGPT available at the time of the study, the LLMs and its responses are constantly evolving. This study represents results that are valid for a specific period. Only the unsigned public ChatGPT interface was evaluated; paid-account configurations and other LLM platforms were not assessed. Additionally, the study focuses exclusively on English responses and is based on 25 questions, without incorporating real-world user interactions or considering personalisation. The quality of responses may vary across languages or cultural contexts, which limits the generalisability of the results. Furthermore, scoring with GQS and DISCERN tools is based on subjective expert judgement and may vary between raters. However, these are the most widely used scores in this context, and they were evaluated by three independent specialist authors using an evidence-based approach to minimise potential bias. Although all three reviewers were experienced psychiatrists with relevant clinical experience, only one had specific expertise in rTMS. As rTMS is a specialised field within psychiatry and expertise is particularly important in this field, differences in reviewers’ experience, familiarity with the treatment, and level of rTMS-specific expertise may have contributed to variability in scoring. Future research will help address these limitations by examining multilingual responses, personalised interactions, real-world user feedback, and evaluations across updated AI model versions, by comparing different LLMs in their ability to respond, as well as by including a larger number of reviewers with specific expertise in rTMS.
Conclusion
Our findings indicate that ChatGPT provides responses of acceptable quality to frequently asked patient questions about rTMS. The responses flow clearly and appear to be useful for educating patients. However, the responses have limited reliability and accuracy, and understanding the information provided requires a high reading level. Therefore, ChatGPT should be used only as a supportive tool and with guidance from healthcare professionals (in the case of rTMS, with guidance from an expert in the field). While ChatGPT provides useful patient information, it is sometimes incomplete regarding TMS protocols or overly optimistic given inconsistent evidence, requiring review by a clinician. ChatGPT is being increasingly used by patients, and considering the current limitations of studies on ChatGPT and the poor reliability, accuracy, and readability outcomes, ongoing evaluation of the safety and appropriate use of this technology is necessary, especially in the field of patient education.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/neu.2026.10100.
Data availability statement
The contributions presented in the study are included in the article/tables/Supplementary Material; further inquiries can be directed to the corresponding author, and data will be made available on request.
Author contributions
B.K. and C.T. were responsible for the study’s conception and design. C.T., G.G., and M.K. participated as reviewers for the main part of the Study. At the same time, E.M., M.K., and C.T. supervised the whole study and made critical revisions. B.K. prepared the first draft of the manuscript and did the analysis of the results. T.V. provided comments on the manuscript. All authors have read and approved the final manuscript.
Funding statement
The study did not receive funding.
Competing interests
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
No Disclosure: Büşra Kalem, MD, Hatice Gülhan Gülel, MD, Theokritos Veskoukis, MD, Eva Meisenzahl-Lechner, MD, Milenko Kujovic, MD, Christian Theisen, MD.
Ethical standards
This study did not involve human participants, animals, or protected data, so it did not require ethical approval.
Open access
