Background
Self-reported data are widely used in large-scale epidemiological and psychiatric research due to their efficiency, cost-effectiveness and scalability. Compared to clinician-administered assessments, self-report tools allow researchers to collect health-related information across diverse populations with minimal logistical demands, making them particularly suited to longitudinal studies and national health surveillance (Tourangeau et al., Reference Tourangeau, Rips and Rasinski2000; Althubaiti, Reference Althubaiti2016). This approach is especially valuable in psychiatric research, where repeated clinical assessments may be limited by cost, access or stigma (Thornicroft et al., Reference Thornicroft, Brohan, Rose, Sartorius and Leese2009).
Traditionally, epidemiological studies have relied on psychometrically validated instruments to measure constructs such as depression, anxiety, post-traumatic stress and personality disorders (Manea et al., Reference Manea, Gilbody and McMillan2012; Ho et al., Reference Ho, Karatzias, Cloitre, Chan, Bressington, Chien, Hyland and Shevlin2019). These tools offer standardized, validated approaches to symptom assessment. However, many studies also depend on participants’ retrospective self-reports of formal diagnoses, typically collected using single-item checklist questions (e.g., ‘Have you ever been diagnosed with X?’). These items are widely used to assess prevalence, comorbidity and service utilization (Okura et al., Reference Okura, Urban, Mahoney, Jacobsen and Rodeheffer2004), yet their psychometric properties, particularly their test–retest reliability, have not been sufficiently examined across diverse diagnostic categories or in clinically complex populations (Houck et al., Reference Houck, Forcehimes, Gutierrez and Bogenschutz2013).
Existing reliability research has focused largely on physical health conditions. Studies have reported good agreement for diabetes and hypertension, with kappa values in the moderate to substantial range (Muggah et al., Reference Muggah, Graves, Bennett and Manuel2013). In contrast, conditions like osteoarthritis or chronic lung disease yield more variable results, reflecting how diagnostic clarity and symptom salience shape recall (Simpson et al., Reference Simpson, Boyd, Carlson, Griswold, Guralnik and Fried2004). Far fewer studies have examined psychiatric and neurodevelopmental diagnoses, limiting understanding of their reliability in self-report (Stang et al., Reference Stang, Brandenburg, Lane, Merikangas, Von Korff and Kessler2006). Moreover, most research to date has examined only a few conditions (e.g., only physical diseases) at a time (e.g., Okura et al., Reference Okura, Urban, Mahoney, Jacobsen and Rodeheffer2004; Steinkirchner et al., Reference Steinkirchner, Zimmermann, Donhauser, Dietl, Brandl, Koller, Loss, Heid and Stark2022; Jensen et al., Reference Jensen, Lau, Davidsen, Ekholm and Christensen2023), restricting cross-domain comparisons and overlooking how diagnostic complexity may influence consistency.
Gendered diagnostic disparities further complicate self-report. Females are more likely to be underdiagnosed or diagnosed later for mental health conditions, often due to differing symptom profiles and the use of social masking or internalizing behaviours (Loomes et al., Reference Loomes, Hull and Mandy2017; Young et al., Reference Young, Adamo, Ásgeirsdóttir, Branney, Beckett, Colley, Cubbin, Deeley, Farrag, Gudjonsson, Hill, Hollingdale, Kilic, Lloyd, Mason, Paliokosta, Perecherla, Sedgwick, Skirrow, Tierney, van Rensburg and Woodhouse2020). Female participants and health service users are also underrepresented in psychiatric research (Pedersen et al., Reference Pedersen, Lindstrom, Powe, Louie and Escobar-viera2022). Assessing how women report health data and medical diagnoses over time is thus critical for understanding the reliability of diagnostic recall in marginalized populations (Green et al., Reference Green, Travers, Howe and McDougle2019). Such data will be important to justify whether or not self-report health histories and diagnoses are reliable in large-scale studies involving diverse populations.
This study addressed these gaps by evaluating the 6-month test–retest reliability of 28 clinical diagnoses and health histories as self-reported by a sample of adult female mental health service users. In research, high consistency across time (i.e., test–retest stability or reliability) is the foundational metric. Patients’ self-report health histories and diagnoses are often asked in large-scale epidemiological and psychiatric research, because access to hospital records may not always be feasible. By assessing a broad spectrum of conditions, including psychiatric, physical, neurodevelopmental and functional domains, this study provides novel evidence on the stability of diagnostic recall, informing the development of inclusive, methodologically sound self-report tools for epidemiological and clinical research.
Methods
Participants
This study analysed data from the International Female Mental Health Survey, which is a collaborative project approved by the institutional review board at the Leshan Normal University. In 2024, the research team recruited potential participants in affiliated clinics and community centres and through online social media advertising. Eligibility criteria included being female aged 18 or above, able to provide informed consent, possess the ability to read and write in English, receive any kind of mental health services in the past 12 months, and have not been officially diagnosed with reading disorders or any cognitive or intellectual impairments. Participants were invited to complete online surveys at two times, approximately 6 months apart.
In total, 293 female mental health service users provided valid responses at T1 and T2. Specifically, 57.0% had seen a psychiatrist, 50.5% had seen a clinical psychologist and 73.7% had seen a therapist in the past 12 months, indicating a highly clinically engaged sample. Their ages ranged from 18 to 64 years (M = 26.6; SD = 7.21). Socioeconomic indicators revealed that 68.9% had a bachelor’s degree, 43.4% were currently employed and 35.5% were married. Most participants reported living in Malaysia (20.1%), North America (19.8%), the Philippines (16.4%), the United Kingdom (14.7%) and Indonesia (12.3%).
Measures
Participants completed a set of questions which asked about their health backgrounds and clinical diagnoses at baseline, and they completed the same set of questions again at T2. A total of 28 health histories and diagnoses were assessed, spanning mental health conditions (e.g., depression, anxiety, Post-Traumatic Stress Disorder i.e., PTSD, schizophrenia, bipolar disorder), developmental disorders (e.g., autism, Attention Deficit Hyperactivity Disorder i.e., ADHD), physical illnesses (e.g., asthma, cancer, diabetes) and impairments (e.g., hearing, mobility).
Sample items include the following: ‘Have you ever been diagnosed with kidney diseases?’ ‘Have you ever been diagnosed with mobility impairment (e.g., upper or lower limb loss or disability, manual dexterity disability)?’ ‘Have you ever been diagnosed with a Depressive Disorder?’ The list of 28 health conditions can be found in Table 1.
Agreement metrics and Cohen’s kappa
Table 1 Long description
Agreement metrics compare whether each diagnosis was reported at two time points (T1 and T2), summarizing prevalence at each time, overall agreement, Cohen’s kappa, and the share of yes–yes, no–no, discrepant, and new T2 records. Agreement is highest for type 1 diabetes and stroke at 99.3 percent, and remains very high for hypothyroidism at 98.3 percent and type 2 diabetes and cancer at 98.7 percent. Kappa indicates almost perfect agreement for several conditions including personality disorder (0.89), PCOS (0.88), eating disorder (0.86), asthma (0.86), ADHD (0.84), eczema (0.84), and bipolar disorder (0.83). Substantial agreement appears for depressive disorder (0.75), autism (0.74), PTSD or complex PTSD (0.73), anxiety disorder (0.68), OCD (0.67), hyperlipidemia (0.65), and type 1 diabetes (0.66). Moderate agreement is reported for hearing and mobility impairment (both 0.58), neuromuscular (0.54), schizophrenia spectrum (0.48), hypertension (0.46), kidney diseases (0.49), hyperthyroidism (0.44), and cancer (0.59), while somatic symptom disorders are fair (0.27) and stroke and cerebral palsy are poor (0). Depressive disorder and anxiety disorder have the lowest overall agreement (88.1 percent and 84.5 percent) and the highest discrepant rates (11.9 percent and 15.5 percent), despite high yes–yes percentages. New records at T2 are generally low, peaking for OCD at 6.1 percent and reaching 6.8 percent for both depressive disorder and anxiety disorder. Interpretation should consider that kappa can be reduced when a diagnosis is rare or very common, even when overall agreement is high.
* p < .05, **p < .01, ***p < .001.
Data analysis
To examine the test–retest reliability (temporal stability) of the self-reported health records, we provided descriptive statistics at T1 and T2. Following previous studies on the test–retest reliability of other psychological and health records (e.g., Goldberg and Freyd, Reference Goldberg and Freyd2006), we calculated agreement rates and Cohen’s kappa. Agreement thresholds were interpreted according to Landis and Koch’s benchmark criteria (Landis and Koch, Reference Landis and Koch1977).
Results
Table 1 presents detailed test–retest reliability metrics for 28 self-reported health records and diagnoses.
Overall, the results demonstrated high levels of temporal consistency. Agreement rates were above 90% for the majority of diagnoses, and Cohen’s kappa coefficients largely fell within the moderate (κ = 0.41–0.60) to substantial (κ = 0.61–0.80) ranges. Several conditions exhibited ‘almost perfect’ agreement (κ > 0.81), underscoring the good reliability of self-reported health records in this sample.
Among physical health conditions, asthma (κ = 0.86), eczema (κ = 0.84) and type 1 diabetes (κ = 0.66) demonstrated especially high temporal reliability, with agreement rates exceeding 94%. Hypertension and hyperlipidemia yielded slightly lower but still acceptable kappa values (κ = 0.46 and κ = 0.65, respectively). For relatively rarer physical conditions such as cancer, kidney disease and stroke, agreement rates were similarly high (≥98%), but Cohen’s kappa values were reduced due to low base rates (e.g., κ = 0.00 for stroke, κ = 0.59 for cancer).
Among mental health conditions, bipolar disorder (κ = 0.83), eating disorders (κ = 0.86) and personality disorders (κ = 0.89) showed excellent reliability. Similarly, schizophrenia (κ = 0.48) showed high agreement (96.0%) but only moderate kappa, again likely impacted by its lower prevalence. Depression (κ = 0.75) and anxiety disorders (κ = 0.68) were the most frequently endorsed mental health conditions and still demonstrated substantial agreement. PTSD/complex PTSD (CPTSD) also reached substantial agreement (κ = 0.73), though its 10.2% discrepancy rate was somewhat higher compared to other mental health conditions.
Developmental and neurodivergent conditions also showed high test–retest reliability. Autism (κ = 0.74) and ADHD (κ = 0.84) achieved high agreement rates and strong kappa values.
Discussion
This study evaluated the 6-month test–retest reliability of self-reported clinical diagnoses and health records in a sample of 293 adult female mental health service users. Across 28 conditions – including physical, psychiatric, neurodevelopmental and functional domains – results indicated strong reliability, particularly for chronic and clinically salient conditions such as diabetes, asthma and schizophrenia. The results demonstrated high levels of temporal consistency. Agreement rates were ≥84.5% for all conditions, while Cohen’s kappa values were above 0.50 for 21 out of 28 conditions. Even among diagnoses with greater complexity or stigma, such as PTSD or personality disorder, moderate to substantial agreement was maintained. These findings provide compelling support for the temporal stability of self-reported diagnostic data in this population.
To our knowledge, this is one of the most comprehensive studies to assess the test–retest reliability across such a wide range of diagnoses in a single, clinically engaged sample. While prior studies have examined self-reports of physical illness, few have concurrently evaluated psychiatric and developmental conditions, particularly in women. The inclusion of historically underdiagnosed conditions in female populations, such as ADHD, autism and somatic disorder, further contributes to this literature, addressing gendered diagnostic disparities and enhancing the representativeness of self-report tools.
The findings carry several implications. They suggest that self-report checklists, as widely used in survey studies, can yield reliable health background data, supporting their utility in social and epidemiological research, especially when clinical validation is not feasible or when hospital records are not accessible. The observed consistency across domains also affirms the feasibility of transdiagnostic survey tools to study comorbidity and service access among women with complex clinical profiles. The results are consistent with previous studies showing that self-report health records are generally reliable and valid, although most previous studies only focused on a limited number of conditions (e.g., only physical diseases) (e.g., Wielgosz et al., Reference Wielgosz, Dawood, Stockwell, Varner, Newes-adeyi, Ellington, Vargas, Bruno, Powers, Morrill, Reichle, Battarbee and Tita2022; Jensen et al., Reference Jensen, Lau, Davidsen, Ekholm and Christensen2023). More importantly, we provide evidence showing that physical, psychiatric, neurodevelopmental and functional conditions are reliably reported even in people with mental health conditions.
Nonetheless, several limitations warrant consideration. First, the study assumes diagnostic stability over time – a reasonable expectation for chronic physical conditions, but a more complex issue for psychiatric diagnoses, which may fluctuate or emerge due to symptom progression, evolving interpretation or increased self-awareness. For example, 17 participants newly reported PTSD at T2, along with 15 for ADHD and 14 for anxiety disorder, possibly reflecting delayed or late diagnosis rather than inconsistent recall. Nevertheless, new diagnosis records are minimal, and we believe that these would not sustainably affect the results (see Table 1). Second, discrepancies in psychiatric self-report may stem not from poor reliability, but from variation in how diagnoses are communicated (e.g., ‘depression’ vs. ‘major depressive disorder’), the specificity of labels used or differences in insight. These mismatches may reflect interpretation variability rather than error. Third, the sample exclusively comprised female mental health service users with recent clinical engagement and more educational backgrounds – a population likely to have higher diagnostic literacy, healthcare access and familiarity with psychiatric terminology than the general public. Given that it is a convenience sample, findings may not generalize to individuals with lower education levels, less clinical contact or undiagnosed conditions. The reliability of self-report health histories among those with less knowledge and lower levels of education should be further examined in future studies. Finally, Cohen’s kappa is sensitive to base rates and may underestimate reliability in low-prevalence conditions despite high raw agreement. Future research should consider complementary methods such as latent class analysis or probabilistic modelling to better assess reliability in such contexts.
Despite these limitations, we offer new data showing that mental health service users do reliably report their clinical diagnoses and health records in epidemiological surveys. Social and health science researchers can employ such checklists to collect health background information of research participants in resource-limited settings.
Availability of data and materials
Data that support the findings of this study are available from the corresponding authors upon reasonable request.
Acknowledgements
NA.
Author contributions
G.F.Y. and H.W.F. contributed to the design of the international survey project. H.W.F. contributed to the conceptualization of this study. A.C.Y.W. and H.W.F. conducted data analysis and prepared the first draft. H.W.F., G.F.Y., M.E.S.R., E.S.J., F.M., A.E.Z.L., G.D., P.D.B., G.K.K., K.U., S.H., N.I.D. and R.N. collected data for this study. All authors critically reviewed the manuscript and contributed to the final draft of the manuscript. All authors read and approved the final manuscript.
Financial support
No funding was received for conducting this study.
Competing interests
None.
Ethical standards
All participants provided online written informed consent before participation. This study obtained ethical approval from the institutional review board at the Leshan Normal University, China.
Open access
