We previously trained and tested a machine learning model to predict incident type 2 diabetes (T2D) in patients with mental illness using electronic health record (EHR) data from routine clinical practice. This hyperparameter-tuned XGBoost model – based on information on sex, age and mean HbA1c within the past two years – predicts whether a patient will meet criteria for T2D in the two years following each contact to psychiatric services. The performance of such a model may drift over time due to changes in standards of care, baseline risk, registration procedures, etc. Therefore, we set out to examine its temporal stability.

In the present study, we fitted the model described above on EHR data from January 1, 2013 to December 31, 2017, and examined its stability in predicting T2D two years into the future for patients with contacts in 2018, 2019, 2020, 2021, 2022, and 2023, respectively. Performance was quantified using the area under the receiver operating characteristic curve (AUROC).

The model showed good discriminative performance across all years (AUROC for 2018 = 0.84, 2019 = 0.83, 2020 = 0.87, 2021 = 0.80, 2022 = 0.84, 2023 = 0.80), with a linear model estimating a decrease in AUROC for T2D prediction of only 0.004 per year (95%CI: –0.024; 0.016). Secondary analyses suggested that models trained on recent data (past year) may achieve similar or better discrimination than models trained on longer historical periods.

The temporal stability of the T2D prediction model supports its potential for clinical implementation, facilitating early detection and tailored interventions, while highlighting the importance of continued evaluation.