From raw data to actionable insights: preprocessing real-world data for machine learning in diabetes care

Background: Machine Learning (ML) applied to healthcare Real World Data (RWD) may improve patient management. RWD, however, requires extensive preprocessing to make it ML-ready. Our aim was to explore the impact of preprocessing on ML models applied to RWD from 20 years of type 2 diabetes patients visits. Methods: Our cohort consisted of patients with at least two glycated hemoglobin (HbA1c) measurements three years apart. We set up three different experimental settings consisting of different data preprocessing pipelines. Logistic Regression (LR), XGBoost and a Decision Tree Classifier (DTC) were then applied and tuned to optimize precision. Results: The final dataset comprised 12 variables from 1,651 patients treated between 2003 and 2023. 921 (56%) patients had a HbA1c decrease at three years. This group had a higher baseline HbA1c, higher BMI and shorter first visit gap from the date of diagnosis (p < 0.0001). Precision scores for LR, XGBoost did not vary across different experimental conditions while DTC benefitted from missing data imputation. Shapley Additive Explanations confirmed the Exploratory Data Analysis findings, with worse baseline values being predictors of HbA1c decrease at three years. Conclusions: ML models' performance and their explanation did not vary substantially across experimental conditions, with worse baseline values being predictors of HbA1c decrease at three years. Insights such as this, extracted by ML application to RWD, enable clinical discussion and may foster improvements in patient management.