Hostname: page-component-7b7567c8c-7pwfn Total loading time: 0 Render date: 2026-11-16T04:34:10.526Z Has data issue: false hasContentIssue false

Explaining Person-by-Item Responses using Person- and Item-Level Predictors via Random Forests and Interpretable Machine Learning in Explanatory Item Response Models

Published online by Cambridge University Press:  31 July 2025

Sun-Joo Cho*
Affiliation:
Vanderbilt University’s Peabody College, United States
Goodwin Amanda
Affiliation:
Vanderbilt University’s Peabody College, United States
Jorge Salas
Affiliation:
Vanderbilt University’s Peabody College, United States
Sophia Mueller
Affiliation:
Vanderbilt University’s Peabody College, United States
*
Corresponding author: Sun-Joo Cho; Email: sj.cho@vanderbilt.edu
Rights & Permissions [Opens in a new window]

Abstract

This study incorporates a random forest (RF) approach to probe complex interactions and nonlinearity among predictors into an item response model with the goal of using a hybrid approach to outperform either an RF or explanatory item response model (EIRM) only in explaining item responses. In the specified model, called EIRM-RF, predicted values using RF are added as a predictor in EIRM to model the nonlinear and interaction effects of person- and item-level predictors in person-by-item response data, while accounting for random effects over persons and items. The results of the EIRM-RF are probed with interpretable machine learning (ML) methods, including feature importance measures, partial dependence plots, accumulated local effect plots, and the H-statistic. The EIRM-RF and the interpretable methods are illustrated using an empirical data set to explain differences in reading comprehension in digital versus paper mediums, and the results of EIRM-RF are compared with those of EIRM and RF to show empirical differences in modeling the effects of predictors and random effects among EIRM, RF, and EIRM-RF. In addition, simulation studies are conducted to compare model accuracy among the three models and to evaluate the performance of interpretable ML methods.

Information

Type
Application and Case Studies - Original
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (https://creativecommons.org/licenses/by/4.0), which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited.
Copyright
© The Author(s), 2025. Published by Cambridge University Press on behalf of Psychometric Society
Figure 0

Table 1Empirical study: descriptive statistics of predictors.

Figure 1

Table 2Empirical study: accuracy of EIRM, RF, and EIRM-RF

Figure 2

Table 3Empirical study: results of the EIRM

Figure 3

Figure 1Empirical study: feature importance measures for EIRM-RF.Note: In the predictor (feature) name, “nom” indicates a nominal predictor, and “c” indicates the mean-centered predictor.

Figure 4

Figure 2Empirical study: partial dependence plots from EIRM-RF.Note: Continuous predictors were mean-centered; each tick mark in the x-axis represents values of a continuous predictor.

Figure 5

Figure 3Empirical study: partial dependence plots from EIRM-RF.Note: Continuous predictors were mean-centered; each tick mark in the x-axis represents values of a continuous predictor.

Figure 6

Figure 4Empirical study: partial dependence plots of EIRM-RF.Note: Continuous predictors were mean-centered; each tick mark in the x-axis represents values of a continuous predictor.

Figure 7

Figure 5Empirical study: accumulated local plots of EIRM-RF.Note: Continuous predictors were mean-centered; each tick mark in the x-axis represents values of a continuous predictor.

Figure 8

Figure 6Empirical study: the H-statistic of all predictors from EIRM-RF.Note: In the predictor (feature) name, “nom” indicates a nominal predictor, and “c” indicates the mean-centered predictor; Values on the x-axis indicate the strengths of the interaction effects.

Figure 9

Figure 7Empirical study: the H-statistic of the two-way interactions between the item format and all other predictors from EIRM-RF.Note: In the predictor (feature) name, “nom” indicates a nominal predictor, and “c” indicates the mean-centered predictor; Values on the x-axis indicate the strengths of the interaction effects.

Figure 10

Table 4Simulation study 1: model accuracy of EIRM, RF, and EIRM-RF

Figure 11

Table 5Simulation study 2: performance of interpretable ML methods under EIRM-tree.

Figure 12

Figure A.1Visualization of a generated tree structure.

Figure 13

Table A1Normalized importance measure and H-statistic for each predictor.

Figure 14

Figure A2Selected plots of ‘true’ partial dependence (top) and accumulated local effects (bottom).Note: Continuous predictors were mean-centered; To present patterns in partial dependence and accumulated local effects, the x- and y-axes of the two selected predictors were not displayed on the same scale.

Figure 15

Figure A3Visualization of a generated tree structure.

Figure 16

Table A2Normalized importance measure and H-statistic for each predictor.

Figure 17

Figure A4Selected plots of “true” partial dependence (top) and accumulated local effects (bottom).Note: Continuous predictors were mean-centered; To present patterns in partial dependence and accumulated local effects, the x- and y-axes of the two selected predictors were not displayed on the same scale.

Figure 18

Table B1Model accuracy of EIRM-RF