This article proposes a mixed-effects machine-learning framework for modeling complex, nonlinear relations between predictors and continuous outcomes in multilevel cross-classified data. The proposed method, termed LMM–XGBoost, embeds extreme gradient boosting (XGBoost) within a linear mixed model (LMM) to combine flexible modeling of nonlinear and interaction effects with random effects that model dependence. In addition, an iterative estimation procedure for LMM–XGBoost is developed, a group-aware permutation importance measure that respects multilevel dependence is proposed, and a combined-group cross-validation (CV) strategy for hyperparameter tuning, out-of-fold (OOF) prediction, and importance estimation is developed for cross-classified designs. The simulation study shows that the proposed estimation method for LMM–XGBoost yields good parameter recovery under non-zero random-effect variances. In addition, relative to standard LMM and XGBoost, LMM–XGBoost achieves lower OOF prediction error and more accurate recovery of variable importance. The study further shows that combined-group CV and group-aware permutation importance yield less biased error estimates and substantially higher agreement with the true importance rankings than conventional permutation measures. An empirical application using the Add Health study illustrates how the proposed methods can identify important factors across multiple domains associated with adolescent depressive symptoms.

Traditional large-scale educational data are typically static and updated periodically, making it difficult to capture the dynamic changes in real time. However, recent technological advancements allow online exam platforms to collect students’ response data in real time. While item response theory (IRT) estimation methods are widely recognized for their accuracy, they are primarily designed for offline environments. When real-time data continuously arrives and online parameter estimation is required, these methods become computationally impractical. To address this challenge, we propose a recursive stochastic algorithm, i.e., truncated average stochastic Newton algorithm (TASNA), for the efficient online parameter estimation within the IRT framework. This algorithm significantly improves computational efficiency compared to the expectation–maximization (EM) algorithm implemented in the mirt package in R. The algorithm offers a powerful alternative to the traditional offline EM method. Furthermore, we investigate the asymptotic properties of the algorithm, proving its almost sure convergence and asymptotic normality. Numerical experiments using both simulated and real data demonstrate the practicality of the proposed method.

Various item selection algorithms have been proposed for cognitive diagnostic computerized adaptive testing (CD-CAT), with the goal of efficiently diagnosing examinees’ strengths and weaknesses. However, these algorithms often come with significant computational costs, which can hinder their practical implementation. A likelihood-based profile shrinkage (LBPS) algorithm is proposed to simplify the item selection process and reduce the computational cost in CD-CAT. Our simulation results indicate that incorporating LBPS into existing item selection methods yields substantial computational efficiency gains, with greater reductions in computation time as the number of attributes and test length increase. Additionally, LBPS maintains estimation accuracy at both the attribute and pattern levels. These findings suggest that LBPS is a scalable and effective solution for the item selection of CD-CAT in complex scenarios.

In item response theory (IRT), the conventional latent trait scale ($\theta $) is inherently arbitrary, lacking a fixed unit or origin and often tied to specific population distributional assumptions (e.g., standard normal). This limits the direct comparability and interpretability of scores across different tests, populations, or model estimation methods. This article introduces the “bit scale,” a novel metric transformation for unidimensional IRT scores derived from fundamental principles of information theory, specifically surprisal and entropy. Bit scores are anchored to the properties of the test items rather than the test-taker population. This item-based anchoring ensures the scale’s invariance to population assumptions and provides a consistent metric for comparing latent trait levels. We illustrate the utility of the bit scale through empirical examples: demonstrating consistent scoring when fitting models with different $\theta $ scale assumptions, and using anchor items to directly link scores from different test administrations. A simulation study confirms the desirable statistical properties (low bias and accurate standard errors) of Maximum Likelihood-estimated bit scores and their robustness to extreme scores. The bit scale offers a theoretically grounded, interpretable, and comparable metric for reporting and analyzing IRT-based assessment results. Software implementations in R (bitscale) and Python (IRTorch) are available and practical implications are discussed.

Test speededness, caused by time constraints, can impact examinees’ performance, leading to decreased response accuracy, particularly toward the end of the test. Most existing methods for detecting test speededness rely on specific distributional assumptions for response times (RTs), such as the lognormal distribution, which may lead to incorrect statistical inference if the true data distribution deviates from these assumptions. This article proposes a novel Bootstrap-CUSUM method for detecting test speededness, which is robust to non-normality in log-RTs. By constructing a cumulative sum (CUSUM) person-fit statistic for log-RTs and using the multiplier bootstrap to estimate its empirical distribution, our method facilitates individual-level detection and changepoint estimation. We prove the theoretical consistency of the method under both null and alternative hypotheses. Simulation studies show that the Bootstrap-CUSUM method outperforms the likelihood ratio test, Wald test, and score test in terms of correct classification rate, true detection rate, and false positive rate, demonstrating superior robustness and adaptability across different data distributions. The real data analysis further demonstrates the practical utility of the proposed method for detecting test speededness.

Over the past two decades, there has been growing interest in analyzing the effects of educational programs on outcomes using process data from computer-based testing and learning environments. However, most analyses focus on final outcomes at the end of a test or session, overlooking their functional nature over time and neglecting causal mechanisms. To address this gap, this article proposes a novel causal mediation framework for identifying and estimating functional natural direct effects, functional natural indirect effects, and functional total effects, along with their subgroup effects. We define these effects using potential outcomes and provide nonparametric identification strategies depending on whether post-treatment covariates are present or not. We then develop estimation methods using generalized additive models, a flexible and robust tool for analyzing functional data. Through a simulation study, we assess the finite-sample performance of the proposed approach by comparing it to parametric regression methods. We also demonstrate our approach by examining the effects of extended time accommodations on two functional outcomes using process data from the National Assessment of Educational Progress. Our mediation approach with functional outcomes effectively captures dynamic causal mechanisms underlying the program’s effects and pinpoints when and for whom each effect manifests throughout the testing period.

Longitudinal mental health assessments in mobile health (mHealth) settings are useful for monitoring subjects’ mental health statuses but are often difficult to analyze because they generally appear on an ordinal scale and at unequal time intervals. In this article, we explore the use of Gaussian processes (GPs) and hierarchical modeling techniques to understand mental health trajectories based on repeated multi-item mHealth surveys on a Likert scale. We introduce the GP model for health trajectories, which is based on item response theory. In the study of trajectories, a subject’s longitudinal collection of mHealth responses can be thought of as a single high-dimensional observation. We show how the GP is flexible enough to capture trends in individual trajectories even with the challenges associated with high-dimensional data. We also demonstrate how basis splines can be used to effectively capture nonlinear trends in the mean function of the GP. The high-dimension and ordinal nature of the data often make sampling from the posterior distribution in a Bayesian setting too slow to be practical. We show that using a Hilbert approximation for the GP trajectories can facilitate efficient sampling. We apply these methods to a longitudinal study that monitored college students’ self-esteem.

The association between a continuous and an ordinal variable is commonly modeled through the polyserial correlation model. However, this model, which is based on a partially-latent normality assumption, may be misspecified in practice, due to, for example (but not limited to), outliers or careless responses. The typically used maximum likelihood (ML) estimator is highly susceptible to such misspecification: One single observation not generated by partially-latent normality can suffice to produce arbitrarily poor estimates. As a remedy, we propose a novel estimator of the polyserial correlation model designed to be robust against the adverse effects of observations discrepant to that model. The estimator leverages density power divergence estimation to achieve robustness by implicitly downweighting such observations; the ensuing weights constitute a useful tool for pinpointing potential sources of model misspecification. The proposed estimator generalizes ML and is consistent as well as asymptotically Gaussian. As price for robustness, some efficiency must be sacrificed, but substantial robustness can be gained while maintaining more than 98% of ML efficiency. We demonstrate our estimator’s robustness and practical usefulness in simulation experiments and an empirical application in personality psychology where our estimator helps identify outliers. Finally, the proposed methodology is implemented in free open-source software.

In this paper, we generalize the multidimensional discrimination and difficulty parameters in the multidimensional two-parameter logistic model to account for nonidentity latent covariances and negatively keyed items. We apply Reckase’s maximum discrimination point method to define them in an arbitrary algebraic basis. Then, we define that basis to be a geometrical representation of the measured construct. This results in three different versions of the parameters: the original one, based on the item parameters solely; one that incorporates the covariance structure of the latent space; and one that uses the correlation structure instead. Importantly, we find that the items should be properly represented in a test space, distinct from the latent space. We also provide a procedure for the geometrical representation of the items in the test space and apply our results to examples from the literature to get a more accurate representation of the measurement properties of the items. We recommend using the covariance structure version for describing the properties of the parameters and the correlation structure version for graphical representation. Finally, we discuss the implications of this generalization for other multidimensional item response theory models and the parallels of our results in common factor model theory.

Psychological research has long centered around questionnaire assessments, but now digital devices, especially smartphones, enable the collection of real-world behavioral data through mobile sensing. While this data collection method offers unique opportunities, it also introduces new methodological challenges, as mobile-sensing data are highly complex and high in dimensionality (i.e., timestamped events with millisecond resolution), requiring advanced preprocessing to derive psychologically meaningful variables. This article highlights these challenges by reviewing the current state of data preprocessing based on app usage logs from smartphones. Afterward, it presents three preprocessing cases that vary in complexity across the dimensions of data enrichment—which involves adding context to raw data by integrating information from external and internal sources (including ecological momentary assessments)—and data aggregation—which entails summarizing data in different ways, from basic descriptive statistics to sophisticated machine-learning models. For each case, potential pitfalls are identified, and extensions are discussed to refine our preprocessing pipelines and accommodate different data types and research questions. By outlining these preprocessing strategies, this manuscript demonstrates the rich potential of mobile-sensing data for extracting nuanced behavioral variables beyond simple person-level summaries and aims to inspire the development of more advanced research questions based on sensing data.

We propose a latent trait model for the responses and response times on tests that separates capability from persistence. Core of the model is a race between a diffusion process and a censoring process. The diffusion process represents item-level cognitive processing and determines the processing time of a test taker. The censoring process sets the maximal time a test taker is willing to invest into an item. If the processing time is shorter than the maximal time, the response is generated by the diffusion process; otherwise, the response is generated differently. In the first version of the model, the response is generated by a random guess. In the second version of the model, the response is determined by the actual level of the diffusion process. We relate the diffusion process to the capability and response caution of a test taker and the censoring process to his willingness to invest time. Similar to models for rapid guessing, the proposed models take account of disengaged responding, but allow for individual differences in persistence and informed guessing. The model also provides a mathematical specification of persistence. In a simulation study, we investigate model fitting by marginal maximum likelihood estimation. We also apply the model to two empirical data sets.

Behavioral models are instrumental for studying human cognition, yet many inferences derived from such models fail to generalize. We argue that this is driven in part by the increasing complexity of behavioral models, where non-linearities and discontinuities create dynamic parameter interactions that limit the generalizability of inferences across different contexts, experiments, and datasets. We first demonstrate the problems that arise from parameter dependency. We then propose a new methodological framework for understanding the generalizability of behavioral modeling results using multivariate sampling distributions for the model parameters. We derive and validate novel sampling distributions for complex non-linear behavioral models by transforming the mimicry between different parameter values into the chances of one set of parameters being inferred from data generated by another set of parameters. Our approach is computationally scalable to evaluate how model estimates change across the parameter space and different experiments, which can limit the generalizability of experimental results. We then apply our approach to current behavioral models, revealing new theoretical insights. Using our approach, we reinterpret results from recent modeling work in decision-making and category learning. We conclude by discussing the implications of our proposed framework for building stronger, more generalizable psychological research and theory through behavioral modeling.

Differential item functioning (DIF) screening has long been suggested to ensure assessment fairness. Traditional DIF methods typically focus on the main effects of demographic variables on item parameters, overlooking the interactions among multiple identities. Drawing on the intersectionality framework, we define intersectional DIF as deviations in item parameters that arise from the interactions among demographic variables beyond their main effects and propose a novel item response theory (IRT) approach for detecting intersectional DIF. Under our framework, fixed effects are used to account for traditional DIF, while random item effects are introduced to capture intersectional DIF. We further introduce the concept of intersectional impact, which refers to interaction effects on group-level mean ability. Depending on which item parameters are affected and whether intersectional impact is considered, we propose four models, which aim to detect intersectional uniform DIF (UDIF), intersectional UDIF with intersectional impact, intersectional non-uniform DIF (NUDIF), and intersectional NUDIF with intersectional impact, respectively. For efficient model estimation, a regularized Gaussian variational expectation-maximization algorithm is developed. Simulation studies demonstrate that our methods can effectively detect intersectional UDIF, although their detection of intersectional NUDIF is more limited.

This article proposes a genetic algorithm, the array histogram-based sampling algorithm (AHBSA), to assemble linear multidimensional forced-choice questionnaires (MFCQs), which are increasingly popular for measuring personality, with forced-choice items including any number of statements (block size). The algorithm also works for traditional multidimensional Likert and cognitive test forms, where items can be seen as having a block size of one. Real and simulated statement pools are used to evaluate AHBSA’s performance in terms of test reliability and running speed in assembling Likert forms and MFCQs with two (pair) and three (triplet) statements. Compared to mixed integer programming (MIP) and random assembly, AHBSA achieves as high or higher reliabilities under all conditions: in Likert forms, AHBSA reaches optimal solutions as MIP does, and in the MFCQs, AHBSA gains notable increases in reliabilities over MIP when the forms have no constraint on item direction (i.e., positively versus negatively keyed statements). AHBSA takes more time to converge than MIP. The findings and limitations are discussed, and suggestions for future work are provided.

In psychometric sciences, such as social or behavioral sciences, and, similarly, in medical sciences, it is increasingly common to deal with longitudinal data organized as high-dimensional multidimensional arrays, also known as tensors. Within this framework, the time-continuous property of longitudinal data often implies a smooth functional structure on one of the tensor modes. To help researchers investigate such data, we introduce a new tensor decomposition approach based on the PARAFAC decomposition. Our approach allows researchers to represent a high-dimensional functional tensor as a low-dimensional set of functions and feature matrices. Furthermore, to capture the underlying randomness of the statistical setting more efficiently, we introduce a probabilistic latent model in the decomposition. A covariance-based block-relaxation algorithm is derived to obtain estimates of model parameters. Thanks to the covariance formulation of the solving procedure and thanks to the probabilistic modeling, the method can be used in sparse and irregular sampling schemes, making it applicable in numerous settings. Our approach is applied in the psychometric setting to help characterize multiple neurocognitive scores observed over time in the Alzheimer’s Disease Neuroimaging Initiative study. Finally, intensive simulations show a notable advantage of our method in reconstructing tensors.