This article proposes a mixed-effects machine-learning framework for modeling complex, nonlinear relations between predictors and continuous outcomes in multilevel cross-classified data. The proposed method, termed LMM–XGBoost, embeds extreme gradient boosting (XGBoost) within a linear mixed model (LMM) to combine flexible modeling of nonlinear and interaction effects with random effects that model dependence. In addition, an iterative estimation procedure for LMM–XGBoost is developed, a group-aware permutation importance measure that respects multilevel dependence is proposed, and a combined-group cross-validation (CV) strategy for hyperparameter tuning, out-of-fold (OOF) prediction, and importance estimation is developed for cross-classified designs. The simulation study shows that the proposed estimation method for LMM–XGBoost yields good parameter recovery under non-zero random-effect variances. In addition, relative to standard LMM and XGBoost, LMM–XGBoost achieves lower OOF prediction error and more accurate recovery of variable importance. The study further shows that combined-group CV and group-aware permutation importance yield less biased error estimates and substantially higher agreement with the true importance rankings than conventional permutation measures. An empirical application using the Add Health study illustrates how the proposed methods can identify important factors across multiple domains associated with adolescent depressive symptoms.

Test speededness, caused by time constraints, can impact examinees’ performance, leading to decreased response accuracy, particularly toward the end of the test. Most existing methods for detecting test speededness rely on specific distributional assumptions for response times (RTs), such as the lognormal distribution, which may lead to incorrect statistical inference if the true data distribution deviates from these assumptions. This article proposes a novel Bootstrap-CUSUM method for detecting test speededness, which is robust to non-normality in log-RTs. By constructing a cumulative sum (CUSUM) person-fit statistic for log-RTs and using the multiplier bootstrap to estimate its empirical distribution, our method facilitates individual-level detection and changepoint estimation. We prove the theoretical consistency of the method under both null and alternative hypotheses. Simulation studies show that the Bootstrap-CUSUM method outperforms the likelihood ratio test, Wald test, and score test in terms of correct classification rate, true detection rate, and false positive rate, demonstrating superior robustness and adaptability across different data distributions. The real data analysis further demonstrates the practical utility of the proposed method for detecting test speededness.

We propose a latent trait model for the responses and response times on tests that separates capability from persistence. Core of the model is a race between a diffusion process and a censoring process. The diffusion process represents item-level cognitive processing and determines the processing time of a test taker. The censoring process sets the maximal time a test taker is willing to invest into an item. If the processing time is shorter than the maximal time, the response is generated by the diffusion process; otherwise, the response is generated differently. In the first version of the model, the response is generated by a random guess. In the second version of the model, the response is determined by the actual level of the diffusion process. We relate the diffusion process to the capability and response caution of a test taker and the censoring process to his willingness to invest time. Similar to models for rapid guessing, the proposed models take account of disengaged responding, but allow for individual differences in persistence and informed guessing. The model also provides a mathematical specification of persistence. In a simulation study, we investigate model fitting by marginal maximum likelihood estimation. We also apply the model to two empirical data sets.

This article proposes a genetic algorithm, the array histogram-based sampling algorithm (AHBSA), to assemble linear multidimensional forced-choice questionnaires (MFCQs), which are increasingly popular for measuring personality, with forced-choice items including any number of statements (block size). The algorithm also works for traditional multidimensional Likert and cognitive test forms, where items can be seen as having a block size of one. Real and simulated statement pools are used to evaluate AHBSA’s performance in terms of test reliability and running speed in assembling Likert forms and MFCQs with two (pair) and three (triplet) statements. Compared to mixed integer programming (MIP) and random assembly, AHBSA achieves as high or higher reliabilities under all conditions: in Likert forms, AHBSA reaches optimal solutions as MIP does, and in the MFCQs, AHBSA gains notable increases in reliabilities over MIP when the forms have no constraint on item direction (i.e., positively versus negatively keyed statements). AHBSA takes more time to converge than MIP. The findings and limitations are discussed, and suggestions for future work are provided.

Not-reached (dropout) and omitted (intermittent missingness) behaviors are often inevitable in timed computerized tests. These missingness behaviors may be related to the subject’s latent traits, the difficulty of the item, or even the unobserved item response itself. In order to better understand the underlying test-taking behaviors, a Bayesian hierarchical framework is adopted to jointly model the item response, the item response time, the not-reached, and the omitted behaviors. For missing data, a sequential multinomial model is developed for the not-reached behavior, and a conditional model is proposed for the omitted behavior conditioning on the not-reached behavior. A decomposed logarithm of the pseudo marginal likelihood (LPML) is then developed to assess the fit of the missing data models. It can be further used to quantify the importance of modeling item response and response time jointly versus individually in identifying the missing data mechanism. The empirical performance of the proposed models and the model assessment criterion are examined through extensive simulations. The proposed methodology is further applied to analyze the Program for International Student Assessment (PISA) 2018 Test.

Classical symmetric association measures, such as correlation and chi-square indices, are widely used in applied psychology. However, these indices have limitations in identifying asymmetric implicative relationships. Standard regression analysis of Y on X, frequently interpreted as evidence of a directed dependence $X\to Y$, does not preclude the reverse direction ($Y\to X$). While various proposals in the literature have sought to provide non-symmetric association measures between binary events, most have overlooked the potential information in the contrapositive ($\bar {B}\to \bar {A}$), in addition to the main assertion ($A\to B$). When multiple variables are involved, asymmetric dependence is frequently represented as intricate dependency networks, which can be challenging to summarize and interpret in terms of higher-order clusters or latent dimensions. This article introduces a novel statistical implication index designed to address both limitations. The efficacy of this asymmetric index is demonstrated through its ability to detect one-way implication relationships, using both positive and contrapositive evidence. It also facilitates dimensional reduction by establishing aligned sets of nodes in a graph representation, under the condition that a Rasch model holds on these nodes, thus filling the gap between graphical and dimensional models. The efficacy of this index is substantiated through both simulated and real-world data illustrations.

Over 45 years ago, William Revelle proposed a reliability measure based on the worst split-half of a test or scale, commonly known as Revelle’s beta, to assess the general factor saturation. However, to this day, there is no reliable method for computing this measure, as existing approaches are either computationally infeasible or insufficiently accurate in identifying the worst split-half. This difficulty arises because the number of candidate splits increases exponentially with the number of items. In this article, we show that computing Revelle’s beta is conceptually equivalent to divisive (“top-down”) hierarchical clustering. This insight allows us to reduce the number of candidate splits to a quadratic problem, making the computation feasible. We specify theoretical conditions under which this approach is guaranteed to recover the worst split-half. To validate the efficiency of our approach, we conduct simulation studies and analyze real-world data. Code implementations accompanying this work are available online, together with Supplementary Material.

Recent studies on cognitive diagnostic models (CDMs) have extended the framework to longitudinal data. Various methods combined CDMs and hidden Markov models (HMMs) to assess changes in attributes over time and to evaluate the effects of interventions and covariates. A requirement for model fitting, inference, and interpretation is that models are identifiable. In this article, we derive identifiability conditions for experimental design used in education research. Specifically, we consider three designs: pretest/posttest single-group design, counterbalancing, and multiple-group longitudinal design. Drawing on existing HMM research, we examine the extent to which these designs satisfy common identifiability assumptions and propose new constraints for the counterbalancing and multiple-group longitudinal designs. These two setups are recommended in situations where item parameters differ over time or when there is heterogeneity in transition patterns across groups. We introduce a general HMM model for the multiple-group longitudinal design and a Gibbs sampling algorithm to estimate the parameters. We assess parameter recovery through a Monte Carlo simulation study and apply the model to a dataset from a study which evaluates the effectiveness of two interventions relative to a control condition. The results demonstrate the flexibility of the model and its potential to offer new insights into learning processes.

This study investigates the relationship between daily interpersonal stress (binary, time-varying) and suicidal behavior (binary, time-varying) using 90 days of daily diary data from 106 adolescents assessed immediately after discharge from acute psychiatric treatment. It addresses two key complexities: the rarity of suicidal events and non-monotone, non-ignorable missingness in both the outcome and the predictor. Because existing methods often fail to accommodate these complexities, leading to biased estimates, a Bayesian selection model is specified. The model integrates a mixed-effects complementary log–log regression for rare events with a missingness model that accounts for non-monotone, non-ignorable missingness in the outcome. A probit mixed-effects model is used for the time-varying predictor, along with a corresponding missingness model for its non-monotone, non-ignorable missingness. Empirical results support the applicability of the specified model to longitudinal studies involving rare events and complex missing-data structures. Furthermore, a simulation study demonstrates parameter recovery and highlights bias in focal parameters when sensitivity parameters in the outcome and missingness models are ignored.

Several textbooks give the incorrect formula to calculate the standard error estimates for standardized regression coefficients. As a remedy, two analytic methods have been developed: (1) the delta method and (2) the covariance structure modeling method. However, neither method is applicable to compute the standard error estimates for unstandardized regression coefficients of products of Z-scores. In the literature, a nonparametric bootstrap procedure is advocated to test the significance of unstandardized regression coefficients of products of Z-scores. In this article, we propose a simple analytic approach that can produce the standard error estimates not only for standardized regression coefficients when interaction terms are not included, but also for unstandardized regression coefficients when interaction terms of Z-scores are included. Two numeric examples are used to compare our analytic approach with the existing methods, and simulation studies are conducted to further evaluate the performances of regular regression, our analytic approach, and the nonparametric bootstrap procedure at finite sample sizes. It is found that (1) regular regression performs well only when the variances of predictor variables are small, (2) our analytic approach performs well at the sample size of 200 or larger, and (3) the nonparametric bootstrap procedure performs (almost) perfectly in all conditions.

Recent research shows that amortized variational inference (AVI) can be used to efficiently estimate high-dimensional latent variable models on large datasets. However, its use has remained limited to item response theory (IRT), and generalizing the approach to discrete latent variable models is not straightforward. We propose two ways to deal with this problem. In an initial simulation, we verify that these approaches can be used to estimate simple discrete latent variable models, such as latent class analysis and the generalized deterministic inputs, noisy and gate model. In these cases, AVI provides accurate parameter estimates, although the computational advantage over marginal maximum likelihood (MML) and standard variational inference (VI) is limited. We then apply the same approach to estimate mixture IRT models. In this case, AVI is computationally faster than MML estimation and standard VI. To demonstrate the practical applicability of our AVI approach, we use it to fit a seven-dimensional mixture IRT model to a narcissism inventory. Whereas quadrature-based methods cannot feasibly estimate models of this dimensionality, the efficient AVI approach even allows for computation of bootstrapped standard errors. We provide our code, along with an easy-to-use tool for fitting these models to new datasets.

Time-varying coefficient modeling (TVCM), which represents regression coefficients as smooth functions of continuous time, provides a flexible framework for uncovering complex patterns of change in levels and associations in intensive longitudinal data. However, conventional TVCM remains limited to investigating directional effects across individuals. By introducing a TVCM formulation of the multivariate normal distribution, the present study extends TVCM to explore change in undirected associations (couplings) and variability, thereby broadening its utility for psychological research. We discuss three versions of this approach: an aggregate-level model and two hierarchical versions capturing interindividual differences in unfolding change, either via person-specific intercepts accounting for onset differences or through fully person-specific coefficient functions smoothed via partial pooling. To illustrate the proposed developments, we apply them to six weeks of intensive longitudinal data from 16 anxiety patients undergoing therapy and examine unfolding changes in the level and volatility of nervousness and threat monitoring, their coupling, as well as between-person heterogeneity in each of these. We further show how inspecting first-order derivatives of the coefficient functions supports identifying periods of stability and change. Finally, we discuss extensions incorporating person-level characteristics to explain heterogeneity in patterns of change and predict outcomes.

Learning diagnosis is essential for effective education, with formative assessments shown to significantly enhance academic performance. Diagnostic classification models (DCMs) have been developed to assess students’ learning status and provide remedial instruction. However, the impact of mastery or non-mastery of specific attributes on long-term learning development remains uncertain. If certain non-mastered attributes hinder the growth of mathematical ability, early intervention becomes essential. In this study, we developed a random-effects DCM for multilevel growth curves (RDC–MGC) model to identify the specific effects of attribute mastery on individual-level mathematics ability growth. The simulation studies showed that the Bayesian estimation procedure provided appropriate parameter recovery and coverage probabilities, whereas ignoring the multilevel structure resulted in biased parameter estimates. The model was applied to arithmetic test data from second- to sixth-grade elementary school students. Diagnosis was conducted in the second grade, and the effects of mastery on mathematics ability growth from the third to sixth grades were assessed. The results showed that attribute mastery in second grade was associated with both the intercept and slope of individual ability growth, suggesting the potential importance of early-stage diagnostic information for understanding later mathematical development. Potential extensions of the proposed RDC–MGC model are also discussed.

ShinyItemAnalysis (SIA) is an R package and shiny application for an interactive presentation of psychometric methods and analysis of multi-item measurements in psychology, education, and social sciences in general. In this article, we present a new feature introduced in the recent version of the package, called “SIA modules,” which allows researchers and practitioners to offer new analytical methods for broader use via add-on extensions. SIA modules are designed to integrate with and build upon the SIA interactive application, enabling them to leverage the existing infrastructure for tasks, such as data uploading and processing. They can access and further use a range of outputs from various analyses, including models and datasets. Because SIA modules come in R packages (or extend existing ones), they can be bundled with their datasets, utilize object-oriented systems, or even compiled code. We illustrate the concepts using sample modules from the newly introduced SIAmodules package and other packages. After providing a general overview of building Shiny applications, we describe how to develop the SIA add-on modules with the support of the new SIAtools package. Finally, we discuss the possibilities of future development and emphasize the importance of freely available, interactive psychometric software for disseminating methodological innovations.

This article synthesizes popular approaches to Bayesian estimation of psychometric models that rely on the multivariate normal distribution, including point estimation of posterior central tendency, Gibbs sampling, and Hamiltonian sampling. We place emphasis on the geometry of the posterior distribution, on building intuition by drawing connections to regression, and on providing background about how relevant matrix results are obtained. We also consider how the results can be extended to handle related psychometric models, including popular item response models. The goal is to provide researchers with ideas and tools for designing their own Bayesian estimation methods, which we hope will lead to continued developments and improvements.