2 The proposed model

Building upon the Beta IRM (Noel & Dauvier, Reference Noel and Dauvier2007), we propose an extended Beta Mixture IRM to identify careless respondents in VAS data. Adapting concepts from previous confirmatory mixture models (Arias et al., Reference Arias, Garrido, Jenaro, Martinez-Molina and Arias2020; Kam & Cheung, Reference Kam and Cheung2023; Ulitzsch, Pohl, et al., Reference Ulitzsch, Pohl, Khorramdel, Kroehne and von Davier2022; Ulitzsch, Yildirim-Erbasli, et al., Reference Ulitzsch, Yildirim-Erbasli, Gorgun and Bulut2022; van Laar & Braeken, Reference van Laar and Braeken2022), we classify respondents into attentive ( $Z_i = 1$ ) and careless ( $Z_i = 0$ ) groups based on their response patterns’ alignment with measurement models formulated to capture different types of respondent behavior. The proposed model is expressed as follows (Figure 3):

(4) $$ \begin{align} \begin{aligned} f(x_{ij}) & = \pi_i \cdot f(x_{ij} \mid z_i = 1) + (1 - \pi_i) \cdot f(x_{ij} \mid z_i = 0) \\ X_{ij} \mid Z_i &= 1 \sim \operatorname{Beta}\left(\exp \left(\frac{w_j\theta_{id[j]} - \delta_j + \tau_j}{2}\right), \exp \left(\frac{-\left(w_j\theta_{id[j]} - \delta_j\right) + \tau_j}{2}\right)\right) \\ X_{ij} \mid Z_i &= 0 \sim \operatorname{Beta}(m, n), \\ \end{aligned} \end{align} $$

where $f(\cdot )$ denotes a general probability density function, and $x_{ij}$ represents the realization of the continuous bounded response from respondent i to item j. The latent variable $Z_i$ indicates class membership, where $Z_i=1$ denotes the attentive class, and $Z_i=0$ denotes the careless class, and $\pi _i=\Pr (Z_i=1)$ represents the probability that individual i belongs to the attentive class. The response distribution is a mixture of two components. The first component, $f\left (x_{i j} \mid z_i=1\right )$ , models responses for attentive individuals as a Beta distribution governed by the person’s latent trait levels $\theta _{id[j]}$ , item wording $w_j$ , item difficulty level $\delta _j$ , and the dispersion parameter $\tau _j$ . To facilitate disentangling careless from attentive responding, we recommend modeling raw responses obtained from both positively and negatively worded items (as in Kam & Cheung, Reference Kam and Cheung2023; Ulitzsch, Yildirim-Erbasli, et al., Reference Ulitzsch, Yildirim-Erbasli, Gorgun and Bulut2022; Vogelsmeier et al., Reference Vogelsmeier, Uglanova, Rein and Ulitzsch2024). Information on item wording is incorporated into the model through the parameter $w_j$ , which takes the value 1 or -1.

Figure 3

Model structure.

Note: For attentive respondents, responses are assumed to follow a Beta IRM, governed by item and person parameters; a two-factor, six-item structure is used as an example. For careless respondents, responses are assumed to stem from a common Beta distribution, with Beta(2, 5) used as an illustrative example in the figure.

The second component, $f\left (x_{i j} \mid z_i=0\right )$ , models responses for careless individuals using a Beta distribution with common parameters $\alpha _{ij}=m$ and $\beta _{ij}=n$ that generalize across all items and persons. This reflects the assumption that careless respondents do not process item content, and that, hence, the selected responses should neither be governed by respondent’s trait levels nor impacted by item characteristics.

The Beta distribution of the careless component is assumed to “absorb” different careless response patterns; hence, its parameters will reflect the marginal distribution over all types of careless response patterns in the population (see Ulitzsch, Pohl, et al., Reference Ulitzsch, Pohl, Khorramdel, Kroehne and von Davier2022), approximated by a Beta distribution. The unstructured Beta distribution assumed for the careless class offers a flexible means of capturing the marginal distribution of diverse careless behaviors in the population. For example, a $\text {Beta}(1,1)$ distribution suggests marginally uniform selection across the scale, $\text {Beta}(5, 5)$ indicates a marginal tendency to select around the midpoint, and $\text {Beta}(0.5, 0.5)$ reflects marginal preferences for the opposite ends of the slider (Figure 4).

Figure 4

Density plot for different Beta distributions.

We acknowledge that careless behaviors may take various forms and may differ across respondents (e.g., some may position their slider randomly, while others may move their slider toward one end of the scale). As such, the careless component model will most likely be misspecified. However, since the particular type(s) of careless responding displayed are typically not of substantive interest, we do not aim to disentangle possibly different subtypes of careless behavior or to interpret the internal structure of the careless class. Instead, the careless component of the mixture model is conceptualized as a residual class with minimal structure, designed to capture any response pattern that systematically deviates from attentive behavior.

For model estimation, we suggest Bayesian estimation due to its flexibility in handling complex models (Gelman et al., Reference Gelman, Carlin, Stern, Dunson, Vehtari and Rubin2013), with priors assigned to model parameters. These prior values can be informed by theoretical considerations and previous research, or, when prior knowledge is unavailable, can be set as diffuse. In this study, we apply diffuse priors. For the item-specific difficulty and dispersion parameters, $\delta _j$ and $\tau _j$ , we use normal priors with $\mathcal {N}(0,10)$ . The latent trait mean for the attentive group is fixed at zero for model identification. The multidimensional latent traits $\boldsymbol {\theta }_i$ are modeled as $\mathcal {N}_D(\mathbf {0}, \boldsymbol {\Sigma }_\theta )$ , where the means are fixed to zero for model identification. The covariance matrix $\boldsymbol {\Sigma }_{\theta }$ is decomposed into a correlation matrix $\boldsymbol {R}$ and a vector of standard deviations $\boldsymbol {\sigma }_{\theta }$ , such that $\boldsymbol {\Sigma }_{\theta } = \text {diag}(\boldsymbol {\sigma }_{\theta })\boldsymbol {R}\text {diag}(\boldsymbol {\sigma }_{\theta })$ . We place a half-Cauchy prior on each dimension of $\boldsymbol {\sigma }_\theta $ , i.e., $\sigma _{\theta ,d} \sim \text {Cauchy}^+(0, 5)$ , and a Lewandowski–Kurowicka–Joe (LKJ, Lewandowski et al., Reference Lewandowski, Kurowicka and Joe2009) prior on the correlation matrix: $\boldsymbol {R} \sim \text {LKJ}(1)$ . When $D = 1$ , the prior for latent variable simplifies to $\theta _i \sim \mathcal {N}(0, \sigma _\theta ^2)$ without correlation structure. For the parameters m and n, which shape the Beta distribution of careless response patterns, we employ a half-Cauchy distribution $\text {Cauchy}^+(0, 5)$ for accommodating a wide range of values. For the individual attentiveness and carelessness probabilities $\pi _i$ and $1-\pi _i$ , we employ a Dirichlet prior parameterized as $(\pi _i,1-\pi _i)\sim \text {Dirichlet}(\nu (\pi _{\mathcal{P}}, 1-\pi _{\mathcal{P}}))$ , where $\pi _{\mathcal{P}}$ represents the population-level proportion of attentive respondents and $\nu $ is a concentration parameter (Kemp et al., Reference Kemp, Perfors and Tenenbaum2007; Salakhutdinov et al., Reference Salakhutdinov, Tenenbaum and Torralba2012). For $\pi _{\mathcal{P}}$ and its counterpart $1-\pi _{\mathcal{P}}$ , we employ a diffuse Dirichlet prior $(\pi _{\mathcal{P} },1-\pi _{\mathcal{P} })\sim \text {Dirichlet}(1, 1)$ , implying a uniform distribution. For the concentration parameter $\nu $ , a half-Cauchy prior with location 0 and scale 5 is used.

3 Simulation study 1: Performance of the Beta mixture model

The aim of the simulation study was to evaluate the performance of the proposed model in two aspects: its accuracy in detecting careless respondents exhibiting different behavioral patterns and its estimation accuracy of model parameters of the attentive response model that are adjusted for the occurrence of careless responding.

3.2 Results

The model exhibited high convergence rates for all conditions ( $\geq 99\%$ ), except when the careless response pattern was overly consistent. In the overly consistent condition, convergence became increasingly difficult as the proportion of careless respondents rose. Convergence rates exceeded 89% when $1 - \pi _{\mathcal{P}} = 0.05$ to $0.15$ , but declined to 86% and 68% at $1 - \pi _{\mathcal{P}} = 0.25$ and $0.40$ , respectively. This might be due to the fact that the simulated overly consistent careless responses could not be well accommodated by a Beta distribution. To address convergence issues, we increased the number of MCMC iterations for the overly consistent condition with 40% careless respondents. The convergence rate substantially improved—from 68% to 97%—when the number of burn-in iterations was allowed to go up to 100,000. These results suggest that large proportions of careless responses following distributions that challenge the model also pose challenges for model convergence; however, this issue can be mitigated by increasing the number of iterations.

Estimation and classification accuracy were evaluated based on the converged replications. Tables 1 and 2 summarize the classification accuracy based on the threshold 0.5 and the population-level proportion of attentive respondents, respectively. The classification results showed excellent overall performance, with both the threshold-based and population-level proportion methods achieving high accuracy in classifying respondents as attentive or careless. However, when using the threshold method, the FPR reached unacceptably high levels when the proportion of careless respondents ( $1-\pi _{\mathcal{P}}$ ) was between 0.05 and 0.15. This suggests that when $1-\pi _{\mathcal{P}}$ is low, for most careless respondents, the estimated $\pi _i$ values exceeded 0.5, leading to their misclassification as attentive. Although overall accuracy remained high with consistently strong sensitivity and precision, this is likely due to class imbalance where the majority of respondents are attentive. Moreover, the FPR varied across data-generating scenarios, being low in overly consistent conditions, but higher in random and normal conditions. Under these conditions, the FPR reached as high as 0.477 and 0.456, respectively, when $1-\pi _{\mathcal{P}} = 0.05$ .

Table 1

Classification results based on the threshold

Note: Beta = Random responses generated from $\text {Beta}(0.5, 0.5)$ ; Preference = Overly consistent pattern; Normal = Random responses generated from a truncated normal distribution; The term $1-\pi _{\mathcal {P}}$ indicates the true population-level proportion of careless respondents; FPR = False positive rate; FNR = False negative rate.

Table 2

Classification results based on $\pi _{\mathcal {P}}$

Note: Beta = Random responses generated from $\text {Beta}(0.5, 0.5)$ ; Preference = Overly consistent pattern; Normal = Random responses generated from a truncated normal distribution; The term $1-\pi _{\mathcal {P}}$ indicates the true population-level proportion of careless respondents; FPR = False positive rate; FNR = False negative rate.

The high FPR observed under the single-threshold (0.5) approach may be attributed to the hierarchical prior structure and the high proportions of attentive respondents. When the two classes are more balanced (e.g., $\pi _{\mathcal{P}} = 0.4$ ), the FPR is generally within an acceptable range. In contrast, under imbalanced conditions where $\pi _{\mathcal{P}}$ is high, the hierarchical prior structure—with $\nu (\pi _{\mathcal{P}}, 1-\pi _{\mathcal{P}})$ serving as a hyperprior for $(\pi _i, 1 - \pi _i)$ —pulls individual-level estimates $\pi _i$ toward higher values (i.e., attentiveness), thereby diminishing the effectiveness of the 0.5 threshold in detecting careless respondents.

The classification method utilizing the population-level proportion ( $\pi _{\mathcal {P}}$ ) exhibited a slightly lower overall accuracy. Nevertheless, it maintained a high level of performance, especially when the proportion of careless respondents was low. Moreover, it provided significantly improved management of the FPR, with FPRs consistently remaining below 0.05 across most conditions, except in several cases where careless responses deviated from the Beta distribution, in which FPRs remained under 0.1. This method also maintained high sensitivity and precision, with sensitivity values ranging from 0.919 to 0.986 and precision consistently above 0.95. Given the ability of the population-level proportion method to better control the FPR while maintaining strong classification performance, we recommend using this approach when the proportion of careless respondents is low.

Table 3 displays results related to the estimation accuracy of the attentive item parameters, the standard deviation of the content trait, and the population-level proportion of attentive respondents. The attentive item parameters (difficulty $\delta _j$ and response variability $\tau _j$ ) were accurately estimated (e.g., high correlations between true and estimated values and small RMSEs). Likewise, the standard deviation of latent trait ( $\sigma _\theta $ ) was accurately estimated across all conditions, exhibiting very low relative bias and RMSE. To more closely evaluate latent trait estimation, we further examined the factor score estimates for attentive respondents under one selected condition ( $1 - \pi _{\mathcal {P}} = 0.15$ and careless responses were generated from $\text {Beta}(0.5, 0.5)$ ). We compared the proposed model with the Beta IRM, which does not account for careless responding. Results showed that both models achieved high correlations between estimated and true factor scores, with average correlations across replications of 0.959 for both the proposed model and the Beta IRM. However, the proposed model yielded lower RMSE values (0.288 versus 0.346), indicating it produces more precise estimates with less variability around the true factor scores.

Table 3

Estimation accuracy of item parameters and $\pi _{\mathcal {P}}$

Note: Beta = Random responses generated from $\text {Beta}(0.5, 0.5)$ ; Preference = Overly consistent pattern; Normal = Random responses generated from a truncated normal distribution; The term $1-\pi _{\mathcal {P}}$ indicates the true population-level proportion of careless respondents; RB = Relative bias; RMSE = Root mean square error.

The accuracy of the estimated population-level proportion of attentive respondents ( $\pi _{\mathcal {P}}$ ) depended on the data-generating distribution of careless responses. While the relative bias was essentially zero when careless responses were generated from a Beta distribution, we observed estimates to be downward biased in the other two conditions. That is, under conditions where careless responses cannot be perfectly accommodated by a Beta distribution, researchers might underestimate the proportion of attentive respondents—and, consequently, overestimate the proportion of careless respondents. For instance, when careless responses were generated with an overly consistent pattern, the average estimated proportion of careless respondents was 0.075 and 0.186 in conditions with data-generating proportions of 0.05 and 0.15, respectively. However, we note that the amount of bias was small, such that we deem the model to be sufficiently reliable to roughly gauge the extent of careless respondents in these scenarios. The estimation results of $\pi _{\mathcal {P}}$ also help explain the variation in classification accuracy across three careless response patterns (Table 2). Classification accuracy based on the population-level cutoff ( $\pi _{\mathcal {P}}$ ) was highest when careless responses were generated from a Beta distribution, likely due to the fact that $\pi _{\mathcal {P}}$ was estimated most accurately under this condition.

Figure 5, comparing data-generating distributions of careless responses and how these were approximated by the proposed model, provides intuition for the sources of variation in estimation accuracy across conditions. The parameter recovery results of these distributional patterns are also presented in Table 4. When careless responses were generated from a Beta distribution (Figure 5a), the distribution of careless responses was well recovered, as was to be expected. The parameter estimates of the Beta distributions closely matched the true values, with low RMSE indicating high estimation accuracy. Note that RMSE values for the shape parameters m and n could only be computed for the Beta condition, as these parameters are not defined under alternative generating distributions. When careless responses were not generated from a Beta distribution, the proposed model could only approximate but not fully replicate the true distribution of careless responses. For overly consistent responses generated from unif(0, 0.2) or unif(0.8, 1), the estimated distributions typically showed a U-shaped pattern, characterized by underestimated densities around the midpoint and overestimated densities near the scale boundaries (Figure 5b). The parameters of the Beta-approximated distribution showed great variability, as indicated by the large standard deviations in the parameter estimates. Moreover, when data were generated using a truncated normal distribution, the estimated distributions exhibited lighter tails compared to the true distribution (Figure 5c). The Beta approximation in this case resembled a $\text {Beta}(2.5, 2.5)$ distribution—symmetric around 0.5 and bell-shaped, bearing similarity to a normal distribution.

Figure 5

Beta approximation for careless responses.

Note: Beta = Random responses generated from $\text {Beta}(0.5, 0.5)$ ; Preference = Overly consistent pattern; Normal = Random responses generated from a truncated normal distribution. Blue lines indicate the Beta approximation for careless responses across all simulated datasets, while red dashed lines represent the distribution used for data generation.

Table 4

Parameter estimation for the careless response distributions

Note: Beta = Random responses generated from $\text {Beta}(0.5, 0.5)$ ; Preference = Overly consistent pattern; Normal = Random responses generated from a truncated normal distribution; The term $1-\pi _{\mathcal {P}}$ indicates the true population-level proportion of careless respondents; Mean = Averaged estimates across all converged replications; SD = Standard deviation of these estimates; RMSE is reported only for the Beta condition, where the true values of the Beta distribution are known.

4 Simulation study 2: Comparison with the mixture CFA model

As mentioned in the introduction, the unique characteristics of VAS data render previous mixture modeling approaches for careless respondent detection unsuitable, as their underlying assumptions (e.g., normality) may not hold for VAS data. To further demonstrate this, we conducted a simulation study to evaluate the limitations of a mixture CFA model for VAS data. Using the simulated datasets from Simulation Study 1, we applied a mixture CFA model (Roman et al., Reference Roman, Schmidt, Miller and Brandt2024; Zhang et al., Reference Zhang, Ulitzsch and Domingue2025) to the data:

(5) $$ \begin{align} \begin{aligned}f(x_{ij}) & = p_i \cdot \phi(x_{ij}|z_i=1) + (1 - p_i) \cdot \phi(x_{ij}|z_i=0) \\ X_{ij}|Z_i&=1 \sim \mathcal{N}(\mu_{j} + \lambda_j \cdot \theta_{i}, \sigma_{\epsilon,j}) \\ X_{ij}|Z_i&=0 \sim \mathcal{N}(\mu_{C}, \sigma_{C}), \end{aligned}\end{align} $$

where $\phi \left (\cdot \right )$ represents the normal density function. The item parameters $\mu _{j}$ , $\lambda _j$ , and $\sigma _{\epsilon ,j}$ represent intercepts, item loading, and residual standard deviations in the attentive group. It is important to note that, because the simulation design is based on a unidimensional model, $\lambda _j$ is a scalar rather than a vector. In the mixture CFA model, $\lambda _j$ is estimated using a half-normal prior with a mean of 0 and a standard deviation of 10. Its direction (positive or negative) is constrained based on the item’s wording. Careless responses are assumed to stem from a normal distribution $\mathcal {N}(\mu _{C}, \sigma _{C})$ with a common mean and standard deviation. The remaining simulation settings, such as the number of iterations, were kept consistent to ensure comparability.

The model convergence rate significantly declined under the overly consistent careless response condition compared to the proposed model (51% versus 93% when $1-\pi _{\mathcal {P}} = 0.15$ ). This decline is likely due to careless responses strongly deviating from normality, which might impair the mixture CFA model’s ability to differentiate the two groups. In contrast, for the other two conditions involving random careless responses, the model showed better convergence rates (over 80%).

Based on the converged replications, Table 5 summarizes the classification results of the mixture CFA model when $1-\pi _{\mathcal {P}} = 0.15$ . Notably, as $1-\pi _{\mathcal {P}}$ decreases, the classification performance worsens, with the FPR reaching 0.7 under the threshold method. For brevity, we focus on the conditions where $1-\pi _{\mathcal {P}} = 0.15$ . Table 5 highlights that the mixture CFA model fails to classify attentive and careless respondents effectively in terms of accuracy, FPR, and FNR. It is worth noting that when careless responding was generated using a truncated normal distribution, the mixture CFA model performed better than under other conditions. However, it still fell short of achieving acceptable levels of classification error rates. Although the generated and estimated distributions of careless responses were closely aligned in this scenario, the model’s limitations likely stem from a violation of its assumptions for the attentive group.

Table 5

Classification results of the mixture CFA model when $1-\pi _{\mathcal {P}} = 0.15$ .

Note: Beta = Random responses generated from $\text {Beta}(0.5, 0.5)$ ; Preference = Overly consistent pattern; Normal = Random responses generated from a truncated normal distribution; The term $1-\pi _{\mathcal {P}}$ indicates the true population-level proportion of careless respondents; FPR = False positive rate; FNR = False negative rate.

5 Empirical illustration

We applied the model to empirical data with three objectives: (a) to illustrate the practical application of the proposed model for pinpointing careless respondents, (b) to assess the impact of adjusting for careless respondents on parameter estimation, and (c) to compare the prevalence of careless behaviors between VAS and Likert data. The data used in this analysis come from a study that collected responses through two distinct scale formats—VAS and Likert—across three questionnaires, labeled A, B, and C, that differed in which scales and formats were administered (Kalistová & Cígler, Reference Kalistová and Cígler2021). These questionnaires measured two constructs: height, which captures individuals’ perceptions and experiences related to their physical stature (Tancoš, Reference Tancoš2018), and autonomy, which reflects the extent to which individuals feel free to make their own decisions and express their preferences (Johnston & Finney, Reference Johnston and Finney2010). We utilized data from questionnaire C to demonstrate the application of the proposed model, as it includes responses for both the height and autonomy scales in both VAS and Likert formats, allowing for a direct comparison between these response formats.

Questionnaire C is organized into four sequential blocks: 1) Height assessed using a VAS, 2) Autonomy assessed using a VAS, 3) Height assessed using a Likert scale, and 4) Autonomy assessed using a Likert scale. The Likert scale used in this study is a 5-point scale. The height scale contains 11 items (labeled h1–h11), of which 6 are negatively-worded, while the autonomy scale includes seven items (labeled a1–a7) with 3 negatively-worded items. Rows containing complete missing data in either the VAS or Likert scale were excluded from the analysis, resulting in a sample size of $N = 851$ .

We applied four models to analyze the data. For the VAS data, we used the Beta IRM (Noel & Dauvier, Reference Noel and Dauvier2007), which assumes all responses are attentive and incorporates two factors—height and autonomy. This was compared to the proposed model, which is designed to identify careless respondents. For the Likert scale data, we applied the generalized partial credit model (gPCM), which does not account for careless responding. We also included an ordinal mixture model (Ulitzsch, Pohl, et al., Reference Ulitzsch, Pohl, Khorramdel, Kroehne and von Davier2024), which was designed to detect careless respondents in ordinal data. The ordinal mixture model follows a similar approach as the proposed model to disentangle attentive from careless patterns; however, it is designed for ordinal data and utilizes the gPCM instead of the Beta IRM for the attentive class. For the careless class, the model estimates marginal category probabilities of inattentively choosing a given category over all types of careless behavior. Specifically, the probability of observing response category $X_{ij}=k \in \{0, 1, \dots , K\}$ is given by:

(6) $$ \begin{align} \begin{aligned} \Pr(X_{ij} & = k) = \pi_{i} \cdot \Pr(X_{ij} = k \mid Z_i = 1) + (1-\pi_{i}) \cdot \Pr(X_{ij} = k \mid Z_i = 0) \\ \Pr(X_{ij} & = k \mid Z_i = 1) = \frac{\exp\left( \sum_{l=0}^{k} \left(w_{j} \cdot a_j \cdot \theta_{id[j]} - b_{jl} \right) \right)}{\sum_{r=0}^{K} \exp\left( \sum_{l=0}^{r} \left(w_{j} \cdot a_j \cdot \theta_{id[j]} - b_{jl} \right) \right)} \\ \Pr(X_{ij} &= k \mid Z_i = 0) = \kappa_k, \quad \text{with} \sum_{k=0}^K \kappa_k=1, \end{aligned} \end{align} $$

where $a_j$ is the discrimination parameter, controlling how strongly the item differentiates between respondents with different trait levels and $w_j$ is a pre-specified wording parameter. The step difficulty parameter $b_{jl}$ governs the relative difficulty of selecting higher response categories. Note that $\sum _{l=0}^{0} \left (w_{j} \cdot a_j \cdot \theta _{id[j]} - b_{jl} \right ) \equiv 0$ . For more detailed information regarding the ordinal mixture model, please refer to Ulitzsch, Pohl, et al. (Reference Ulitzsch, Pohl, Khorramdel, Kroehne and von Davier2024).

Prior settings for the proposed model and Beta IRM were the same as in the simulation study. However, unlike the simulation study, the empirical study modeled two factors: height and autonomy. To estimate the correlation between these factors, an LKJ (Lewandowski et al., Reference Lewandowski, Kurowicka and Joe2009) prior with a shape parameter of 1 was used for the factor correlation matrix. For the ordinal mixture model and gPCM, we assigned weakly informative priors to the discrimination ( $\ln a_j \sim \mathcal {N}(0, 10)$ ) and threshold parameters ( $b_{jl} \sim \mathcal {N}(0, 10)$ ). The inattentive response probabilities $\boldsymbol {\kappa }$ are drawn from a Dirichlet prior with all parameters equal to 1. The priors for estimating the class probabilities in the ordinal mixture model were consistent with those specified for the proposed model in the simulation study. The variances of the latent traits were fixed to 1 for model identification in both the ordinal mixture model and gPCM. In all models used in the empirical study, two MCMC chains were generated with 10,000 iterations, with the first half designated as burn-in iterations. All models reached convergence, as indicated by EPSR values of less than 1.1 (Gelman, Reference Gelman1996).

5.1 Investigating careless responding in VAS data

The population-level proportion of careless respondents ( $1-\pi _{\mathcal {P}}$ ) for the VAS was 0.08, with a 95% credibility interval of [0.069, 0.092]. The parameters for the Beta distribution of careless responses, with $m = 0.388$ (95% CI: [0.357, 0.420]) and $n = 0.728$ (95% CI: [0.640, 0.823]), indicated that, marginally, careless respondents tended to select the left side of the slider scale (Figure 6).

Figure 6

Beta(0.388, 0.728): Model-implied distribution of careless responses in the empirical illustration.

Given results from the simulation study (Figures 1 and 2), we identified the careless respondents using $\pi _{\mathcal {P}}$ . We ranked the $\pi _i$ values and identified the least likely respondents as careless by selecting the bottom $\lceil (1-\pi _{\mathcal {P}}) \times N \rceil = 69$ individuals according to their individual probabilities.

Figure 7 compares the item correlations within and across the two scales for the attentive and careless groups. Responses from the attentive group showed strong correlations within both the height and autonomy scales. The negatively-worded items exhibited negative correlations with the positively worded items within the same scale. This demonstrates that attentive respondents answered according to item wording, aligning with the scale design. The correlations between items across the two scales were very low, highlighting the distinctiveness of the height and autonomy measures. Additionally, the low factor correlation that was not credibly different from zero further suggested the distinct separation between the height and autonomy constructs ( $\rho _{ha}$ in Table 6).

Figure 7

Inter-item correlations for attentive and careless groups for VAS.

Note: The item labels “h” and “a” represent the respective scales for “height” and “autonomy.”

Table 6

Summary of estimates for VAS data

Note: The 2.50% and 97.50% columns indicate the lower and upper bounds of the 95% credible interval. The symbols $\delta $ and $\tau $ represent item difficulty and dispersion, $\sigma $ and $\rho $ denote the standard deviations of the height and autonomy factors and the correlation between them, m and n indicate the Beta distribution shape for careless responses, and $1-\pi _{\mathcal {P}}$ is the estimated population-level proportion of careless respondents.

In contrast, the careless group displayed a more chaotic correlation structure. There were weaker and more inconsistent correlations within the same scale, and the negatively-worded items did not display the expected negative correlations. This suggests that careless respondents tended to ignore item wording and might not have distinguished between scales, leading to a breakdown in the expected correlation structure.

5.2 Impact of disregarding careless respondents on parameter estimates

Results of parameter estimates and credible intervals for the difficulty and dispersion parameters are provided in Table 6. After accounting for the careless respondents, the factor correlation ( $\rho _{ha}$ ) slightly increased from 0.032 (95% CI: [ $-$ 0.039, 0.106]) to 0.054 (95% CI: [ $-$ 0.029, .134]), though both estimates remained not credibly different from zero. Figure 8 displays a comparison of the estimates between the proposed model and the Beta IRM that does not take careless responding into account. Generally, both models yielded similar estimates for item difficulty parameters ( $\delta $ ) across the height and autonomy scales. However, the proposed model consistently yielded higher estimates for item dispersion parameters. As depicted in Figure 2, increased dispersion implies enhanced item informativeness, indicating an improvement in item psychometric properties upon accounting for careless responses. Figure 9 further compares the factor score estimates between the proposed model and the Beta IRM, plotted against person-level posterior attentiveness probability obtained from the proposed model. As can be seen, differences in factor score estimates were especially pronounced for individuals with lower attentiveness probabilities, while the two models yielded similar factor score estimates when the posterior attentiveness probability was high.

Figure 8

Item parameter estimates for VAS obtained from different models.

Note: The dashed line shows where the obtained estimates are equal; the labels “h” and “a” represent the item parameters of the “height” and “autonomy” scales.

Figure 9

Difference in factor score estimates.

Note: $\pi _i$ denotes the probability that person i belongs to the attentive group; the labels “h” and “a” represent the respective factors of “height” and “autonomy.”

5.3 Comparison between VAS and Likert scale

With a population-level estimate of 0.047 (95% CI [0.035, 0.061]), the proportion of careless respondents in the Likert scale data was lower than that observed in the VAS data. To evaluate whether the difference was indeed credibly different from zero, we estimated it as a derived parameter (0.034; [0.015, 0.052]). Partially mirroring results obtained for the VAS, marginally, careless respondents tended to prefer the lower two ( $\kappa _0 =$ 0.268, [0.160, 0.422]; $\kappa _1=$ 0.260, [0.136, 0.389]) as well as the middle upper categories ( $\kappa _3=$ 0.239 [0.178, 0.304]).

There was some overlap in the identification of careless respondents between the VAS and the Likert scale. Table 7 presents the contingency table for classification in the VAS and Likert scale data. While some respondents were identified as careless on only one of the scales, an overall agreement of 90.71% between the two scales reflects a high level of consistency in the classifications. Cohen’s $\kappa $ (0.23, 95% CI: [0.12, 0.34]) indicated a fair level of agreement between the Likert and VAS classifications (Cohen, Reference Cohen1960). A chi-square test suggested a significant association between the respondent classifications of attentive and careless across two scales ( $\chi ^2(1, N =851)= 44.615, p < 0.001$ ).

Figure 10 compares the individual probabilities of belonging to the attentive group across the two scales. The results showed a moderate correlation between the individual probabilities ( $r = 0.41$ ). Notably, if a classification threshold of 0.5 was applied, none of the respondents would be identified as careless for both scale formats. This finding aligns with the simulation results and suggests that the 0.5 threshold might have difficulty identifying careless respondents when the population-level proportion of such respondents is low.

Table 7

Contingency table for classification.

Figure 10

Comparison of individual probabilities ( $\pi _i$ ) between VAS and Likert scale.

Figure 11 shows the estimated threshold and discrimination parameters between the ordinal mixture model and gPCM. Consistent with the comparison between the proposed model and the Beta IRM (Figure 8), most of the threshold estimates remained very similar before and after accounting for careless responders. The discrimination parameters, however, increased after considering careless responses in the mixture model. Notably, negatively worded items exhibited a greater increase in discrimination estimates. Moreover, the factor correlation in the Likert scales also slightly increased from 0.039 (95% CI: [ $-$ 0.037, 0.113]) to 0.055 (95% CI: [ $-$ 0.017, 0.139]); however, both estimates remained not credibly different from zero.

Figure 11

Item parameter estimates for Likert scale obtained from different models.

Note: The dashed line shows where the obtained estimates are equal; the labels “h” and “a” represent the item parameters of the “height” and “autonomy” scales.