What are IRT models?

The foundation of (unidimensional) IRT modelsFootnote ¹ † is the assumption that a ‘causal’ common latent variable underlies the responses to a set of scale items; thus, item responses are diagnostic of an individual's position on an underlying continuous latent variable. Having made that basic assumption, the next step in fitting an IRT model is to estimate the functional form linking levels on the latent variable to the probability of endorsing the item in the keyed direction (or with polytomous response formats, linking the latent variable with the probability of responding in each category). This process requires specification of a formal ‘measurement model’ of item responding. One common model applied to dichotomously scored personality items, such as the EY19, is the two-parameter logistic model (2PL) shown in equation (1).

In the above, individual differences on a continuous latent variable are denoted by theta (θ). For purposes of statistical identification, this latent variable is typically defined to have a mean of 0 and variance of 1 in the calibration population. In turn, each item is characterized by two properties: the slope of the item response curve (IRC) at the inflection point (α), and the location on the latent variable where the probability of responding to an item is 0.50 (β). Slopes in this model typically range between 0.7 and 2.0 with higher values (i.e. steeper IRCs) indicating more discriminating items. Location parameters, on the same metric as z scores, typically range between −2 and 2, with positive values indicating that higher levels of the latent variable are required to endorse the item content and negative values indicating that the item is relatively easy to endorse, even for individuals low on the latent variable.

Equation (1) makes clear that the probability of responding to an item in the keyed direction, the IRC, is determined by both the individual's level on the latent variable (θ) and item properties of discrimination and location (α,β). Item parameter estimates, along with traditional item statistics for the EY19, are shown in Table 2. These parameters were estimated using marginal maximum likelihood with the mirt library (Chalmers et al. Reference Chalmers, Pritikin, Robitzsch and Zoltak2015) available in the R program (R Development Core Team, 2015). While mirt was used for these analyses, other R libraries (i.e. irtoys: Partchev, Reference Partchev2015; ltm: Rizopoulos, Reference Rizopoulos2015) and commercial software (i.e. EQSIRT: Wu & Bentler, Reference Wu and Bentler2011; FlexMIRT: Cai, Reference Cai2013; IRTPRO: Cai et al. Reference Cai, Thissen and du Toit2011; Mplus: Muthén & Muthén, Reference Muthén and Muthén2012) are capable of performing the same analyses. For a review of commercial IRT software see Han & Paek (Reference Han and Paek2014).

Table 2.

Classical test statistic, factor loadings, and 2PL model item parameters

r, Raw item-test correlations; λ, factor loadings.

First, observe in Table 2 that, generally speaking, IRT parameters roughly correspond to their traditional counterparts. That is, items with higher item-test correlations correspond to higher slopes (or discriminations), and items with lower endorsement rates have more positive location parameters. Second, there is a lot of variation in the slope parameters indicating that the items differ importantly in their relation with the latent variable. Third, although not a great deal, there is some variation in the item location parameters with most items having endorsement rates <0.50 and positive IRT location parameters. As will be clear shortly, this importantly affects where on the latent variable continuum the EY19 provides measurement precision, that is, the greatest information.

For didactic purposes, Fig. 1 displays the IRCs for the most (item 3) and least (item 17) discriminating items, and for the item with the lowest (item 19) and highest (item 11) location parameters. Note that, although item 11 has the highest location parameter, it also has the lowest item-test correlation. Based on item content, ‘Would you agree that almost everything enjoyable is illegal or immoral?’, it is reasonable that factors other than impulsivity are influencing item endorsement. Items 3, 4, and 7 appear to be the most differentiating items. The content for these items involves not thinking before speaking, getting into a ‘jam’ because of not thinking before speaking, and a directly stated question, ‘are you an impulsive person?’ The least discriminating items tend to occur toward the end of the scaleFootnote ² , for example, items 17, 18, and 13. These items may be contaminated by other personality characteristics such as self-esteem (standing up for oneself if someone shouts) and spontaneity (an unplanned evening out).

Fig. 1.

Item response curves for items with the highest and lowest slopes (top two plots) and items with the highest and lowest location (bottom two plots).

Having estimated the item parameters, the next step is to consider how well these items measure individual differences on the underlying latent variable. In IRT, there is no notion of ‘scale score reliability’ or reporting of coefficient alpha internal consistency values, rather, what is critical is the contribution of each item in estimating an individual's position on the latent variable. This can be determined by translating the IRC for each item into an item information curve (IIC). In the case of the 2PL model this is shown in equation (2).

The IIC indicates how much psychometric information (i.e. reduction in uncertainty) an item provides at each level of the latent variable. Items with larger slopes provide more information or discrimination. The location of the IIC along the latent variable continuum is determined by the item location – items provide the most information at the location parameter. For illustrative purposes, the IIC for each item from Fig. 1 is shown in Fig. 2. Because of the assumption of unidimensionality (and local independence to be described below), IICs are additive. Thus, a test information curve (TIC) can be derived by simply summing the IICs. This is shown in Fig. 3. The amount of test information conditional on the latent variable is important because it is inversely related to the standard error of the maximum-likelihood estimate of the latent variable as shown in equation (3).

Fig. 2.

Item information curves for items with highest and lowest discrimination and location.

Fig. 3.

Test information curve and standard error for Eysenck Impulsiveness subscale.

This formula demonstrates that the more information at a specific point along the latent variable, the higher the measurement precision (i.e. lower standard error). Thus, a conditional information value of 4 is required to produce a standard error of 0.50, 9 is required for a standard error of 0.33, and 16 is required for a standard error of 0.25. For illustrative purposes, the standard error of measurement curve for the EY19 is also shown in Fig. 3. Clearly, this is a peaked information function suggesting that the items are especially good at differentiating among individuals who are around one standard deviation above the mean on the latent variable. Observe that the measurement precision is much lower below the mean of the latent variable (zero), precisely because there are no items with location parameters in that range. We believe that such findings not only inform about the EY19, but also potentially tell researchers something important about the construct, as we argue below.

Assumptions of IRT models and assessing fit

An important strength of IRT models, in part, is that they force a researcher to consider the assumptions underlying their application and to evaluate the degree to which the estimated model (i.e. the estimated item parameters) accounts for, reproduces, or ‘fits’ the data. In other words, IRT modeling forces researchers to study in detail how well the items serve as indicators of a latent variable and the overall quality of measurement across the latent variable continuum. This process of assumption checking and model-fit evaluation often reveals troubling concerns with an instrument, concerns that are easily overlooked in traditional scale analyses.

IRT models make three fundamental assumptions (Embretson & Reise, Reference Embretson and Reise2000). Evaluating these assumptions and judging the consequences of their violation, relies on application of complex statistics. We do not have the space here to elaborate on the plethora of available methods and their technical details but instead we hope to convey the major ideas.

First, because IRT models (i.e. the IRC) force a monotonically increasing relation between the latent variable and the probability of endorsing the item [see equation (1) and Fig. 1], the item response data must be monotonically increasing; as trait levels increase, so should the item endorsement rates. This assumption can be checked through the examination of rest-score functions (Meijer et al. Reference Meijer, Tendeiro, Wanders, Reise and Revicki2015). Rest-score functions are simply plots of the raw total score (minus the item score) and the item proportion endorsed within a given rest-score grouping.

In Fig. 4 we display the rest-score functions (and confidence bands) for all 19 items. These were computed using the mokken library in R (Van der Ark, Reference Van der Ark2014). These plots reveal that the rest-score functions for items 4 and 6 are ‘ideal’ in terms of applying a 2PL model – as rest-scores increase, the proportions endorsed start near zero, then increase in a systematic way across the entire rest-score range, and end up near 1.0. In contrast, the rest-score functions for item 11, appears troubling because response rates do not increase much as trait levels increase. Recall that this is the item with only 5% endorsement and a location above 4.0. In short, this item may be too extreme to be of any value in the measurement of impulsivity.

Fig. 4.

Rest-score functions for all 19 items. Items in order (1–19) from left to right.

Also troubling, in terms of applying the 2PL model, is that some items appear to require non-zero lower asymptote parameters, for example, items 13, 17, 18, and 19. These are all items with relatively low slopes and relatively high endorsement rates. On the other hand, many items, such as 1, 5, and 7, appear to require a non-one upper asymptote to model appropriately. For those items, regardless of how high an individual scores on the EY19, the probability of endorsing a statement appears to have an upper bound <1.0. These examples represent the problem of model ‘under-parameterization’ – the 2PL model does not properly capture the response process.

More complex models such as a 3PL (a model with a non-zero positive lower asymptote), a 3PLU (a model with a non-one upper asymptote), or 4PL (a model where both a lower and upper asymptote parameter are estimated) may be required (see Reise & Waller, Reference Reise and Waller2003, for discussion within the context of psychopathology measurement). The 4PL model is shown in equation (4).

where c is a lower asymptote parameter and d is an upper asymptote parameter setting limits on the lower and upper tails of the IRC, respectively. The 3PLU model fixes all the c parameters to zero and estimates d. The 3PL fixes all the d parameters to 1.0 and estimates c. If c is fixed to 0 and d is fixed to 1.0, this model reduces to the 2PL in equation (1).

For illustration purposes, we estimated 3PLU and 3PL models on the present data and the results are shown in Table 3. Clearly, for several items, estimating additional parameters changes the slope and locationFootnote ³ of the IRCs, and thus the IRC and IIC, and ultimately the test information and standard error. However, note that χ² tests based on comparison of log-likelihoods indicated that the 2PL is not a significant decrement in fit compared to the 3PL (χ² = 12.61, df = 19, p = 0.86) or 3PLU (χ² = 11.54, df = 19, p = 0.90) and thus, the 2PL should be preferred due to parsimony. Moreover, the mirt program yielded a warning that parameter estimates in the 3PL and 3PLU models may be unstable, and suggested imposing priors. In other words, for accurate parameter estimation these models may require larger sample sizes than are presently available.

Table 3.

Item parameter estimates for the 3PLU, 3PL, and log-logistic models

3PLU is a three-parameter logistic model with upper asymptote; 3PL is a three-parameter logistic with lower asymptote; α, item discrimination; β, item location; c, lower asymptote; d, upper asymptote; η = α; λ = exp(−αβ).

The second and third assumptions are highly inter-related, but can be considered as distinct concepts and empirically evaluated separately. Specifically, item response models assume unidimensionality and local independence. Unidimensionality means that the correlations among the items can be explained by a single common factor that represents the target construct a researcher designed the instrument to assess (impulsivity in the present case). The problem is that no psychological data that contains meaningful content heterogeneity will strictly meet this assumption. Thus, researchers typically examine item response data for evidence of ‘essential’ unidimensionality – the data are close enough to unidimensionality such that the item parameters are relatively unbiased reflections of the relation between the target latent variable and the item responses, and examinee's theta estimates reflect individual differences on the intended latent variable (i.e. are not overly biased by multidimensionality).

One of many ways to assess essential unidimensionality is the evaluation of the relative size of eigenvalues. In the present data, for example, the first five eigenvalues are 6.2, 1.2, 0.6, 0.4, and 0.3 suggesting that the first factor is much stronger than the additional common factors by a factor of roughly 6:1. Moreover, when the data were factor analyzed, no interpretable 2-, 3-, or 4-factor models could be identified. Alternatively, IRT estimation programs such as mirt also report more familiar model fit indices for the unidimensional model that are similar to those routinely used in structural equation modeling (SEM; see Maydeu-Olivares et al. Reference Maydeu-Olivares, Cai and Hernández2011). In the current data, we found: RMSEA = 0.056 (0.052–0.061), CFI = 0.917, and SRMSR = 0.055. Judging by conventional standards, collectively, these vales indicate a good fit of a one-dimension model. Note, however, that the generalizability of SEM fit benchmarks to an IRT context is questionable.

Finally, local independence (LI) is the assumption that, after controlling for the latent variable, the correlations among items are zero. This indeed is also the technical definition of unidimensionality (i.e. when responses are locally independent after controlling for a single factor). As noted above, however, it is still possible for an instrument to be essentially unidimensional but the single latent factor does not reproduce the item responses exactly. This occurs commonly in personality and psychopathology scales due to redundant item content. When item content is too redundant, the correlation among those items may be inflated (due to sharing variance from a general and item specific factors). In turn, this may ‘pull’ the latent variable toward the item pair with the inflated correlation resulting in overestimated slope parameters (see also Steinberg & Thissen, Reference Steinberg and Thissen1996).

In the present data, we searched for local independence violations by computing the Chen & Thissen (Reference Chen and Thissen1997) index, again using mirt. This index compares the observed response proportions in each 2 × 2 contingency table between item pairs, with those predicted based on the estimated model parameters. The values can be interpreted like z scores where higher positive values indicate large positive residuals and large negative values indicate a large negative residual. We will only be concerned here with the large positive values, say, when the index is >10.

By this criterion, the most problematic item pair is items 16 and 5. A glance at Table 1 reveals these to be essentially the same question asked in slightly different ways (basically, ‘do you think carefully’ before acting). Other potential problem pairs are items 1 & 4, 2 & 7, 6 & 4, 12 & 8, 17 & 16, and 17 & 19. Although not obviously overly content redundant, there may be contingencies between these item pairs that are not accounted for by the unidimensional IRT model. These LI violations are important because if an item from one of these pairs were to be removed from the scale, the item parameters for the other items may change. If there were no LI violations, not only would the data be unidimensional in a strict statistical sense, the item parameter estimates would be invariant, that is, their values would not depend on what other items are included in the scale.

The above analyses are typically conducted either prior to estimating an IRT model (e.g. monotonicity assessment) or immediately after fitting a hypothesized model (LI evaluation). After a model has been fitted, however, a researcher must also conduct some empirical investigations of item fit. The evaluation of item fit remains a contentious issue in the literature and there is no current consensus on how best to proceed. Nevertheless, for illustrative purposes here we review a standard χ² and graphical approaches for judging item fit.

To judge statistical fit, we reviewed chi-square item-fit values output from mirt based on a formula presented in Orlando & Thissen (Reference Orlando and Thissen2000, Reference Orlando and Thissen2003). The computation of these indices is complex because it involves an iterative or recursive formula. Simply stated, these indices are based on comparison of the number of individuals endorsing each item conditional on the overall composite raw score, compared with the proportion predicted based on the estimated IRT model parameters. These χ² indices are notoriously powerful, but in the present data, only items 1 and 2 produced the warning flags of statistical significance below p < 0.05.

To follow-up on these results, Fig. 5 presents ‘fit plots’. Specifically, for each individual, their location on the latent trait is estimated. Individuals are then grouped into, say 10, intervals along the theta continuum. For each interval, the observed response proportion is computed and compared to the estimated IRC. In the left panel is the fit plot for item 4 which had the best fit as judged by χ². Clearly, the IRC does an excellent job of representing observed response proportions – all the confidence bands contain the IRC. Interestingly, the right panel displays the fit plot for the worst fitting item (item 1). Here, the problem seems to lie in the tendency for the estimated IRC to overestimate the observed response proportions around the mean theta, and then underestimate them for trait levels above the mean. Again though, despite statistical significance, the empirical fit is certainly ‘in the ballpark’ and we would not be overly concerned that the estimated item parameters for item 1 are greatly in error.

Fig. 5.

Item fit plots for the best (item 4) and worst (item 1) items.

The applications of IRT

The above establishes that, for the most part, the EY19 is amenable to a 2PL model. The three concerns are: (a) some items do not contribute meaningfully to the measurement of the latent variable, (b) several item pairs may display LI violations, possibly inflating the correlation among those item pairs and thus distorting model parameter estimates (and thus test information and standard error), and (c) many of the items appear to require a more complicated IRT model, such as a 3PLU model where the upper asymptote of the IRC is estimated, rather than assumed to be 1.0 as in the 2PL model. What exactly to do about these concerns would require extended discussion and possibly some simulations to judge the impact of any model misspecification.

There have been many texts and articles written comparing IRT psychometrics with traditional approaches. These articles typically emphasize four virtues of IRT modeling: (a) scale analysis and construction, (b) linking the scales from multiple measures of the same construct, (c) assessing differential item functioning (measurement invariance), and (d) administering tests via computerized adaptive testing (CAT). Each of these topics has its own large literature and we can only try to convey the major ideas here.

What are the challenges to applying IRT models?

IRT models are attractive for many reasons as described above. In previous writings, however, we have urged caution in applying latent variable modeling techniques in general, and IRT modeling in particular in personality, psychopathology, and health outcomes domains (Reise & Waller, Reference Reise and Waller2009). It should be noted that IRT methods were developed almost entirely within the context of large-scale educational/achievement assessment to meet the pressing needs of this industry. At a minimum, unlike typical performance measures, educational measurement specialists seldom are concerned with response styles, self-deception, acquiescence, and so on. For this and many other reasons, the transition of IRT models and methods to other construct domains is not always that simple. Below we review a few of these conceptual and empirical challenges.

Approximately twenty-five years ago Bollen & Lennox (Reference Bollen and Lennox1991) demonstrated that the type of psychometric interpretation implicit in IRT-type models (i.e. latent variable measurement) critically depends on the latent variable being a ‘source’ of item responses. When constructs are products of the indictors (items), such as ‘social economic standing’ or ‘quality of family functioning’, the latent variable changes depending on which particular items are included on a measure. For instance, Fayers & Hand (Reference Fayers and Hand2002) note that quality of life scales often include items that function as causal indicators, that is, the concept being measured is defined by the items. With this latter type of construct, where items are ‘causal’, the entire logic and mathematics of IRT falls apart. Unfortunately, sometimes the IRT hammer appears to be applied to every possible nail without any consideration of the nature of the latent variable.

More recently, Reise & Revicki (Reference Reise and Revicki2015), for example, have argued that educational constructs and health outcomes differ, typically, in a variety of ways including: (a) the interpretation of individual differences on the construct (i.e. unipolar v. bipolar), (b) the expected distribution on the latent variable (e.g. normal v. positively skewed), and (c) the conceptual bandwidth of constructs or, stated differently, the diversity of trait manifestations. In what follows, we briefly review these concerns and their implications for the application of IRT to the measurement of psychiatric constructs.

Consider the case of developing a measure of a broad bandwidth construct such as verbal ability in adults. In particular, consider three reasonable expectations: (a) there is a nearly infinite number of reading comprehensive paragraphs, vocabulary, analogies, grammar items, and so on from which to build a pool of items, (b) a normal distribution is likely adequate to describe individual differences, in the population, and (c) both ends of this distribution are meaningful, and psychologically interpretable, that is, the construct is bipolar. This is an ideal situation to apply IRT models because we can expect to find items that vary in location parameter across the entire trait range and thus have good measurement precision across that trait range, which in turn makes CAT a feasible option.

Moreover, although IRT makes no formal normality assumption, item parameters are typically estimated assuming a normal prior for the latent variable. Violating this assumption (i.e. a misspecification) can only lead to biased parameter estimates. Estimation methods that potentially remedy the parameter bias that occurs when the latent variable is misspecified (Woods, Reference Woods2006) are only now emerging in commercial software (Monroe & Cai, Reference Monroe and Cai2014). Alternatively, new IRT modeling approaches that are specifically designed to account for skewed latent trait distributions are currently being researched, for example, see Molenaar (Reference Molenaar2014). It will be interesting to see how these new psychometric developments ultimately change the way IRT models are applied to psychiatric data in future years.

Now consider the measurement of ‘impulsivity’ as exemplified by the EY19 or measures of psychopathology more generally. Although the trait arguably manifests across a wide range of behavioral domains, once one has asked a few questions about thoughtless and reckless behavior (e.g. spending), lack of cognitive mediation in decision making (e.g. deliberates carefully before taking action, makes up mind quickly), and labile attention and interests (frequently changes interests or hobbies), the pool of content quickly runs dry. This is certainly not an ideal situation for the development of an item pool for the administration of a CAT; if even feasible, it appears to us that such a pool would contain many overly content redundant items.

It also, to us, remains debatable whether impulsivity, as least as measured by the EY19, is truly a bipolar construct with scores that are interpretable across the entire latent variable continuum, from low (reflecting ‘self-control’ or ‘conscientiousness’) to high (reflecting ‘impulsiveness’). We believe that impulsivity may be more of a quasi-trait (see Reise & Waller, Reference Reise and Waller2009), or what Lucke (Reference Lucke, Reise and Revicki2015) has termed a unipolar trait – definable and meaningful only at one end of the continuum (e.g. pathological gambling). In other words, we speculate that only variation among high scores reflect anything substantively meaningful, while low scores reflect the mere absence of impulsivity, not necessarily high degrees of self-control.

Observe that some evidence of the quasi-trait nature of impulsivity is to be found in the present results. First, in the EY19, and in other self-report measures we have examined, one cannot find a good range of item location parameters. We note that this concern has been raised by Reise & Waller (Reference Reise and Waller2009) as being generally true in psychopathology measurement. Second, and related to the first, is that scores on the EY19 are not normally distributed, either raw or latent trait estimates, and are, in fact, highly positively skewed. If one takes the next step and entertains the idea of a unipolar trait of impulsivity, then one analytic approach that may be taken is to fit a zero-inflated mixture IRT model (Wall et al. Reference Wall, Park and Moustaki2015).

In this type of model, a latent class is defined to be a ‘no trait’/‘no symptoms’ or ‘normal’ group and the percentage of such cases in the population is estimated. Then, assuming normality, one or more latent classes are identified to represent ‘traited’, ‘symptomatic’, or ‘clinical’ groups, and IRT item parameters are estimated based only on this latter latent class. This model may be ideal for the assessment of constructs where a researcher believes that low and zero scores are essentially meaningless, while variation in higher scores is meaningful and valid. This model, however, was developed for data where there are many more zero scores than observed in the EY19 data, and thus we will not illustrate an application.

Alternatively, in the presence of high skew, instead of assuming normality, a more reasonable latent trait distribution may be the log-normal (where zero is the lowest latent trait score possible, low scores reflect an absence of the trait and high scores reflect severity of pathology relative to absence of the disorder). Moreover, a more appropriate IRT model may be the log-logistic instead of the logistic (see Lucke, Reference Lucke, Millsap, van der Ark, Bolt and Woods2014, Reference Lucke, Reise and Revicki2015, for technical details). The log-logistic model is shown in equation (5).

where λ is a multiplicative parameter with higher values shifting the IRC to the right, and η is an item discrimination parameter.

This model can be estimated using Bayesian methods as described in Lucke's research, but for simplicity, observe that ‘naive’ parameters can be found as simple transformations of the parameters from the 2PL model. Specifically, η = α and λ = exp(− αβ); theta estimates in the log-logistic can be found as exp(θ _2PL) = θ _LL. The item parameter values are shown in Table 3 and the corresponding IRCs for the 19 items are shown in Fig. 7. Compared to a logistic model [equation (1)], in the log-logistic [equation (5)] individual differences at the high end of the trait are expanded and individual differences at the low end are contracted. This means that for individuals scoring low on the measure, theta estimates are rather homogeneous and near zero. For individual's scoring relatively high on the measure, theta estimates are more spread out.

Fig. 7.

Item response curves for the EY19 items in the log-logistic model.

At this point, we are not prepared to argue one way or another in regards to the correct latent variable measurement model for impulsivity, or other psychiatric constructs. This modeling issue, however, is important to raise here not only to motivate researchers to consider the continuous v. quasi-continuous nature of constructs, but also to point out that typically IRT modeling and associated applications may not work well in particular construct domains. This may be especially true in psychiatric domains where the construct is unipolar, the latent variable highly positive skewed, and the range of possible trait indicators very narrow.