INTRODUCTION

One of the most frequently applied methods for assessing clinically meaningful change is the procedure of Jacobson and Truax (1991; McGlinchey et al., 2002). An inherent element of this procedure is the assessment of statistically reliable change by means of a so-called reliable change index (RCI). This index is widely known as the JT index, for historical reasons also referred to as the “classical approach” (Maassen, 2000a) since it is actually a reintroduction of a statistic established for some time in psychometric theory (Lord & Novick, 1968; McNemar, 1962). Over the past decade, a series of alternative reliable change indices have been proposed, but in recent articles (Maassen, 2000a, 2000b, 2001) the present author has argued that the advantages of some of those alternatives compared with the classical approach are more apparent than real. This led him to endorse Speer's (1999) plea for a moratorium on the use of adjusted RC methods and the use of the classical approach during this period (Maassen, 2001). Other authors (McGlinchey et al., 2002) agreed with this suggestion and argued for the need to focus on analyzing and improving the methods that exist.

The present article is an attempt to follow this advice and aims at clarifying the confusion associated with the JT index, particularly its standard error. In recent texts, the present author used to denote the standard error by the symbol σ_ED without further explanation of the way this quantity should be calculated, because this could be found in the earlier standard texts referred to (e.g., McNemar, 1969). However, it has since become clear that some researchers are uncertain as to which formula should be used. The point was recently explicitly raised in this journal by Hinton-Bayre (2000) in a letter to the editor responding to texts by Dikmen et al. (1999) and Temkin et al. (1999). The latter authors (Temkin et al., 2000), as well as Abramson (2000), replied to Hinton-Bayre's remarks, albeit very briefly. This article elaborates more thoroughly on this topic, clarifying the assumptions that underlie different forms of the standard error existing in the literature. It will be shown that Temkin et al.'s claim that they used the preferable formula is only theoretically true under restricted circumstances.

Suppose we try to assess the reliability of the change in a given person i using the classical approach. Although the use of this approach (or the JT procedure) is restricted to situations where practice effects can be ignored, it will be convenient in what follows to admit that person i has benefited from an effect π_i that may be related to his or her initial score, e.g., a practice effect as a consequence of the repeated testing. The difference score D_i observed in a given person i is regarded as an unbiased estimation of the true change. Since within the classical test theory true change is defined as the expected change under the current circumstances, it includes the practice effect. According to classical test theory it can be split into the following components: D_i = Y_i − X_i = Δ_i + π_i + E_Xi − E_Yi. X_i and Y_i respectively are the pretest and posttest score observed in person i. Δ_i, the true difference score excluding the practice effect, and π_i, the practice effect, denote fixed parameters of the theoretical score distribution of person i. Within the population of persons as well as within the theoretical score distribution of person i, the following usual assumptions of classical test theory are made: (1) both error components are mutually independent and independent of the true pretest and posttest scores; (2) both error components are normally distributed with zero mean; (3) the standard error of the difference of the two error components is equal for all participants. It should be noted that these assumptions on the error components are questionable in many clinical situations. However, since all the methods compared in this article are based on these assumptions, they will not be challenged within this context. (For situations where the assumption of equal standard deviations of the error components is not met, see, e.g., Mellenbergh & Van den Brink, 1998.) The variance of the observed difference is then equal to the standard error of measurement of the difference (squared), which will be denoted as Var(D_i) = σ_ED² and the statistic

has a standard normal distribution. This expression demonstrates that the practice effect in person i appears only in the numerator, assuming it to be fixed. For methods dealing with practice effects, the reader is further referred to Chelune et al. (1993), who presented an extension of the JT approach, or to Maassen (2003), McSweeny et al. (1993), and Temkin et al. (1999), who presented regression approaches. The statistic (1) boils down to the JT Reliable Change Index (RCI) under the null hypothesis of zero true change and the assumption of zero practice effect in person i:

Jacobson and Truax (1991) describe the Reliable Change Index as a psychometric criterion “that tells us whether the change reflects more than the fluctuations of an imprecise instrument” (p. 344). According to Christensen and Mendoza (1986), “RC determines if the pretest to posttest change score exceeds that which would be expected on the basis of measurement error” (p. 305).

How should the standard error (of measurement of the difference score) be calculated? The complete formula is provided by McNemar (1969):

This expression requires knowledge (preferably from external sources) of the standard errors of measurements of the initial and final testing. To circumvent these requisites, Jacobson and Truax (1991) assumed that pretest and posttest are parallel measures, and estimated the common reliability coefficient and the common variance by respectively the test–retest correlation and the variance of the initial testing in the particular study. The expression then simplifies to:

Jacobson and Truax (1991) denoted the standard error of their RCI as S_diff , following Christensen and Mendoza (1986), who pointed out that the standard error in the RCI earlier proposed by Jacobson et al. (1984) was wrong. Christensen and Mendoza used, however, the following expression:

which was called standard error of difference by the authors. This name, the notation S_diff and Equation (5) may induce misunderstandings among clinicians as we will show, but what was meant by Christensen and Mendoza (1986) is clear in their statement that it “represents the amount of difference which one could expect between two scores obtained on the same test by the same individual as a function of measurement error alone” (p. 307). And indeed, if this person has experienced no actual change, or if the actual change (possibly including a practice effect) is assumed to be fixed, then the only stochastic components of the test scores are the error components. If it is assumed that these components are uncorrelated and that their variances are equal across assessments, then S_diff² = 2.S_E². In practice, some authors (e.g., Temkin et al., 1999) have interpreted Expression (5) as referring to the standard deviation of the observed difference scores in the research group at hand. Hereafter, this interpretation will be referred to as Equation (5*).

If the parameter values required for Equation (3) can not be derived from an external source, another option would be to assume only that the reliability coefficients of pretest and posttest are equal, and to estimate this quantity and the variances of the pretest and posttest scores from the research group:

We now have three options for the calculation of the standard error of the RCI within a particular study: Expressions (4), (5*), and (6). What is the relation between these options?

First, it may be noted that Expression (6) is preferable to Expression (4). This is obvious in cases where it is not plausible to assume that the variances of the pretest and posttest scores in the research group are equal. We have already pointed out above that the assumptions underlying the use of Expression (4) imply equality of the variances. In that case the common variance would be better estimated by (S_x² + S_y²)/2, which is realized by Expression (6) (see also Abramson, 2000). Second, regarding Expressions (5*) and (6) we note a conceptual difference. Equation (5*) indicates what size of an observed change score should be regarded exceptional within the distribution of observed difference scores of the research sample at hand. Equation (6), as derived from Equation (3), indicates what size of an observed change score should be regarded exceptional within the theoretical distribution of a respondent's score distribution under the null hypothesis that no true change occurred. Algebraically, the difference between Expressions (5*) and (6) can be shown to be:

We see that Expression (5*) is never smaller than Expression (6) and that it exceeds Expression (6) to the extent that the test-retest correlation is high and the initial and final variance differ. How this difference can be conceptually explained, is revealed by a division of the variance of the observed differences into two components: a true variance and an error variance component S_diff² = Var(Δ + π) + S_ED².

Temkin et al. (1999) presented change outcomes observed within a normative sample, where no intervention was conducted and where it is plausible to assume that the true change of all the participants is equal to zero. Thus, any numerical difference between the standard errors (5*) and (6) is induced by differential practice effects. For example, if the practice effects are highly positively correlated with the pretest score, then the pretest and posttest scores will also be highly correlated, while the posttest variance will be greater than the pretest variance, a phenomenon commonly known as fanspread. This reveals the problem with Expression (5*). Differential practice effects increase the posttest variance as compared to the pretest variance, which is clearly accounted for by the standard error calculated according Expression (6). When using Expression (5*) an extra term is implicitly included that, once more, accounts for differential practice effects.

In order to gain more insight into the influence of the practice effects in actual research, we first examine theoretically the regression of the posttest on the pretest within the normative population. Let ξ_j and η_j respectively be the true pretest and posttest score of a person j belonging to the normative population, and let μ_ξc, μ_ηc and Π_c respectively be the population means of the true pretest scores, the true posttest scores and the practice effects. Then η_j = μ_ηc + β_c(ξ_j − μ_ξc) + ε_j holds, where index c refers to a normative population. Note that β_c should be distinguished from the observed regression coefficient b_c, the latter being attenuated by imperfect measurement of the initial score, which is expressed by: β_c = b_c /ρ_xx (McNemar, 1969, p. 173). The population means are linked as μ_ηc = μ_ξc + Π_c, which converts the regression equation into:

Formula (8) indicates how the practice effect of person j can be estimated from his pretest score. It can be seen that, if β_c = 1, there in no better estimation for all the practice effects in the normative population than the population mean Π_c. If β_c > 1, the estimation of the practice effect is proportional to the value of the pretest score (or Cov(ξ,π) > 0). Note that in this case regression from the mean (fanspread) may very well occur. If β_c < 1, the estimation of the practice effect and the pretest score are inversely proportional quantities (or Cov(ξ,π) < 0). In this case regression to the mean probably occurs, induced by practice effects rather than by measurement errors. Thus, when β > 1, as well as when β < 1, the initial and final variance are anticipated to be different, and, considering Equation (7), the standard errors (5*) and (6) are anticipated to be different accordingly.

To examine the practical implications of our discourse, we address the outcomes of two studies, that involve normative samples where no interventions have been conducted, those of McSweeny et al. (1993) and Temkin et al. (1999). In these studies, the authors expected practice effects to occur, which casts doubt on the assumption of equal variances. Hinton-Bayre (2000) has already presented a table, based on the outcomes of the Temkin et al. study, which compares estimations of the RCI interval width according to the Expressions (4) and (5*). In Table 1 the estimations according to Expression (6) are added,¹

as well as the outcomes of the McSweeny et al. study. Hinton-Bayre noted that his comparison yielded no clear pattern. This was to be expected since it can be algebraically shown (this will not be done here) that the sign of the difference of the estimations depends on the values of the parameters involved. The comparison of the outcomes of Expressions (5*) and (6) does show a pattern: Consistent with the derivations above, the standard error according to Expression (5*) is always highest. In general, the difference between the initial and final variance is small, and consequently the difference between the estimations of the standard error according to the Expression (5*) and (6) is small and sometimes negligible.

Table 1 shows that the posttest variance exceeds the pretest variance in every instance where β > 1 evidencing the phenomenon of fanspread. Regression to the mean, induced by practice effects, is also evidenced by Table 1 showing that the pretest variance exceeds the posttest variance in every instance where β < 1. Two examples are worth noticing. The steepest positive regression coefficient β (= 1.27) is found with regard to the Performance IQ in the McSweeny et al. study. Consequently, the posttest variance is seen to be considerably greater than the pretest variance, and the standard error calculated according to Expression (5*) is 8% greater than according to Expression (6), which is not negligible. The results for the Tactual Performance Total scale in the Temkin et al. study show even more dramatic differences. The two variances differ considerably and the standard error according to Expression (5*) is 25% greater than according to Expression (6). Temkin et al. (1999) anticipated the occurrence of practice effects in their study, implying that according to the classical approach with a 90% confidence interval they would find less “reliable deteriorations” and more “reliable improvements” than the 5% expected on the basis of chance. Indeed, Expression (5*) applied to the TPT Total scale yielded only 1% of the participants showing deterioration. On the other hand, the other scales yielded 7–20% of the participants showing improvement, whereas only 4% showed improvement as measured by the TPT Total scale. These are strong indications, of course, that the confidence interval was too wide.