3.1 Marginal model and covariance structure

In general, Equation (3.4) can be expressed as a vectorized expression for each person i as follows:

(3.5) $$ \begin{align} \boldsymbol{y}_i = \boldsymbol{X}_i\boldsymbol{\beta}_i + \boldsymbol{Z}_i^{(s)}\boldsymbol{s}_i + \boldsymbol{Z}_i^{(d)}\boldsymbol{d}_i + \boldsymbol{Z}_i^{(\nu)}\boldsymbol{\nu}_i + \boldsymbol{\omega}_i + \boldsymbol{\varepsilon}_i, \quad i = 1, \dots, N, \end{align} $$

where $\boldsymbol {y}_i = \left ( y_{i11}, \dots , y_{i1M}, y_{i21}, \dots , y_{iDM} \right )^\top $ is a vector with all observed measurements for person i. $\boldsymbol {X}_i$ , $\boldsymbol {Z}_i^{(s)}$ , $\boldsymbol {Z}_i^{(d)}$ , and $\boldsymbol {Z}_i^{(\nu )}$ are the design matricesFootnote ⁵ for the fixed effect (vector) $\boldsymbol {\beta }_i = \beta _0$ and the person random effect (vector) $\boldsymbol {s}_i = s_i$ , person random effect (vector) $\boldsymbol {d}_i = (d_i, \dots , d_D)^\top $ , and the day-level serial effect (vector) $\boldsymbol {\nu }_i = (\nu _{i1}, \dots , \nu _{iD})^\top $ , respectively. $\boldsymbol {X}_i$ and $\boldsymbol {Z}_i^{(s)}$ are a column vector of ones with length $DM$ , and $\boldsymbol {Z}_i^{(d)}$ and $\boldsymbol {Z}_i^{(\nu )}$ are $\underset {D \times D}{\boldsymbol {I}} \otimes \underset {M \times 1}{\boldsymbol {1}}$ , where $\boldsymbol {I}$ is an identity matrix with diagonal elements equal to one, and zero elsewhere, and $\boldsymbol {1}$ is a column vector of ones. The operator $\otimes $ denotes the Kronecker product. Under the balanced design assumption, everyone has the same number of measurements, so the design matrices $\boldsymbol {X}_i$ , $\boldsymbol {Z}_i^{(s)}$ , $\boldsymbol {Z}_i^{(d)}$ , and $\boldsymbol {Z}_i^{(\nu )}$ are the same for all persons i. The remaining vectors $\boldsymbol {\omega }_i = (\omega _{i11}, \dots , \omega _{i1M}, \omega _{i21}, \dots , \omega _{iDM})^\top $ and $\boldsymbol {{\varepsilon }}_i = ({\varepsilon }_{i11}, \dots , {\varepsilon }_{i1M}, {\varepsilon }_{i21}, \dots , {\varepsilon }_{iDM})^\top $ collect moment-level serial effects and measurement errors, respectively. Compared to a standard LMM formulation, Equation (3.5) explicitly distinguishes the serial processes $\boldsymbol {\nu }_i$ and $\boldsymbol {\omega }_i$ from the measurement error $\boldsymbol {{\varepsilon }}_i$ , to stress that the fluctuation of within-person observations could come from two personal dynamical processes instead of coming from the measurement random errors (Diggle, Reference Diggle1988; Ma et al., Reference Ma, Mermelstein and Hedeker2020; Verbeke & Molenberghs, Reference Verbeke and Molenberghs2000).

The total (co)variance of the marginal model (Equation (3.5)), denoted $\boldsymbol {V} = \operatorname {\mathrm {Var}}(\boldsymbol {y}_i)$ , is the sum of the covariance matrices of the random components:

(3.6) $$ \begin{align} \begin{aligned} \boldsymbol{V} &= \operatorname{\mathrm{Var}}(\boldsymbol{y}_i) \\ &= \boldsymbol{Z}_i^{(s)}\operatorname{\mathrm{Var}}(\boldsymbol{s}_i)\boldsymbol{Z}_i^{(s)\top} + \boldsymbol{Z}_i^{(d)}\operatorname{\mathrm{Var}}(\boldsymbol{d}_i)\boldsymbol{Z}_i^{(d)\top} + \boldsymbol{Z}_i^{(\nu)}\operatorname{\mathrm{Var}}(\boldsymbol{\nu}_i)\boldsymbol{Z}_i^{(\nu)\top} + \operatorname{\mathrm{Var}}(\boldsymbol{\omega}_i) + \operatorname{\mathrm{Var}}(\boldsymbol{{\varepsilon}}_i)\\ &= \boldsymbol{\Sigma}_S + \boldsymbol{\Sigma}_D + \boldsymbol{\Sigma}_\nu + \boldsymbol{\Sigma}_\omega + \boldsymbol{\Sigma}_E. \end{aligned} \end{align} $$

Each component in this decomposition represents a distinct source of variation: $\boldsymbol {\Sigma }_S = \sigma _s^2 \underset {DM \times DM}{\boldsymbol {J}}$ , where $\boldsymbol {J}$ is a matrix with all elements equal to 1, represents the person-specific covariance matrix, accounting for the between-person variation between their effects; $\boldsymbol {\Sigma }_D = \sigma _d^2 \underset {D \times D}{\boldsymbol {I}} \otimes \underset {M \times M}{\boldsymbol {J}}$ represents the day random effect variance; $\boldsymbol {\Sigma }_{\nu } = \tau _d^2 \underset {D \times D}{\boldsymbol {H}_d} \otimes \underset {M \times M}{\boldsymbol {J}}$ represents the covariance between day-level serial process, following the AR(1) structure; $\boldsymbol {\Sigma }_{\omega } = \tau _m^2 \underset {D \times D}{\boldsymbol {I}} \otimes \underset {M \times M}{\boldsymbol {H}_m}$ represents the covariance between moment-level serial process; and, lastly, $\boldsymbol {\Sigma }_E = \underset {D \times D}{\boldsymbol {I}} \otimes \sigma _{\varepsilon }^2\underset {M \times M}{\boldsymbol {I}} = \sigma _{\varepsilon }^2 \underset {DM \times DM}{\boldsymbol {I}}$ .

The distinction between the day-related components, $\boldsymbol {\Sigma }_D$ and $\boldsymbol {\Sigma }_\nu $ , becomes clearer when examining their underlying structures at level 2 (see Equation (3.2)) and examining their respective covariance matrix structures:

$$ \begin{align*} \operatorname{\mathrm{Var}}(\boldsymbol{d_i}) = \sigma_d^2 \underset{D \times D}{\boldsymbol{I}} = \begin{pmatrix} \sigma_{d}^2 & 0 & \cdots & 0 \\ 0 & \sigma_{d}^2 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \sigma_{d}^2 \end{pmatrix} \quad \text{ and } \quad \boldsymbol{H}_d = \begin{pmatrix} 1 & \phi_d & \cdots & \phi_d^{D-1} \\ \phi_d & 1 & \cdots & \phi_d^{D-2} \\ \vdots & \vdots & \ddots & \vdots \\ \phi_d^{D-1} & \phi_d^{D-2} & \cdots & 1 \end{pmatrix}. \end{align*} $$

$\boldsymbol {H}_d$ depicts the autocorrelation of daily fluctuation based on the AR(1) process. In contrast, $\operatorname {\mathrm {Var}}(\boldsymbol {d_i})$ captures the residual variance of these daily fluctuations that is not captured by the serial correlation.

Similarly, the difference between $\boldsymbol {\Sigma }_{\omega }$ and $\boldsymbol {\Sigma }_{E}$ , both at level 1 (see Equation (3.1)), can be understood by examining their covariance matrix structures:

$$ \begin{align*} \boldsymbol{H}_m = \begin{pmatrix} 1 & \phi_m & \cdots & \phi_m^{M-1} \\ \phi_m & 1 & \cdots & \phi_m^{M-2} \\ \vdots & \vdots & \ddots & \vdots \\ \phi_m^{M-1} & \phi_m^{M-2} & \cdots & 1 \end{pmatrix} \quad \text{ and } \quad \operatorname{\mathrm{Var}}(\boldsymbol{{\varepsilon}}_{ij}) = \sigma_{{\varepsilon}}^2 \underset{M \times M}{\boldsymbol{I}} = \begin{pmatrix} \sigma_{{\varepsilon}}^2 & 0 & \cdots & 0 \\ 0 & \sigma_{{\varepsilon}}^2 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \sigma_{{\varepsilon}}^2 \end{pmatrix}. \end{align*} $$

In this context, $\boldsymbol {H}_m$ depicts the autocorrelation of momentary fluctuation based on the AR(1) process, while $\operatorname {\mathrm {Var}}(\boldsymbol {{\varepsilon }}_{ij})$ captures the residual variance at each moment that is not explained by the serial correlation or other variation accounted for in the model.

Finally, the total variance of observations for a specific person i, conditional on that person, can be expressed as

(3.7) $$ \begin{align} \boldsymbol{V}_i = \operatorname{\mathrm{Var}}(\boldsymbol{y}_i \mid i) = \boldsymbol{\Sigma}_D + \boldsymbol{\Sigma}_\nu + \boldsymbol{\Sigma}_\omega + \boldsymbol{\Sigma}_E, \end{align} $$

for $i = 1, \dots , N$ . Conditioning on i emphasizes that the focus is solely on person i, thereby removing the person variance component in Equation (3.7). This distinction is fundamental for defining reliability later: the total variance $\boldsymbol {V}$ will be used when discussing the between-person reliability, while the person-specific variance $\boldsymbol {V}_i$ will be used when discussing the within-person reliability.

4.1 Reliability in CTT

The concept of reliability in psychometrics originates from CTT (Lord et al., Reference Lord, Novick and Birnbaum1968). In CTT, it is assumed that the observed score $y_{it}$ for individual i at timeFootnote ⁶ t can be decomposed into an individual true score $T_{i} = \operatorname {\mathrm {E}}[y_{it}]$ and a measurement error ${\varepsilon }_{it}$ , that is, $y_{it} = T_{i} + {\varepsilon }_{it}$ . Assuming that the true score and error are uncorrelated and that errors are mutually independent, the variance of the observed score can be partitioned as $\sigma _y^2 = \sigma _T^2 + \sigma _{\varepsilon }^2$ , where $\sigma _T^2$ is the true score variance, and $\sigma _{\varepsilon }^2$ is the measurement error variance. Consequently, reliability is defined as the squared correlation (or coefficient of determination) between the observed score and the true score across the population:

(4.1) $$ \begin{align} \rho_{yT}^2 = \operatorname{\mathrm{Cor}}(y_{it}, T_i)^2 = \frac{\sigma_T^2}{\sigma_T^2 + \sigma_{\varepsilon}^2}, \end{align} $$

which simplifies to the ratio of the true score variance to the total observed score variance.

This definition can be seen as an instance of a more general principle: the variance partitioning approach. Under this approach, the total variance of the observations is separated into a systematic and an unsystematic part, and then reliability is defined as a ratio of the systematic variance over the total variance (Cranford et al., Reference Cranford, Shrout, Iida, Rafaeli, Yip and Bolger2006; Cronbach, Reference Cronbach1972; Ghiselli et al., Reference Ghiselli, Campbell and Zedeck1981; Guttman, Reference Guttman1945; Laenen et al., Reference Laenen, Alonso and Molenberghs2007; Laenen et al., Reference Laenen, Alonso, Molenberghs and Vangeneugden2009a, Reference Laenen, Alonso, Molenberghs and Vangeneugden2009b; Lord et al., Reference Lord, Novick and Birnbaum1968; McDonald, Reference McDonald1999; McGraw & Wong, Reference McGraw and Wong1996; Schuurman & Hamaker, Reference Schuurman and Hamaker2019; Shrout & Lane, Reference Shrout, Lane, Mehl and Conner2012a; ten Hove et al., Reference ten Hove, Jorgensen and van der Ark2022). Systematic variance refers to score changes that are attributed to identifiable and often predictable influences—such as stable individual differences or an orderly pattern change in some trend. In contrast, unsystematic variance stems from random, unpredictable fluctuations, including noise characterized by a complete absence of order. This unsystematic variance constitutes the primary source of measurement error. Thus, the general formula for reliability under this approach is

(4.2) $$ \begin{align} R = \frac{\sigma_{\text{sys}}^2}{\sigma_{\text{sys}}^2 + \sigma_{\text{unsys}}^2} = 1 - \frac{\sigma_{\text{unsys}}^2}{\sigma_{\text{total}}^2}, \end{align} $$

where the subscripts $\text {sys}$ and $\text {unsys}$ indicate the systematic and unsystematic variance components, respectively. Within the CTT framework, the systematic variance is the true score variance, and the unsystematic variance is the error variance. Consequently, $R = \frac {\sigma _T^2}{\sigma _T^2 + \sigma _{\varepsilon }^2} = \rho _{yT}^2$ . Conceptually, this coefficient quantifies the ability of a measurement instrument to differentiate between individuals despite the presence of measurement errors across the population. Assume two persons, A and B, and the observed score of A is higher than that of B. The higher the reliability, the more confidence we have in inferring that person A’s true score is also larger than person B’s true score.

Alternatively, there is a second approach to define reliability, denoted as the correlation approach. Under this approach, one defines reliability as a correlation between two parallel measurements of the same individuals (e.g., repeated administrations of the same test or administrations of strictly parallel forms) (Bartko, Reference Bartko1966; Lin et al., Reference Lin, Tuerlinckx and Vanbelle2025; Lord et al., Reference Lord, Novick and Birnbaum1968; Molenberghs et al., Reference Molenberghs, Laenen and Vangeneugden2007; Shrout & Fleiss, Reference Shrout and Fleiss1979; Vangeneugden et al., Reference Vangeneugden, Laenen, Geys, Renard and Molenberghs2004, Reference Vangeneugden, Laenen, Geys, Renard and Molenberghs2005). Because the true score and its variance are usually unavailable, researchers often define reliability based on parallel measurements (real or hypothetical), say $y_{it}$ and $y_{it'}$ , $t \neq t'$ . Under CTT, parallel measurements of the same individual mean having equal true score $(T_{it} = T_{it'}$ , i.e., tau-equivalent) and error variance $(\operatorname {\mathrm {Var}}({\varepsilon }_{it}) = \operatorname {\mathrm {Var}}({\varepsilon }_{it'})$ ). The reliability coefficient is defined as the correlation between two parallel measurements across the population, that is,

(4.3) $$ \begin{align} \rho_{t, t'} = \operatorname{\mathrm{Cor}}(y_{it}, y_{it'}) = \frac{\sigma_T^2}{\sigma_T^2 + \sigma_{\varepsilon}^2} = \rho_{yT}^2. \end{align} $$

This correlation is also called intraclass correlation (ICC; Bartko, Reference Bartko1966; McGraw & Wong, Reference McGraw and Wong1996; Shrout & Fleiss, Reference Shrout and Fleiss1979).Footnote ⁷ The viewpoint of the correlation approach emphasizes the “agreement”Footnote ⁸ between any pairs of replicates from the same individual across the population. Reliability is called test–retest reliability if the parallel measurements $y_{it}$ and $y_{it'}$ are obtained by replications over time.Footnote ⁹ The correlation approach has a clear and direct interpretation because it tells us how closely two measurements agree.

Comparing Equations (4.2) and (4.3), it is clear that the two approaches lead to the same reliability coefficient under CTT, that is, $\rho _{yT}^2 = \rho _{t,t'} = R$ . The variance partitioning approach is further generalized to study designs beyond CTT through the GT (Brennan & Kane, Reference Brennan and Kane1977a, Reference Brennan and Kane1977b; Cronbach, Reference Cronbach1972),Footnote ¹⁰ while the correlation approach keeps the same name. To underscore that the correlation and variance partitioning approach are two sides of the same coin, Bartko (Reference Bartko1966) stated that a reliability coefficient, when defined as a ratio of variances estimated through a linear model, is only valid when it can be interpreted as a form of correlation coefficient. This interpretation ensures that the reliability reflects the degree of similarity between repeated measurements rather than being a purely statistical artifact. However, we have shown that both approaches do not always lead to the same reliability coefficients in the context of multilevel data structures (Lin et al., Reference Lin, Tuerlinckx and Vanbelle2025).

4.2 Reliability in ILD: Challenges

Extending the concept of single-item reliability to ILD is not straightforward for several reasons. First, in CTT or GT as in multilevel reliability studies, the true score is assumed to be a stable attribute of an individual, constant over time or across different measurement occasions (Cranford et al., Reference Cranford, Shrout, Iida, Rafaeli, Yip and Bolger2006; Nezlek, Reference Nezlek2017; Shrout & Lane, Reference Shrout, Lane, Mehl and Conner2012a). In contrast, in ILD, the true score of individuals may vary over time due to situational factors, temporal dynamics, or other influences. This within-person fluctuation in true scores is a key feature of ILD, and it often reflects meaningful patterns of change that researchers are interested in. Second, this within-person fluctuation also generates new opportunities, because it allows us to distinguish the within-person variance from the between-person variance. Third, intensive longitudinal studies often involve additional sources of variability (e.g., day effects) that cannot be accounted for in CTT. Therefore, the traditional framework or methods for multilevel reliability analysis may not adequately capture the complexity of ILD data, and new approaches are needed to define and estimate reliability in that context.

Reliability for (intensive) longitudinal data can be defined either using the correlation approach or the variance partitioning approach. For example, in the context of the single-item measures, Vangeneugden et al. (Reference Vangeneugden, Laenen, Geys, Renard and Molenberghs2004, Reference Vangeneugden, Laenen, Geys, Renard and Molenberghs2005) formulated reliability via the correlation approach, while Laenen et al. (Reference Laenen, Alonso and Molenberghs2007, Reference Laenen, Alonso, Molenberghs and Vangeneugden2009a, Reference Laenen, Alonso, Molenberghs and Vangeneugden2009b) utilized the variance partitioning approach under the LMM framework. However, these earlier works did not explicitly distinguish reliability across different hierarchical levels, which is crucial for modern ILD studies. Notably, other studies have successfully employed the variance partitioning approach to define reliability in intensive longitudinal designs (see Leertouwer et al., Reference Leertouwer, Keijsers, van Roekel and Schuurman2025, for a comprehensive guide).Footnote ¹¹ For instance, Schuurman and Hamaker (Reference Schuurman and Hamaker2019) and Schuurman et al. (Reference Schuurman, Houtveen and Hamaker2015) defined single-item reliability at both the between- and within-person levels within the multilevel measurement error autoregressive (ML-MEAR) model. However, their framework primarily considered a single AR(1) process across time within a two-level structure (e.g., repeated measurements nested within persons). While their model could theoretically be extended to accommodate a three-level structure (i.e., moments nested within days nested within persons), their reliability definitions yield static point estimates that do not explicitly account for how reliability fluctuates or decays over varying temporal lags. Furthermore, Dejonckheere et al. (Reference Dejonckheere, Demeyer, Geusens, Piot, Tuerlinckx, Verdonck and Mestdagh2022) proposed a model-free approach utilizing specific immediate- and delayed-retest designs to empirically isolate measurement error variance, which is then plugged back into the classical reliability formula. While highly innovative, Dejonckheere et al.’s (Reference Dejonckheere, Demeyer, Geusens, Piot, Tuerlinckx, Verdonck and Mestdagh2022) approach requires a specialized data collection design that is absent in standard, pre-existing ILD datasets like ours.

To address these challenges and bridge the methodological gap, the present study focuses specifically on formalizing single-item test–retest reliability within a dynamic LMM framework (recall Equation (3.5)). Although the two approaches (correlation vs. variance partitioning) lead to the same reliability coefficients within CTT (and other simple situations), this is not necessarily the case for more complex data structures (Lin et al., Reference Lin, Tuerlinckx and Vanbelle2025). Therefore, safely interpreting reliability in ILD applications requires understanding how these two approaches might diverge mathematically under complex true-score structures. In the following, we will formally derive reliability coefficients based on the correlation approach first and on the variance partitioning approach. For ease of distinction, the notation $\rho $ will denote reliability defined through the correlation approach, while the notation R will correspond to the one constructed by the variance partitioning approach.

4.3 Correlation approach to reliability for LMM

The correlation approach has been used before to define reliability for longitudinal data and for multilevel data. Vangeneugden et al. (Reference Vangeneugden, Laenen, Geys, Renard and Molenberghs2004, Reference Vangeneugden, Laenen, Geys, Renard and Molenberghs2005) proposed a general formula for reliability in longitudinal data using the correlation approach in the two-level LMM with one serial process, but they did not discuss how to distinguish the reliability at different hierarchical levels. On the other hand, Lin et al. (Reference Lin, Tuerlinckx and Vanbelle2025) further defined reliability coefficients for multilevel data at the different hierarchical levels, that is, between-person reliability and within-person reliability, but they did not consider any serial process. Thus, in this section, we will define reliability coefficients for a three-level model with serial correlation at two different time scales (day and moment).

4.3.1 Between-person reliability

Following the correlation approach (Lin et al., Reference Lin, Tuerlinckx and Vanbelle2025; Vangeneugden et al., Reference Vangeneugden, Laenen, Geys, Renard and Molenberghs2004, Reference Vangeneugden, Laenen, Geys, Renard and Molenberghs2005), the between-person test–retest reliability $\rho ^{\text {B}}$ is defined as the correlation between two measurements $y_{ijk}$ and $y_{ij'k'}$ of the same person i at two moments k and $k'$ on days j and $j'$ across the population. For conciseness, we define the time differences $\Delta _{jj'} = |j-j'|$ and $\Delta _{kk'} = |k-k'|$ . Based on our three-level LMM model with two AR(1) processes (Equation (3.4)), the general formula for this correlation is (the full derivation is provided in Appendix A)

(4.4) $$ \begin{align} \rho^{\text{B}}(\Delta_{jj'}, \Delta_{kk'}) = \operatorname{\mathrm{Cor}}(y_{ijk}, y_{ij'k'}) = \frac{\sigma_s^2 + \delta_{jj'}\sigma_d^2 + \tau_d^2\phi_d^{\Delta_{jj'}}+ \delta_{jj'}\tau_m^2\phi_m^{\Delta_{kk'}}}{\sigma_s^2 + \sigma_d^2 + \tau_d^2 + \tau_m^2 + \sigma_{\varepsilon}^2}, \end{align} $$

where the superscript $\text {B}$ indicates the between-person level and $\delta _{jj'}$ is the Kronecker delta equal to $1$ only if two moment measures are taken on the same day (i.e., $j=j'$ ) and equal to 0 otherwise. The within-day serial correlation process does not play a role if the two moments considered are from different days $j \neq j'$ . Thus, $\delta _{jj'}$ determines whether $\sigma _d^2$ and $\tau _m^2\phi _m^{\Delta _{kk'}}$ appear in the numerator. Furthermore, $\Delta _{jj'}$ and $\Delta _{kk'}$ play vital roles in regulating the influence of the autoregressive effects $\phi _d$ and $\phi _m$ . Note that the two measurements $y_{ijk}$ and $y_{ij'k'}$ are not necessarily parallel (or tau-equivalent) in the CTT sense, because they may have different true scores due to the presence of within-person dynamic patterns. However, $\rho ^{\text {B}}(\Delta _{jj'}, \Delta _{kk'})$ is still reflecting a crucial aspect of reliability: the agreement between two measurements across the population. Next, we will discuss several specific cases of the general formula.

Between-person moment-to-moment reliability: Suppose that researchers are interested in the between-person reliability across different moments (e.g., morning vs. afternoon) within a day, that is, the agreement of the measurements at two different moments $k \neq k'$ within the same day j across the population. Because $\delta _{jj}=1$ and $\Delta _{jj'}=0$ , the between-person moment-to-moment reliability coefficient of Equation (4.4) simplifies to

(4.5) $$ \begin{align} \rho_{\text{M}}^{\text{B}}(\Delta_{kk'}) = \operatorname{\mathrm{Cor}}(y_{ijk}, y_{ijk'}) = \frac{\sigma_s^2 + \sigma_d^2 + \tau_d^2 + \tau_m^2\phi_m^{\Delta_{kk'}}}{\sigma_s^2 + \sigma_d^2 + \tau_d^2 + \tau_m^2 + \sigma_{\varepsilon}^2}, \end{align} $$

where the subscript $\text {M}$ indicates moment-to-moment. This reliability is a function of the time lag between the two moments $\Delta _{kk'}$ : The reliability decreases as $\Delta _{kk'}$ increases, because the two moments are more separated in time and thus the correlation between them is lower.

It might initially appear counterintuitive that day-related variances $(\sigma _d^2$ and $\tau _d^2$ ) persist in the numerator when computing moment-to-moment reliability. This occurs because assessing reliability between two moments strictly on the same day $(j=j'$ ) means these two measurements inherently share the exact same daily environmental characteristics and structural trajectory. Therefore, without conditioning on a specific fixed day, this shared daily variability fundamentally contributes as systematic variance, enhancing their overall similarity.

Between-person day-to-day reliability: Similarly, if researchers are interested in the reliability across different days (such that $\Delta _{jj'}> 0$ and $\delta _{jj'}=0$ ) across the population, the between-person day-to-day reliability coefficient is

(4.6) $$ \begin{align} \rho_{\text{D}}^{\text{B}}(\Delta_{jj'}) = \operatorname{\mathrm{Cor}}(y_{ijk}, y_{ij'k'}) = \frac{\sigma_s^2 + \tau_d^2\phi_d^{\Delta_{jj'}}}{\sigma_s^2 + \sigma_d^2 + \tau_d^2 + \tau_m^2 + \sigma_{\varepsilon}^2},\end{align} $$

where the subscript $\text {D}$ indicates day-to-day. In this case, k and $k'$ can be either the same or different, but because the measurements are located on different days ( $j \neq j'$ ), they are disconnected by the moment-level AR trajectory. As a result, the decayed moment-level covariance $\tau _m^2\phi _m^{\Delta _{kk'}}$ drops out of the numerator. Similarly, because they are sampled from different days, they no longer share the same day effect $d_{ij}$ ; hence, the day variance $\sigma _d^2$ also vanishes from the numerator. Nevertheless, the day-level autoregressive component $\tau _d^2\phi _d^{\Delta _{jj'}}$ remains, reflecting that the two measurements are still systematically bound by the overarching day-level AR trajectory. The day-to-day reliability thus functions exclusively via the lag between days $\Delta _{jj'}$ , continuously decreasing as $\Delta _{jj'}$ increases.

Both Equations (4.5) and (4.6) reflect the reliability based on a single observation at different time points. However, in practice, researchers often use the average of multiple observations to represent a more stable score for a specific period. For example, researchers may use the average of moments within a day to represent the daily score for further analysis. When this is the case, they are wondering about the reliability of the composite score $\bar {y}_{ij\cdot } = \frac {1}{M}\sum _{k=1}^M y_{ijk}$ . Thus, there is another form of day-to-day reliability based on the average of moments within a day. In this case, the day-to-day reliability coefficient becomes

(4.7) $$ \begin{align} \rho_{\overline{\text{D}}}^{\text{B}}(\Delta_{jj'}) = \operatorname{\mathrm{Cor}}(\bar{y}_{ij\cdot}, \bar{y}_{ij'\cdot}) = \frac{\sigma_s^2 + \tau_d^2\phi_d^{\Delta_{jj'}}}{\sigma_s^2 + \sigma_d^2 + \tau_d^2 + \frac{\tau_m^2}{M} + \frac{\sigma_{\varepsilon}^2}{M}}, \end{align} $$

where the subscript $\overline {\text {D}}$ indicates the reliability for day-to-day based on the average of moments. Compared to Equation (4.6), the moment-level variance components $\tau _m^2$ and $\sigma _{\varepsilon }^2$ are now divided by M in the denominator because of the averaging operation, which leads to a smaller denominator and thus a larger result. This makes sense because the average of several measurements is at least as reliable as a single measurement.

Between-person stability and instantaneity: As can be seen in Equations (4.5) and (4.6), the temporal autocorrelation (decaying across time lags) between two measurements is an important component of the aforementioned reliability coefficients. Because of the presence of serial correlation and time lag in the reliability coefficients, it fits well in the distinction between long-term stability and short-term instantaneity, depending on the time lag, as defined by Boyle et al. (Reference Boyle, Stankov, Cattell, Saklofske and Zeidner1995), Cattell (Reference Cattell1964), and Revelle and Condon (Reference Revelle and Condon2019). Stability considers distant time points to evaluate the reliability over the whole measurement period or a further longer period, while instantaneity Footnote ¹² considers two close time points to evaluate reliability immediately or in a short period.Footnote ¹³

Conceptually, we defined stability and instantaneity as the two extremes of a time lag spectrum in intensive longitudinal studies. Stability represents a test–retest correlation (Equation (4.4) or (4.6)) when the time lag is very large such that the influence of the autoregressive effect $\phi _d$ (and also $\phi _m$ ) completely disappears, or when, mathematically speaking, the time lag goes to infinity, that is, $\phi _d^{\Delta _{jj'}} \to 0$ as $\Delta _{jj'} \to \infty $ . As a result, the between-person stability coefficient is defined as

(4.8) $$ \begin{align} \rho_{\infty}^{\text{B}} = \lim_{\Delta_{jj'} \to \infty} \rho_{D}^{\text{B}}(\Delta_{jj'}) = \frac{\sigma_s^2}{\sigma_s^2 + \sigma_d^2 + \tau_d^2 + \tau_m^2 + \sigma_{\varepsilon}^2}, \end{align} $$

where the subscript $\infty $ indicates stability. This coefficient can be alternatively defined as the correlation between the current measurement $y_{ijk}$ and a hypothetical very distant future measurement $y_{i\infty \infty }$ from the same person i and then across the whole population, that is, $\rho _{\infty }^{\text {B}} = \operatorname {\mathrm {Cor}}(y_{ijk}, y_{i\infty \infty })$ . The numerator only consists of the between-person variance component $\sigma _s^2$ . All other sources of variance only appear in the denominator. This coefficient measures the degree to which individuals maintain their relative standing on the measured construct over an extended period, regardless of the specific time interval between assessments. Although the time lag to define stability is, in principle, infinitely long, for a substantive interpretation, it makes sense to think about it as “a very long time period in the context of the study.” Or stated differently, the time lag at which the autocorrelation has effectively vanished. To assess the stability of personality items, we may think of a period of several years. In contrast, if we are measuring mood, a very long time period can be as short as weeks.

On the contrary, instantaneity is defined as the test–retest reliability (Equation (4.4) or (4.5)) when the time lag between two measurements is very short such that the influence of the autoregressive effect is maximal, or mathematically speaking, the time lag goes to zero, that is, $\phi _m^{\Delta _{kk'}} \to 1$ as $\Delta _{kk'} \to 0$ . Thus, the between-person instantaneity coefficient is defined as

(4.9) $$ \begin{align} \rho_{0}^{\text{B}} = \lim_{\Delta_{kk'} \to 0} \rho_{M}^{\text{B}}(\Delta_{kk'}) = \frac{\sigma_s^2 + \sigma_d^2 + \tau_d^2 + \tau_m^2}{\sigma_s^2 + \sigma_d^2 + \tau_d^2 + \tau_m^2 + \sigma_{\varepsilon}^2}, \end{align} $$

where the subscript $0$ indicates instantaneity. This coefficient can be alternatively defined as the correlation between the current measurement $y_{ijk}$ and a hypothetical parallel measurement $y^{\prime }_{ijk}$ from the same person i on the same day j at the same moment k, the only difference between the two measurements being the realized measurement error. Thus, $\rho _{0}^{\text {B}} = \operatorname {\mathrm {Cor}}(y_{ijk}, y^{\prime }_{ijk})$ . It shows that all variance components except the measurement error variance $\sigma _{\varepsilon }^2$ appear in the numerator. This coefficient quantifies the extent to which individuals maintain their relative standing on the measured construct over a very short period (as defined within the context of the study).

4.3.2 Within-person reliability

Within-person reliability focuses on the structural similarity of measurements within the same person instead of across the population of persons. We start again with a general formula for the within-person reliability coefficient $\rho ^{\text {W}}$ , defined as the correlation between two measurements $y_{ijk}$ and $y_{ij'k'}$ for a specific arbitrary person i (the full derivation is provided in Appendix A):

(4.10) $$ \begin{align} \rho^{\text{W}}(\Delta_{jj'}, \Delta_{kk'}) = \operatorname{\mathrm{Cor}}(y_{ijk}, y_{ij'k'} \mid i) = \frac{\delta_{jj'}\sigma_d^2 + \tau_d^2\phi_d^{\Delta_{jj'}} + \delta_{jj'}\tau_m^2\phi_m^{\Delta_{kk'}}}{\sigma_d^2 + \tau_d^2 +\tau_m^2 + \sigma_{\varepsilon}^2}, \end{align} $$

where the superscript $\text {W}$ indicates the within-person level. As before in Equation (4.4), the general formula is a function of the time lags $\Delta _{jj'}$ and $\Delta _{kk'}$ . However, unlike the between-person reliability, the person variance $\sigma _s^2$ disappears from both numerator and denominator because we consider a specific person i.

Again, moving across several time lags, we can define the within-person instantaneity, moment-to-moment reliability, and day-to-day reliability for single and average scores as follows:

(4.11) $$ \begin{align} \rho_{0}^{\text{W}} &= \lim_{\Delta_{jj'} \to 0, \Delta_{kk'} \to 0} \operatorname{\mathrm{Cor}}(y_{ijk}, y_{ij'k'} \mid i) = \frac{\sigma_d^2 + \tau_d^2 + \tau_m^2}{\sigma_d^2 + \tau_d^2 + \tau_m^2 + \sigma_{\varepsilon}^2}, \end{align} $$

(4.12) $$ \begin{align} \rho_{\text{M}}^{\text{W}}(\Delta_{kk'}) &= \operatorname{\mathrm{Cor}}(y_{ijk}, y_{ijk'} \mid i) = \frac{\sigma_d^2 + \tau_d^2 + \tau_m^2\phi_m^{\Delta_{kk'}}}{\sigma_d^2 + \tau_d^2 + \tau_m^2 + \sigma_{\varepsilon}^2}, \end{align} $$

(4.13) $$ \begin{align} \rho_{\text{D}}^{\text{W}}(\Delta_{jj'}) &= \operatorname{\mathrm{Cor}}(y_{ijk}, y_{ij'k'} \mid i) = \frac{\tau_d^2\phi_d^{\Delta_{jj'}}}{\sigma_d^2 + \tau_d^2 + \tau_m^2 + \sigma_{\varepsilon}^2}, \end{align} $$

(4.14) $$ \begin{align} \rho_{\overline{\text{D}}}^{\text{W}}(\Delta_{jj'}) &= \operatorname{\mathrm{Cor}}(\bar{y}_{ij\cdot}, \bar{y}_{ij'\cdot} \mid i) = \frac{\tau_d^2\phi_d^{\Delta_{jj'}}}{\sigma_d^2 + \tau_d^2 + \frac{\tau_m^2}{M} + \frac{\sigma_{\varepsilon}^2}{M}}. \end{align} $$

All within-person reliability coefficients are derived from the conditional correlations, which result in formulas from which the person variance $\sigma _s^2$ is removed. Note that the within-person stability is undefined because it only makes sense for individual differences.

4.4 Variance partitioning approach to reliability for LMM

Reliability coefficients in the context of longitudinal data can also be defined based on the variance partitioning approach (Laenen et al., Reference Laenen, Alonso and Molenberghs2007; Laenen et al., Reference Laenen, Alonso, Molenberghs and Vangeneugden2009a, Reference Laenen, Alonso, Molenberghs and Vangeneugden2009b; Schuurman & Hamaker, Reference Schuurman and Hamaker2019; Schuurman et al., Reference Schuurman, Houtveen and Hamaker2015). There are two distinct (between-person) reliability coefficients worth noticing, $R_{\text {T}}$ (Laenen et al., Reference Laenen, Alonso and Molenberghs2007) and $R_{\Lambda }$ (Laenen et al., Reference Laenen, Alonso, Molenberghs and Vangeneugden2009b). Both reliability coefficients are defined in a similar form as Equation (4.2), but in a matrix version:

$$ \begin{align*} R = 1 - \frac{f(\boldsymbol{\Sigma}_{\text{unsys}})}{f(\boldsymbol{\Sigma}_{\text{total}})}, \end{align*} $$

where $\boldsymbol {\Sigma }_{\text {unsys}}$ is the unsystematic part of the total covariance matrix $\boldsymbol {\Sigma }_{\text {total}}$ and $f(\cdot )$ is a function, which can be either the trace for $R_T = 1 - \frac {\operatorname {\mathrm {tr}}(\boldsymbol {\Sigma }_{\text {unsys}})}{\operatorname {\mathrm {tr}}(\boldsymbol {\Sigma }_{\text {total}})}$ (Laenen et al., Reference Laenen, Alonso and Molenberghs2007) or the determinant for $R_\Lambda = 1 - \frac {|\boldsymbol {\Sigma }_{\text {unsys}}|}{|\boldsymbol {\Sigma }_{\text {total}}|}$ (Laenen et al., Reference Laenen, Alonso, Molenberghs and Vangeneugden2009b), to summarize the covariance matrix into a scalar.

For both cases (i.e., $R_T$ and $R_\Lambda $ ), an important consideration is how to partition the variance components into a systematic and an unsystematic part. This classification depends on the research question. For example, Laenen et al. (Reference Laenen, Alonso and Molenberghs2007, Reference Laenen, Alonso, Molenberghs and Vangeneugden2009a, Reference Laenen, Alonso, Molenberghs and Vangeneugden2009b) argue that, based on their model of interest,Footnote ¹⁴ the systematic part should only include the stable individual differences, that is, the person-related covariance matrix, while the unsystematic part should include all other variance components, including the serial covariance matrix and measurement error covariance matrix. However, this partition is not absolute. In some ILD studies, temporal dynamics, such as daily or momentary fluctuations captured by AR(1) processes, represent meaningful, structured patterns and could therefore be considered as systematic variance. For example, if the research scope is restricted to daily trends (e.g., daily blood pressure level), then the day effect as well as the day-level serial effect should be considered as part of the systematic variance, while the moment-level serial effect should be treated as unsystematic variance. Conversely, if the research focus is on the dynamic process itself (e.g., momentary mood regulation or emotional inertia), then the moment serial effect should be considered as part of the systematic variance.

Depending on three different partitions of the variance components, we can construct three subtypes of $R_{\text {T}}$ Footnote ¹⁵ based on our three-level LMM with two AR(1) processes (Equation (3.5)). As before, we distinguish between-person reliability and within-person reliability for each subtype. This leads to five reliability coefficients based on variance partitioning. In the next section, we will explain how these coefficients are related.

4.4.1 Between-person reliability

As shown in Equation (3.6), the total variance matrix of the observation $\boldsymbol {V}$ is decomposed into five variance components: the person-related covariance matrix $\boldsymbol {\Sigma }_S$ , the day-related covariance matrices $(\boldsymbol {\Sigma }_D$ and $\boldsymbol {\Sigma }_\nu $ ), moment-related covariance matrix $\boldsymbol {\Sigma }_\omega $ , and measurement error covariance matrix $\boldsymbol {\Sigma }_E$ .

Reliability for momentary scores: If the momentary score is the measurement of interest, then the question is to which extent this score can be differentiated from other momentary scores (from the same person or from other persons). In this case, all variance components related to person, day, and moment are considered systematic. Only random measurement error $(\boldsymbol {\Sigma }_E$ ) is treated as unsystematic. The reliability is defined as

(4.15) $$ \begin{align} R_{\text{TM}}^{\text{B}} = 1 - \frac{\operatorname{\mathrm{tr}}(\boldsymbol{\Sigma}_E)}{\operatorname{\mathrm{tr}}(\boldsymbol{V})} = \frac{\sigma_s^2 + \sigma_d^2 + \tau_d^2 + \tau_m^2}{\sigma_s^2 + \sigma_d^2 + \tau_d^2 + \tau_m^2 + \sigma_{\varepsilon}^2} , \end{align} $$

where the subscript $\text {TM}$ indicates reliability based on the trace for the momentary score, and the superscript $\text {B}$ indicates the between-person level. $R_{\text {TM}}^{\text {B}}$ reflects the belief that even though momentary states fluctuate, these fluctuations are part of a meaningful dynamic process. Especially in objective physiological measures, such as heart rate or cortisol levels, the moment-to-moment fluctuations can be systematic and informative about an individual’s current state.

The coefficient $R_{\text {TM}}^{\text {B}}$ is equivalent to the between-person instantaneity coefficient defined by the correlation approach: $R_{\text {TM}}^{\text {B}} = \rho _{0}^{\text {B}}$ . This equivalence tells that both reliability coefficients can be interpreted in two different ways: (1) to which extent can the momentary score be used to differentiate between individuals across the population (based on the variance partitioning approach) and (2) the level of agreement between individual momentary scores over an extremely short period of time (based on the correlation approach). This dual interpretation highlights the multifaceted nature of reliability in ILD.

Reliability of daily scores: When the target of measurement is the score at the day level, the moment-to-moment fluctuations within a day are considered unsystematic noise. In this case, the systematic variance includes only the person- and day-level components $(\boldsymbol {\Sigma }_S, \boldsymbol {\Sigma }_D, \boldsymbol {\Sigma }_\nu $ ), while the unsystematic variance includes the moment-level serial process $(\boldsymbol {\Sigma }_\omega $ ) and measurement error $(\boldsymbol {\Sigma }_E$ ). The reliability is defined as

(4.16) $$ \begin{align} R_{\text{TD}}^{\text{B}} = 1 - \frac{\operatorname{\mathrm{tr}}(\boldsymbol{\Sigma}_\omega + \boldsymbol{\Sigma}_E)}{\operatorname{\mathrm{tr}}(\boldsymbol{V})} = \frac{\sigma_s^2 + \sigma_d^2 + \tau_d^2 }{\sigma_s^2 + \sigma_d^2 + \tau_d^2 + \tau_m^2 + \sigma_{\varepsilon}^2}, \end{align} $$

where the subscript $\text {TD}$ indicates reliability for the daily score.

The coefficient $R_{\text {TD}}^{\text {B}}$ is equivalent to the between-person moment-to-moment reliability $\rho _{\text {M}}^{\text {B}}(\Delta _{kk'})$ for a large within-day time lag $\Delta _{kk'}$ (such that the moment-level autoregressive effect $\phi _m$ vanishes). Consequently, this reliability coefficient quantifies: (1) to which extent the daily score can be used to differentiate between individuals across the population (based on the variance partitioning approach) and (2) the level of agreement between the observed scores over a long interval within the same day (based on the correlation approach). In other words, $R_{\text {TD}}^{\text {B}}$ reflects within-day stability.

Reliability of person (trait) scores: Lastly, if the goal is to measure a stable, person-level trait, all sources of within-person variance (daily fluctuations, momentary dynamics, and measurement error) can be considered unsystematic. The systematic variance is confined to the stable between-person component $(\boldsymbol {\Sigma }_S$ ). This perspective is common when researchers believe temporal dynamics are nuisance factors rather than meaningful signals. The reliability is defined as

(4.17) $$ \begin{align} R_{\text{TP}}^{\text{B}} = 1 - \frac{\operatorname{\mathrm{tr}}(\boldsymbol{\Sigma}_D + \boldsymbol{\Sigma}_\omega + \boldsymbol{\Sigma}_\nu + \boldsymbol{\Sigma}_E)}{\operatorname{\mathrm{tr}}(\boldsymbol{V})} = \frac{\sigma_s^2}{\sigma_s^2 + \sigma_d^2 + \tau_d^2 + \tau_m^2 + \sigma_{\varepsilon}^2}, \end{align} $$

where the subscript $\text {TP}$ indicates reliability of the personal trait-like score. This reliability coefficient should be considered when any day-to-day and moment-to-moment fluctuations, together with their serial correlation, are considered as error and do not truly explain the difference in individual traits. This is reasonable, for example, when considering that serial patterns arise from carry-over effects, such as memory effects or practice effects, which can occur in studies involving self-report measurements.

This coefficient is equivalent to the between-person stability coefficient $R_{\text {TP}}^{\text {B}} = \rho _{\infty }^{\text {B}}$ and indicates that this reliability coefficient quantifies: (1) to which extent the trait-like score can differentiate between individuals across the population (based on the variance partitioning approach) and (2) the level of agreement between trait-like scores within individuals over an extended period (based on the correlation approach).

4.4.2 Within-person reliability

The same variance partitioning logic can be applied at the within-person level. In this case, however, not the total variance $\boldsymbol {V}$ will be partitioned, but the individual-level variance $\boldsymbol {V}_i$ (see Equation (3.7)). $\boldsymbol {V}_i$ is composed of four parts: the day-related covariance matrices ( $\boldsymbol {\Sigma }_D$ and $\boldsymbol {\Sigma }_\nu $ ), the moment-related covariance matrix $\boldsymbol {\Sigma }_\omega $ , and the measurement error covariance matrix $\boldsymbol {\Sigma }_E$ . Depending on the measurement of interest (e.g., a momentary score or a daily score), there are two subtypes of $R_{\text {T}}$ at the within-person level:

(4.18) $$ \begin{align} R_{\text{TM}}^{\text{W}} &= 1 - \frac{\operatorname{\mathrm{tr}}(\boldsymbol{\Sigma}_E)}{\operatorname{\mathrm{tr}}(\boldsymbol{V}_i)} = \frac{\sigma_d^2 + \tau_d^2 + \tau_m^2}{\sigma_d^2 + \tau_d^2 + \tau_m^2 + \sigma_{\varepsilon}^2}, \end{align} $$

(4.19) $$ \begin{align} R_{\text{TD}}^{\text{W}} &= 1 - \frac{\operatorname{\mathrm{tr}}(\boldsymbol{\Sigma}_\omega + \boldsymbol{\Sigma}_E)}{\operatorname{\mathrm{tr}}(\boldsymbol{V}_i)} = \frac{\sigma_d^2 + \tau_d^2}{\sigma_d^2 + \tau_d^2 + \tau_m^2 + \sigma_{\varepsilon}^2}, \end{align} $$

where the superscript $\text {W}$ indicates the within-person level. $R_{\text {TM}}^{\text {W}}$ focuses on the momentary score within a person, so the unsystematic part only includes the measurement error covariance matrix $\boldsymbol {\Sigma }_E$ . This coefficient is equivalent to the within-person instantaneity coefficient $R_{\text {TM}}^{\text {W}} = \rho _{0}^{\text {W}}$ . $R_{\text {TD}}^{\text {W}}$ , focusing on the daily score within a person, considers both the moment-related covariance matrix $\boldsymbol {\Sigma }_\omega $ and measurement error covariance matrix $\boldsymbol {\Sigma }_E$ as the unsystematic part. This coefficient is equivalent to the within-person moment-to-moment reliability $\rho _{\text {M}}^{\text {W}}(\Delta _{kk'})$ when the time lag $\Delta _{kk'}$ within a day is large enough such that the moment-level autoregressive effect $\phi _m$ diminishes.

Both coefficients $R_{\text {TM}}^{\text {W}}$ and $R_{\text {TD}}^{\text {W}}$ quantify: (1) to which extent the measurement of interest (at the moment or day level) can differentiate between different time points within the same person from the variance partitioning viewpoint and (2) the level of agreement between target scores (momentary or daily score) within a person over a short period from the correlation viewpoint.

A further step could be to define $R_{\text {TP}}^{\text {W}}$ as the reliability coefficient for the within-person trait-like score. However, as the explanation in words already suggests, this coefficient cannot be determined. The reason is obvious: the person-level variance $\sigma _s^2$ is removed in both numerator and denominator when we condition on a specific person i. Consequently, there is no systematic variance component left in the numerator if we consider all other variance components as the unsystematic part. Similarly, we could not define the within-person stability coefficient using the correlation approach.

To summarize, we collect the aforementioned reliability coefficients for the three-level LMM with two AR(1) processes in a single row (“M1”) in Table 1 for the between-person level and in Table 2 for the within-person level, respectively. From left to right, the time lag of interest widens, ranging from instantaneity to moment-to-moment fluctuations to day-to-day fluctuations to stability. It can be seen that the reliability coefficients defined by the variance partitioning approach are equivalent to those defined by the correlation approach under some conditions. This equivalence highlights that the correlation and variance partitioning approaches represent two distinct perspectives on the same underlying concept of reliability, which can, in some cases, yield identical results.

Table 1

Comparison of between-person reliability coefficients across four models

Table 1 Long description

The table consists of four rows representing models M 1 through M 4 and five columns for time spans: Instantaneity, Moment-to-moment (two sub-columns), Day-to-day, and Stability.

* Row 1 (M 1: Three-level L M M with two A R 1):

- Instantaneity: rho sub 0 super B equals R sub T M super B equals the fraction with numerator sigma sub s squared plus sigma sub d squared plus tau sub d squared plus tau sub m squared all over denominator sigma sub s squared plus sigma sub d squared plus tau sub d squared plus tau sub m squared plus sigma sub varepsilon squared.

- Moment-to-moment (first): rho sub M super B (Delta sub k k prime) equals the fraction with numerator sigma sub s squared plus sigma sub d squared plus tau sub d squared plus tau sub m phi sub m to the Delta sub k k prime power all over the same denominator as Instantaneity.

- Moment-to-moment (second): rho sub M super B (Delta sub k k prime super plus) equals R sub T D super B equals the fraction with numerator sigma sub s squared plus sigma sub d squared plus tau sub d squared all over the same denominator.

- Day-to-day: rho sub D super B (Delta sub j j prime) equals the fraction with numerator sigma sub s squared plus tau sub d phi sub d to the Delta sub j j prime power all over the same denominator.

- Stability: rho sub infinity super B equals R sub T P super B equals the fraction with numerator sigma sub s squared all over the same denominator.

* Row 2 (M 2: Three-level L M M):

- Instantaneity to Moment-to-moment: rho sub 0 super B equals rho sub M super B equals R sub T M super B equals R sub T D super B equals the fraction with numerator sigma sub s squared plus sigma sub d squared all over denominator sigma sub s squared plus sigma sub d squared plus sigma sub varepsilon squared.

- Day-to-day to Stability: rho sub D super B equals rho sub infinity super B equals R sub T P super B equals the fraction with numerator sigma sub s squared all over denominator sigma sub s squared plus sigma sub d squared plus sigma sub varepsilon squared.

* Row 3 (M 3: Two-level L M M with one A R 1):

- Instantaneity: rho sub 0 super B equals R sub T M super B equals the fraction with numerator sigma sub s squared plus tau sub t squared all over denominator sigma sub s squared plus tau sub t squared plus sigma sub varepsilon squared.

- Moment-to-moment to Day-to-day: rho sub T super B (Delta sub t t prime) equals the fraction with numerator sigma sub s squared plus tau sub t squared phi to the Delta sub t t prime power all over the same denominator as M 3 Instantaneity.

- Stability: rho sub infinity super B equals R sub T P super B equals the fraction with numerator sigma sub s squared all over the same denominator.

* Row 4 (M 4: Two-level L M M equivalent to C T T):

- All time spans: rho equals R equals the fraction with numerator sigma sub s squared all over denominator sigma sub s squared plus sigma sub varepsilon squared.

Note: $\rho $ = reliability based on the correlation approach; R = reliability based on the variance partitioning approach; superscripts $\text {B}$ = between-person level and $\text {W}$ = within-person level; subscripts $0$ = instantaneity, $\text {M}$ = moment-to-moment; $\text {D}$ = day-to-day, $\text {T}$ = test–retest, $\infty $ = stability, $\text {TM}$ = momentary scores, $\text {TD}$ = daily score, $\text {TP}$ = personal trait scores; $\Delta _{kk'}$ = time lag between moments, $\Delta _{kk'}^+$ = a large time lag between moments such that the moment-level autoregressive effect $\phi _m$ diminishes, $\Delta _{jj'}$ = time lag between days, and $\Delta _t$ = time lag between moments/measurements.

4.5 Comparing reliability coefficients for different models

To better understand the relationships among various reliability coefficients, we consider four related models with varying degrees of complexity, where simpler models are nested within the more complex ones. The four models for comparison are:

1. M1: a three-level LMM with two AR(1) processes as defined before in Equation (3.4);

2. M2: a three-level LMM but without any serially correlated processes: $y_{ijk} = \beta + s_i + d_{ij} + {\varepsilon }_{ijk}$ ;

3. M3: a two-level LMM with one AR(1) process at the moment level: $y_{it} = \beta + s_i + \omega _{it} + {\varepsilon }_{it}$ ;

4. M4: a two-level LMM without any serial process: $y_{it} = \beta + s_i + {\varepsilon }_{it}$ .

Models M1 and M2 assume a three-level data structure (moments nested within days, and days nested within persons), while models M3 and M4 assume a two-level structure by ignoring one level (moments nested within persons). For models M3 and M4, we have combined the two time indices j and k into a single time index t, where $t = 1, \dots , D \times M$ . Thus, the time lag between any pair of measurements is denoted as $\Delta _t = |t - t'|$ . In addition, model M3 is also the main model (but without person covariates) discussed in Laenen et al. (Reference Laenen, Alonso and Molenberghs2007, Reference Laenen, Alonso, Molenberghs and Vangeneugden2009b) and Vangeneugden et al. (Reference Vangeneugden, Laenen, Geys, Renard and Molenberghs2004, Reference Vangeneugden, Laenen, Geys, Renard and Molenberghs2005). Model M4 is equivalent to the traditional CTT model where the true score $T_i$ is defined as $T_i = \beta + s_i$ .

Tables 1 and 2 summarize the reliability coefficients for the between-person and within-person levels, respectively. There are some key findings that can be derived from these two tables. First, comparing models M1 and M3 to M2 and M4, respectively, the difference is the inclusion of serial correlation. The models with serial correlation have more comprehensive reliability coefficients compared to those which do not account for any serial correlation. These permit the distinction between instantaneity, moment-to-moment, day-to-day, and stability. Second, because of the three-level structure in model M2, we can still distinguish between the moment-to-moment reliability $\rho _{M}$ and day-to-day reliability $\rho _{D}$ . Nevertheless, both reliabilities are constant because there is no serial correlation in the model. Therefore, regardless of which pair of measurements is selected and used in the correlation, the reliability remains the same. For example, let us focus on the correlation between the measurement of person i on day j and moment k, $y_{ijk}$ , and correlate this with the measurement one moment later (i.e., $y_{ij,k+1}$ ) or two moments later (i.e., $y_{ij,k+2}$ ). It turns out that in model M2, $\rho _{M}(\Delta _{kk'} = 1) = \operatorname {\mathrm {Cor}}(y_{ijk}, y_{ij,k+1}) = \frac {\sigma _s^2 + \sigma _d^2}{\sigma _s^2 + \sigma _d^2 + \sigma _{\varepsilon }^2} = \operatorname {\mathrm {Cor}}(y_{ijk}, y_{ij,k+2})=\rho _{M}(\Delta _{kk'} = 2) $ . Third (and as said above), some reliabilities related to stable traits cannot be defined (or always be 0) for the within-person level (Table 2). This is a necessary consequence of conditioning on the person, which removes the between-person variance that defines these coefficients.

In addition, it is theoretically illuminating to demonstrate how the reliability coefficients derived from our simplified models seamlessly connect to existing reliability formulations in the literature. For the three-level model without autocorrelation (M2), the data structure conceptually mirrors the three-level GT framework often used for multiple-item ESM data when items are modeled as nested within time pointsFootnote ¹⁶ (in contrast to our moments nested within days at the lower levels), which are in turn nested within persons (e.g., Nezlek, Reference Nezlek2017; Nezlek & Gable, Reference Nezlek and Gable2001; Shrout & Lane, Reference Shrout, Lane, Mehl and Conner2012a). If we constrain their GT framework to evaluate the reliability of a single item measured at a single time point, the between-person reliability formula (e.g., Equation 10 in Shrout & Lane, Reference Shrout, Lane, Mehl and Conner2012a) is mathematically identical to our between-person reliability without autocorrelation $(\rho _\infty ^{\text {B}}$ from Table 1). Similarly, their within-person reliability formula (e.g., Equation 11 in Shrout & Lane, Reference Shrout, Lane, Mehl and Conner2012a) shares the exact same form as our within-person reliability $(\rho _0^{\text {W}}$ from Table 2).

Table 2

Comparison of within-person reliability coefficients across four models

Table 2 Long description

The table is organized into six columns. The first column lists the models, and the remaining five columns are grouped under the heading Time span, which includes Instantaneity, Moment-to-moment (split into two sub-columns), Day-to-day, and Stability.

* Row 1: M 1: Three-level L M M with two A R (1).

- Instantaneity: rho sub 0 super W = R sub T M super W = (sigma sub d squared + tau sub d squared + tau sub m squared) all over (sigma sub d squared + tau sub d squared + tau sub m squared + sigma sub varepsilon squared).

- Moment-to-moment (first sub-column): rho sub M super W (Delta sub k k prime) = (sigma sub d squared + tau sub d squared + tau sub m squared phi sub m super Delta sub k k prime) all over (sigma sub d squared + tau sub d squared + tau sub m squared + sigma sub varepsilon squared).

- Moment-to-moment (second sub-column): rho sub M super W (Delta sub k k prime super +) = R sub T D super W = (sigma sub d squared + tau sub d squared) all over (sigma sub d squared + tau sub d squared + tau sub m squared + sigma sub varepsilon squared).

- Day-to-day: rho sub D super W (Delta sub j j prime) = (tau sub d squared phi sub d super Delta sub j j prime) all over (sigma sub d squared + tau sub d squared + tau sub m squared + sigma sub varepsilon squared).

- Stability: No data.

* Row 2: M 2: Three-level L M M.

- Instantaneity through Moment-to-moment: rho sub 0 super W = rho sub M super W = R sub T M super W = R sub T D super W = sigma sub d squared all over (sigma sub d squared + sigma sub varepsilon squared).

- Day-to-day and Stability: No data.

* Row 3: M 3: Two-level L M M with one A R (1).

- Instantaneity: rho sub 0 super W = R sub T M super W = tau sub t squared all over (tau sub t squared + sigma sub varepsilon squared).

- Moment-to-moment through Day-to-day: rho sub T super W (Delta sub t t prime) = (tau sub t squared phi super Delta sub t t prime) all over (tau sub t squared + sigma sub varepsilon squared).

- Stability: No data.

* Row 4: M 4: Two-level L M M (equals C T T). No data provided for any time span.

Note: Same notations in Table 1.

Likewise, the two-level model with a single AR(1) process (M3) corresponds closely to the ML-MEAR model proposed by Schuurman and Hamaker (Reference Schuurman and Hamaker2019), albeit they formulate it using a state-space representation.Footnote ¹⁷ Under the assumption of constant model parameters, the between-person reliability in the ML-MEAR framework (Equation 7 in Schuurman & Hamaker, Reference Schuurman and Hamaker2019) equals our $\rho _\infty ^{\text {B}}$ derived under M3. Analogously, their within-person reliability (Equation 9 in Schuurman & Hamaker, Reference Schuurman and Hamaker2019) is equivalent to our $\rho _0^{\text {W}}$ .

Finally, the traditional reliability in CTT (model M4) can be seen as the most basic special case for the more complex models (M1, M2, and M3) when we ignore the serial correlation and multilevel data structure. Thus, the traditional reliability (Equation (4.1)) is always a constant and cannot capture the rich and dynamic nature of ILD.

It is worth noting that while it is theoretically possible to compute reliability coefficients from simplified models (e.g., ignoring a level of nesting or assuming zero autocorrelation), we strongly advise against choosing a model solely for the sake of obtaining a “better” or specific reliability value. Reliability estimates are only meaningful when derived from a model that adequately represents the data bonding structure. For instance, applying a two-level model to three-level data (ignoring day-level clustering) or neglecting substantial serial correlation may result in biased variance component estimates, which in turn propagate into biased reliability coefficients. Therefore, the selection of the reliability formula (M1–M4) should follow the design of the study and the underlying data structure.

5.1 Methods

Model parameters and reliability coefficients were estimated using Bayesian inference for two reasons. First, the three-level model with two serially correlated processes is not a standard model that can be easily fit using non-Bayesian software. Second, we want to quantify the uncertainty of the reliability coefficients. Both objectives are easier to reach in a Bayesian inferential framework.

The fitting procedure is as follows: we started from a three-level LMM model without any AR(1) structure. Then, we added AR(1) structures to the moment level and day level sequentially if the MCMC sampler converged well and the parameters for the AR(1) process were significantly different from zero. We used R (version 4.5.0; R Core Team, 2025) with the cmdstanr package (version 0.9.0; Gabry et al., Reference Gabry, Češnovar, Johnson and Bronder2025) as an interface of Stan (version 2.36.0; Stan Development Team, 2025a, 2025b) to fit the model. Missing data were handled by setting them as unknown parameters to be estimated. We ran four MCMC chains with 64,000 iterations each, a thinning factor of 8, and considering the first 32,000 iterations as warm-up. The total number of post-warm-up draws across four chains was therefore 16,000.Footnote ¹⁸ The various reliability coefficients were derived afterward based on the posterior samples of the model parameters. The Stan code and R code for data processing, model fitting, and result visualization are available on the Open Science Framework (OSF) at https://osf.io/hnkuc. The original data for the Exam Results Study can be accessed from the EMOTE database or from the authors’ OSF repository.Footnote ¹⁹

6.1 Limitation and future work

Some limitations of the present study should be noted, along with corresponding avenues for future research. First, any reliability coefficient is contingent on the statistical model it is derived from. This holds for coefficients defined in the context of CTT or GT, and our approach is no exception. In this article, we use the equivalent of an empty model as is common practice in the multilevel literature (see, e.g., Hox, Reference Hox2010; Snijders & Bosker, Reference Snijders and Bosker2012). In multilevel modeling, one often starts with fitting an empty model to study the correlation between responses and the partitioning of the variance across the different levels (Hox, Reference Hox2010). The major difference is that we have added the two serial correlation processes to account for possible time dependencies. Obviously, the empty model could be further extended with adjustments to the serial correlation process (e.g., more complicated than an AR(1) process) or heterogeneity of variances (e.g., making them dependent on the day). Such extensions would, however, require careful consideration of model identifiability. In the second step, covariates can be included in the model to explain the variance.

Second, the present study focuses on theoretically defining the reliability of single-item measures in ILD under a three-level LMM with autocorrelation processes. However, other approaches for estimating single-item reliability in ILD have been proposed in literature. For instance, as highlighted in the previous paragraph, the ML-MEAR framework (Schuurman & Hamaker, Reference Schuurman and Hamaker2019; Schuurman et al., Reference Schuurman, Houtveen and Hamaker2015) utilizes a state-space representation that allows for person-specific AR(1) parameters and individual measurement error variances, enabling the computation of personalized within-person reliability coefficients. In addition, this framework can be extended to accommodate multiple items (i.e., ML-MEVAR, where “V” stands for vector), which partitions variance while accounting for cross-lagged effects between items. Nevertheless, unlike our proposed framework, the ML-ME(V)AR does not explicitly distinguish between reliability at different hierarchical time levels (e.g., momentary vs. daily reliability), nor does it evaluate how between- and within-person reliability dynamically changes as a function of the time lag—both of which are central conceptual contributions of our current article.

Alternatively, Dejonckheere et al. (Reference Dejonckheere, Demeyer, Geusens, Piot, Tuerlinckx, Verdonck and Mestdagh2022) proposed a model-free (design-based) approach that leverages specialized data collection schemes (e.g., immediate and delayed retests) to empirically isolate measurement error variance. This model-free approach avoids the need to specify or correctly guess an underlying autocorrelation structure. However, because estimating instantaneous reliability through this method requires a programmed immediate retest design, which is extremely rare in existing or retrospective ILD datasets, we did not incorporate it into our generalized model-based analytical framework. For a comprehensive review and empirical comparison of these methods, we direct readers to recent work by Leertouwer et al. (Reference Leertouwer, Keijsers, van Roekel and Schuurman2025).

Third, this article focuses exclusively on the reliability of single-item measures. A critical limitation of single-item measures is their heightened susceptibility to measurement error, compared to composite scores. Consequently, some intensive longitudinal studies employ two or more items (or multiple raters) to measure a construct, allowing for the quantification of internal consistency or inter-rater reliability (Castro-Alvarez, Tendeiro, Meijer, et al., Reference Castro-Alvarez, Tendeiro, Meijer and Bringmann2022; Cranford et al., Reference Cranford, Shrout, Iida, Rafaeli, Yip and Bolger2006; Geldhof et al., Reference Geldhof, Preacher and Zyphur2014; Hu et al., Reference Hu, Nesselroade, Erbacher, Boker, Burt, Keel, Neale, Sisk and Klump2016; Koch et al., Reference Koch, Jaehne, Riediger, Rauers and Holtmann2025; Lai, Reference Lai2021; Nacke & Mayer, Reference Nacke and Mayer2025; Nezlek, Reference Nezlek2017; Shrout & Lane, Reference Shrout, Lane, Mehl and Conner2012a; Xiao et al., Reference Xiao, Wang and Liu2023) for a comprehensive review, see castroalvarez et al., Reference Castro-Alvarez, Bringmann, Back and Liu2025, leertouwer et al., Reference Leertouwer, Keijsers, van Roekel and Schuurman2025). Therefore, the extension of the current framework to accommodate multiple-item measures represents a valuable and necessary direction for future research. A promising step in this direction is the recent work by Hunter (Reference Hunter2026), who proposed a coefficient $\kappa $ under the multilevel state-space modeling framework to estimate dynamic reliability, which can encompass many of the aforementioned models as special cases.

Finally, the current framework is tailored for continuous outcomes with a normality assumption. However, data in intensive longitudinal studies are often of other types, such as binary, ordinal, or bounded (as with our negative affect data, which were bounded from 0 to 100). A valuable extension of this work would be to adapt the framework for such data types. This would likely involve a generalized linear mixed-effects model (Molenberghs et al., Reference Molenberghs, Laenen and Vangeneugden2007) or another suitable approach (e.g., Vanbelle et al., Reference Vanbelle, Engelhart and Blix2025, for a comparison review on binary scales), and subsequently redefining the reliability coefficients within that new context.