3.1.1 Recent approaches to handle additional dependency

The earliest approach designed to recognize any type of within-study dependence in correlation estimates, including natural and additional dependence, was suggested by Wilson et al. (2016).Reference Wilson, Polanin and Lipsey ²⁴ Their WPL model (following the authors’ last names) built upon use of the multilevel (three-level) model for meta-analysis.Reference Van den Noortgate, López-López, Marín-Martínez and Sánchez-Meca ²⁷ The WPL model for pooling correlations is a no-intercept model using a dummy-coded indicator, $Corr$ , identifying each correlation in the matrix being pooled. For example, for a matrix with p distinct correlations being synthesized, under the assumption of a three-level random-effects model where variability in correlation estimates can be partitioned into sampling error (level-one), within-study (level-two), and between-study (level-three) random effects, the following model would be specified:

(3.1) $$ \begin{align} r_{ij} = {\rho_1}{(Corr_{1ij})} + {\rho_2}{(Corr_{2ij})} +\cdots+ {\rho_p}{(Corr_{pij})} + \nu_j + \eta_{ij}+ \epsilon_{ij}, \end{align} $$

where in this no-intercept model, $r_{ij}$ represents correlation estimate i in study j, $Corr_{kij}$ is the dummy-coded variable with a value of one if $r_{ij}$ is an estimate of correlation $k(k = 1, \ldots, p)$ and zero otherwise, $\rho _{k}$ is the meta-regression model slope coefficient representing the pooled estimate of correlation k, and $\nu _j,\eta _{ij,}$ and $\epsilon _{ij}$ are the level three, two, and one errors, respectively. The level-three and level-two errors are assumed normally distributed with means of zero and variances of $\omega ^2$ and $\tau ^2$ , respectively. The level-one errors are also assumed normally distributed with a mean of zero and known variance for each correlation estimate, $v_{ij}$ . The assumption of the three-level model allows handling of any kind of dependency (both natural and additional) resulting from multiple estimates of correlations in a study.

Wilson et al.Reference Wilson, Polanin and Lipsey ²⁴ formulated the model in Equation (3.1) with a simplified assumption about the random-effects variances. The authors assumed that the level-two and level-three random-effects variances, $\omega ^2$ and $\tau ^2$ , were constant for each correlation in the matrix being pooled. The authors also restricted the level-one covariance matrix in which the sampling covariance between pairs of correlation estimates per study was constrained to zero and instead captured in the single level-two random-effects variance. The original WPL model also constrained to zero the covariance between pairs of level-two and level-three residuals. Regardless of the constraints imposed in the original WPL model, the model handles both natural and additional within-study dependence in correlation estimates for MASEM.

A few recent methodological studies have sought to compare different approaches for handling dependency in correlations for MASEM. Stolwijk et al.Reference Stolwijk, Jak, Eichelsheim and Hoeve ²² summarized existing strategies for pooling correlation matrices in the presence of natural and additional dependency. The study used real data analysis examples and compared ad hoc approaches, including ignoring dependency, aggregating multiple dependent effect sizes, selecting one estimate for each correlation per study, and using the WPL model which, at the time, was the only multivariate method suggested for pooling correlation matrices for MASEM.

Bilici et al.Reference Bilici, Van den Noortgate and Jak ²³ conducted a very recent simulation study that empirically assessed the impact of how additional dependence is handled on estimation of the structural equation model parameters (e.g., path coefficients). The authors compared the same approaches used in Stolwijk et al. The authors found that none of the methods they compared adequately handled additional dependency for MASEM. Even the WPL model was not consistently the best-performing method across all criteria (e.g., bias in standard errors, higher false positive rates, and lower coverage rates). Based on their findings, the authors recommended exploring RVE as a direction for future research focused on handling dependence in correlation estimates for MASEM.

In another very recent study, Scherer and CamposReference Scherer and Campos ²¹ illustrated a possible way to extend the WPL model to a fuller multilevel multivariate formulation to model sampling error covariances between pairs of correlation estimates within a study as well as within-study (level-two) and between-study (level-three) covariances between pairs of correlations’ random effects. The authors also generalized the specification of the random-effects variance estimates, allowing correlation-specific level-two and level-three variance estimates. The intention of Scherer and Campos’ paper was focused on expanding the WPL model specifically for use with hierarchically within-study dependent correlations—multiple correlation matrices within a study (i.e., presence of additional dependence). In addition to detailing the possible specifications for the level-two and level-three random-effects covariance matrices, the authors demonstrated use of the extensions using an applied dataset. The authors used restricted maximum likelihood (REML) estimation to first estimate the multilevel multivariate model for pooling correlation estimates. They then presented an option to obtain cluster-robust standard errors for the pooled correlation coefficient estimates. We view the use of RVE here as especially important because the resulting variances and covariances for the pooled correlation matrix are robust to mis-specification of the known sampling error covariances between pairs of correlations within a study.Reference Tipton ²⁸ Use of Scherer and Campos’ more flexible parameterization of the WPL model combined with use of RVE for results robust to model mis-specification is not only helpful for handling hierarchical within-study dependence but can also be used for any kind of additional (and natural) dependence. Part of our intention with the current study is also to demonstrate use of the more flexible parameterization of the WPL model along with use of RVE for any kind of within-study dependence in correlation estimates by adapting Scherer and Campos’ approach.

3.1.2 Applying robust variance estimation in pooling correlations

RVE is a general statistical technique that adjusts standard errors of parameter estimates in a way that is robust to violations of standard meta-regression model assumptions (e.g., homoscedasticity of errors, shape of distribution of errors, specification of the known covariance between and variances of errors, etc.). In standard meta-analysis, RVE has been widely used to handle various types of within-study dependence in effect sizes.Reference Hedges, Tipton and Johnson ^{8 ,} Reference Pustejovsky and Tipton ⁹ As noted above, use of the RVE technique is especially useful when the effect sizes’ dependency structure is unknown or potentially mis-specified, because it can yield unbiased model estimates and a robust covariance matrix associated with the model’s estimates. Numerous simulation studies have supported use of RVE for handling within-study dependence in effect sizes even when the associated covariance structure (including both sampling error variances and covariances) is mis-specified.Reference Hedges, Tipton and Johnson ^{8 ,} Reference Pustejovsky and Tipton ^{9 ,} Reference Tipton ²⁸

Despite its known advantages for handling within-study dependence in effect size estimates, RVE has rarely been applied in the context of MASEM, and only a few applied MASEM studies have used it. For instance, Atit et al.Reference Atit, Power and Pigott ²⁹ modified the TS-MASEM approach and used multilevel multivariate modeling with RVE in the first stage for pooling elements of the correlation matrix and its covariance matrix. The authors used the resulting matrices to fit their SEM model in stage-two of the TS-MASEM analysis. Given the growing complexity of meta-analytic data and the increasing recognition of a variety of possible within-study dependencies in correlation estimates, further exploration of RVE for use in MASEM is warranted. In the current study, among many other options, we adapted Scherer and Campos’Reference Scherer and Campos ²¹ line of work by demonstrating an approach that incorporates RVE when pooling correlation matrices when studies’ correlation estimates exhibit either or both of natural and additional within-study dependency.

4.3.1 Specifying the meta-regression model for pooling

To obtain the pooled correlation matrix, meta-analysts can adopt the meta-regression WPL model initially introduced by Wilson et al.Reference Wilson, Polanin and Lipsey ²⁴ (see Equation (3.1)). When specifying the model, the meta-analyst must choose an appropriate working model. Of particular importance, the working model should align as well as possible with the correlation coefficients’ dependency structure.Reference Pustejovsky and Tipton ⁹ This step involves specifying the error structure of the model including choosing the known sampling error (level-one) covariance values (reflecting the multivariate nature of correlations within the same matrix) as well as how many additional levels of random effects should be modeled. Along with specifying the number of random-effects levels, the meta-analyst has to choose the covariance structure at each of those levels.Reference Scherer and Campos ²¹ For this demonstration, because our data include both natural and additional dependencies in correlation estimates, we chose the three-level multivariate meta-regression model that includes known within-study sampling error covariances for each study’s correlation estimates and both within- (level-two) and between-study (level-three) random effects. This working model specification offers a flexible model that is well aligned with the data’s structure and with most applied MASEM scenarios.Reference Scherer and Campos ²¹

To recognize the within-study covariance between pairs of correlation estimates, the meta-analyst must first construct the known within-study sampling covariance matrix, $V_r$ . Researchers can use the vcalc() or rcalc() function (for dependent correlation coefficients) in the metafor R package to construct the necessary block diagonal $V_r$ .Reference Olkin and Finn ^{20 ,} Reference Viechtbauer ³⁰ The vcalc() function can handle various effect size types and requires specification of values for the correlation, $\rho $ , between pairs of effect sizes. The $\rho $ is then combined with the user-provided sampling error variances for each effect size estimate to create the $V_r$ . The meta-analyst must choose a feasible value for $\rho $ with the use of the vcalc() function. On the other hand, the rcalc() function is designed specifically for handling natural dependence among pairs of correlations in a correlation matrix calculated for the same sample.Reference Olkin and Finn ²⁰ Thus, the rcalc() function cannot be used to calculate the correlation between two estimates of the same correlation (e.g., two estimates of $r_{SV}$ with additional dependence) that are reported in a study. Because our example includes additional dependence with some studies reporting more than one estimate of any of the three correlations in Table 1, we used vcalc() to create the sampling covariance matrix, $V_r$ .

Construction of the $V_r$ matrix requires knowledge of the correlation between pairs of effect sizes (here, correlations) within a study.Reference Hedges, Tipton and Johnson ⁸ Such values are often unknown, although inaccuracy can negatively impact associated standard errors and statistical inferences when REML is used to estimate the meta-regression model parameters. However, use of RVE has been found to be robust to mis-specification of $V_r$ and provides unbiased parameter and standard error estimates and associated inferences in standard meta-analysis context.Reference Hedges, Tipton and Johnson ^{8 ,} Reference Pustejovsky and Tipton ^{9 ,} Reference Tipton, Pustejovsky and Ahmadi ¹¹ Let us adapt this to the MASEM context to provide an option for researchers, following Scherer and Campos’Reference Scherer and Campos ²¹ example (see Scherer and Campos' Section 4.2.2). The motivation of using RVE here is mainly to address potential mis-specification of $\rho $ values.Reference Hedges, Tipton and Johnson ^{8 ,} Reference Tipton ²⁸

As mentioned earlier, the error structure for the WPL model can be specified in a number of ways depending on the assumptions made about whether to include random effects in the model and at which levels (within- and or between-studies). Given sufficient heterogeneity in correlations (associated with a sufficient number of estimates of correlations within and between studies), we modeled both within- and between-study random effects in addition to using a multivariate model that models the known within-study sampling error covariances between pairs of correlation estimates.

In addition to the number of levels of random effects, the researcher needs to decide how to specify the random-effects covariance matrices at the various levels (here, level-two and level-three). There are various ways to model the random-effects covariance matrix (or matrices if there are multiple levels).Reference Scherer and Campos ²¹ Options include, for example, compound symmetry, heteroscedastic compound symmetry, a diagonal matrix, a single unique variance across correlations, or a fully unstructured matrix.Reference Scherer and Campos ^{21 ,} Reference Viechtbauer ³⁰ Note that most typically, MASEM researchers assume a diagonal covariance matrix allowing each correlation’s random-effects variance to be uniquely estimated while constraining to zero the covariance between random effects. As is common practice for stage one of MASEM, in this demonstration, we assumed a diagonal matrix which corresponds to heteroscedastic compound symmetry with off-diagonal covariance elements fixed to zero for the level-two and level-three random-effects covariance matrices.

In sum, the no-intercept three-level multivariate meta-regression model we assumed for pooling correlation estimates for this first example, Example A, was

(4.1) $$ \begin{align} r_{ij} = \rho_1(Corr_{fs.ij}) + \rho_2 (Corr_{vs.ij}) +\rho_3(Corr_{vf.ij}) + u_{j} + \eta_{ij} + e_{ij}, \end{align} $$

where the dummy-coded indicator $Corr$ variable identifies each of the three correlations being pooled (see Table 1). And $e_{ij}$ is the sampling error for correlation i in study j with $E(e_{ij})=0$ . We constructed the associated block-diagonal sampling error covariance matrix for the $e_{ij}$ s, $V_r$ , assuming a single value for $\rho $ of 0.80 for the correlation between each pair of correlation estimates within each study. $\eta _{ij}$ denotes the within-study (level-two) random effect, and $u_j$ is the between-study (level-three) random effect assumed to have respective means of zero and diagonal covariance matrices $\boldsymbol {\Omega }$ and $\boldsymbol {\Sigma }$ , respectively:

(4.2) $$ \begin{align} \boldsymbol{\Omega} = \begin{bmatrix} \omega_{1}^2 \\ 0 & \omega_2^2 \\ 0 & 0 & \omega_3^2 \end{bmatrix}, \ & \boldsymbol{\Sigma} = \begin{bmatrix} \tau_{1}^2 \\ 0 & \tau_2^2 \\ 0 & 0 & \tau_3^2 \end{bmatrix}. \end{align} $$

There are three unique variances for each of the three correlations being pooled for random effects at the within-study and between-study levels. To estimate the model in Equation (4.1) with the rma.mv() function in the metafor R package, this random-effects covariance matrix specification can be achieved by using the struc = argument either with struct = ‘‘DIAG’’, or equivalently, struct = ‘‘HCS’’ combined with rho = 0, phi = 0.Reference Viechtbauer ³⁰ This model corresponds with a three-level, multivariate model that recognizes heterogeneity and within-study dependence in correlations when pooling them.Reference Scherer and Campos ²¹ The online repository of the current study includes the code used for estimating the meta-regression model with these random-effects covariance matrix specifications.

4.3.2 Correcting standard errors using robust variance estimation

The results from estimating the above meta-regression model provide the pooled correlation estimates contained in Table 2. As mentioned above, MASEM researchers might then choose to use RVE to obtain an estimate of the associated covariance matrix ( $ \mathbf {V_{R.}}$ ) that is robust to mis-specification of $\rho $ between pairs of correlations within each study. Again, use of RVE estimation could be optional, but RVE takes into account any remaining within-study dependence in (multiple estimates of the same) correlations in a study. To calculate the robust $\mathbf {V_{R.}}$ for the $\mathbf {R.}$ , we used the vcovCR() function in the clubSandwich R package.Reference Pustejovsky ³¹ Note that when using RVE, we applied the small-sample adjustment, specifically the bias-reduced linearization adjustment (“CR2” option in the vcovCR() function) as recommended in prior methodological research.Reference Tipton ^{28 ,} Reference Bell and McCaffrey ^{32 –} Reference Pustejovsky and Tipton ^{34 ,} Footnote ^vi The inverse of the resulting $\mathbf {V_{R.}}$ is then used to provide the weight matrix for SEM estimation in stage two of MASEM. Once the $\mathbf {R.}$ and $\mathbf {V_{R.}}$ matrices are obtained, it is necessary to check whether each matrix is positive definite. The positive definiteness of the matrix is required for fitting the SEM. Users can use the is.pd() function in metaSEM R package to check this property.Reference Cheung ⁶

Table 2

Example A. The estimated pooled correlation matrix

Table 2 Long description

The table consists of three columns and three rows representing the variables Spatial, Fluid, and Verbal. The values are as follows:

* Spatial row: The correlation with Spatial is 1.00.

* Fluid row: The correlation with Spatial is 0.49 and the correlation with Fluid is 1.00.

* Verbal row: The correlation with Spatial is 0.30, the correlation with Fluid is 0.38, and the correlation with Verbal is 1.00.

4.4.1 Model fit comparison

One way to test the difference in two (or more) parameters (here, paths) involves comparing the fit of a (full) model in which the two parameters are freely estimated with that of a (restricted) model in which the two parameters are constrained to be equal. The difference in model fit between a pair of nested models as such can be tested using the (log-)likelihood ratio test (LRT).Reference Bollen ³⁹ Each structural equation model’s parameter estimates are those that maximize the model’s likelihood. When each structural equation model is estimated, the likelihood for the model quantifies how well the model fits the observed data—in MASEM, the pooled correlation matrix. The LRT statistic used in comparing the fit of two nested models equals minus two times the difference in the two models’ log-likelihoods (LLs), literally:

(4.3) $$ \begin{align} LRT = -2(LL_{\text{restricted}} - LL_{\text{full}}). \end{align} $$

Given a sufficiently large sample size, the resulting LRT is assumed to follow a $\chi ^2$ distribution with degrees of freedom equal to the difference in the number of parameters estimated in the two nested models being compared (i.e., difference in the two degrees of freedom values). For Example A, this difference is one. If the LRT statistic is statistically significant ( $ p < 0.05$ ), the result indicates that the drop in fit without the additional parameters in the full model significantly degrades model fit. For Example A, a statistically significant LRT would support that a model allowing ${a}$ to differ from ${b}$ fits significantly better than a model with the paths constrained to be equal, Thus, a significant LRT corresponds to an inference that the paths differ significantly from each other and this result implies that the moderator effect is statistically significant. Obviously then, a non-significant LRT statistic indicates a non-significant moderator effect on the parameter of interest.

Figure 2a presents the path coefficient and residual correlation estimates with standard errors in parentheses for the full model in which the paths are freely estimated. Each of the two paths differs significantly from zero ( $ {\hat {a}} =0.30, s_{\hat {a}}=0.05, p<0.05; {\hat {b}} =0.49, s_{\hat {b}}=0.06, p<0.05$ ). Then, we fit the restricted modelFootnote ^vii in which the two paths are constrained to be equal and the results appear in Figure 2b. Note that the unconstrained model in this example is just-identified ( $df = 0$ ). Thus, the $\chi ^2$ test of the restricted model is equivalent to the LRT with the unconstrained model. For full demonstration purposes, we will proceed with the model comparison for this example. The LRT for the two models can be calculated using the anova() function from base R package. The significant LRT result supports the conclusion that the two paths (a and b) differ significantly from each other $(LRT = 26.64, df=1, p < 0.05)$ . In the MASEM moderator analysis context, this result means that the influence of spatial skills on fluid reasoning skills differs significantly from its influence on verbal reasoning skills—evidence of a significant moderator effect.

Figure 2

Example A. Path coefficient and standard error estimates for nested models.

Figure 2 Long description

The image contains two panels labeled a and b.

Panel a, Unconstrained Model.

A box on the left labeled Spatial Skills has two arrows pointing right.

- A blue arrow points to a box labeled Fluid Reasoning Skills with a path coefficient of 0.30 and a standard error of 0.05.

- A red arrow points to a box labeled Verbal Reasoning Skills with a path coefficient of 0.49 and a standard error of 0.06.

- A double-headed black arrow connects Fluid Reasoning Skills and Verbal Reasoning Skills with a correlation of 0.23 and a standard error of 0.09.

Panel b, Constrained Model.

A box on the left labeled Spatial Skills has two arrows pointing right.

- A purple arrow points to Fluid Reasoning Skills with a path coefficient of 0.35 and a standard error of 0.05.

- A second purple arrow points to Verbal Reasoning Skills with an identical path coefficient of 0.35 and a standard error of 0.05.

- A double-headed black arrow connects Fluid Reasoning Skills and Verbal Reasoning Skills with a correlation of 0.23 and a standard error of 0.09.

4.4.2 Path comparison

An alternative approach to comparing models’ fit is to estimate the full model and use the results to directly test the difference between the two paths (i.e., ${a}-{b}$ ). The difference in these parameter estimates can be directly tested using the path estimates and elements of the covariance matrix for the model’s parameter estimates. However, if the sampling distribution of a function of parameters in the model is unknown or difficult to derive, such as the product of parameters in a model, then there are other ways to construct confidence intervals that can be used for statistical inference, such as bootstrapping and likelihood-based confidence interval (LBCI) approaches.Reference Cheung ^{40 ,} Reference Neale and Miller ⁴¹

The metaSEM R packageReference Cheung ⁶ utilizes the OpenMx R packageReference Boker, Neale and Maes ⁴² for estimating the structural equation model for stage two. In OpenMx and metaSEM packages, users can define functions of parameters to be tested including, for example, the difference in the parameters or the product of two parameters (as is needed for testing an indirect effect) using the mx.algebras= argument in the wls() function. To compute the parameter capturing the difference between paths a and b in our Example A, we added the argument: mx.algebras = list(diff = mxAlgebra(a-b, name =‘‘diff’’)). In the metaSEM R package, this defined parameter estimate can then be tested using LBCIs by adding the (intervals.type = ‘‘LB’’) argument.Reference Cheung ⁶

The estimate of the difference in paths ( $ \hat {a}-\hat {b}$ ) was $-0.19$ with a 95% LBCI of ( $ -0.27, -0.12$ ). Because the interval does not contain zero, the result indicates that there is a statistically significant difference between the two paths. This matches the result from the model comparison approach supporting that there is a significant moderator effect $(p<0.05)$ .

5.1.1 Stage 0 for Example B: Specifying the model

The re-specified path model that incorporates the moderator variable values (fluid and verbal reasoning skills) is shown in Figure 3b. This model can capture how the two types of general reasoning skills (fluid and verbal) might differentially mediate the relationship between spatial and mathematical skills. This re-specification allows testing the moderator hypothesis when the moderator variable varies (between and) within studies (as it does in this dataset). A researcher could test the following research question: “Do fluid and verbal reasoning skills differ in the extent to which they mediate the influence of spatial skills on mathematical skills?”

Figure 3

Example B.

Figure 3 Long description

The image contains two panels labeled a and b.

Panel a, titled Mediation Model, features three rectangular boxes. On the far left is Spatial Skills. An arrow points diagonally up to a box labeled General Reasoning Skills, which then points diagonally down to a box on the far right labeled Mathematical Skills. A direct horizontal arrow also connects Spatial Skills to Mathematical Skills.

Panel b, titled Moderated Mediation Model, expands on the first. On the far left is Spatial Skills. Three arrows originate from this box. The top arrow, labeled c, points to Fluid Reasoning Skills. The middle arrow points directly to Mathematical Skills on the far right. The bottom arrow, labeled e, points to Verbal Reasoning Skills. From Fluid Reasoning Skills, an arrow labeled d points to Mathematical Skills. From Verbal Reasoning Skills, an arrow labeled f points to Mathematical Skills. A double-headed curved arrow connects Fluid Reasoning Skills and Verbal Reasoning Skills, indicating a relationship between the two mediators.

5.1.2 Stage 1 for Example B: Pooling correlation estimates

With four variables in the model in Figure 3b, the target correlation matrix is a $4 \times 4$ matrix with six unique correlations to be pooled. The meta-analytic dataset includes 458 correlation estimates from 46 primary studies. We followed the same procedure as in Example A with respect to how we pooled the correlation estimates across studies. We built a no-intercept meta-regression model like in Equation (4.1), although with six correlation indicators. We again estimated a three-level multivariate model that included both within- and between-study random effects and specified a diagonal heteroscedastic variance structure at both levels (similar to the specification presented in Equation (4.2), but for $6\times 6$ covariance matrices). Lastly, to handle the additional dependence in the dataset in that some studies reported multiple estimates of the same correlation, we again used a value of 0.80 for $\rho $ for the known correlation between pairs of within-study correlation estimates to create the sampling error covariance matrix, $V_r$ . And to handle the possible mis-specification of the working model and $\rho $ value, we applied RVE to adjust the covariance matrix estimated for the pooled correlation estimates.

The resulting pooled correlation matrix is presented in Table 3, and its robust covariance matrix is presented in the online repository. Further detailed results including the level-two and level-two random-effects covariance matrices are also presented in the online repository. The overall sample size, 17,824.80, for this dataset was calculated as the sum of each study’s average sample size. These stage one results were then used to estimate the relevant structural equation model in the second stage of TS-MASEM.

Table 3

Example B. The estimated pooled correlation matrix

Table 3 Long description

The table consists of four columns and four rows representing cognitive domains. The diagonal values are all 1.00, representing the correlation of each variable with itself. The off-diagonal values are as follows. Row 2, Verbal: correlates with Spatial at 0.33. Row 3, Fluid: correlates with Spatial at 0.51 and Verbal at 0.41. Row 4, Math: correlates with Spatial at 0.40, Verbal at 0.39, and Fluid at 0.43. The upper right triangle of the matrix is empty.

5.1.3 Stage 2 for Example B: Structural equation model estimation

Recall that the meta-analyst wanted to examine whether the moderator (type of reasoning skills: fluid or verbal) differentially mediated the influence of spatial skills on mathematical skills. To address this mediation effect (i.e., indirect effect), we firstly conducted a test comparing two nested models: (1) a full model in which all structural paths are freely estimated and (2) a restricted model in which the pair of indirect effects (product of each pair of paths) are constrained to be equal (i.e., $c\times d = e \times f$ ; see Figure 3b). See Appendix B for the full RAM specification with the lavaan syntax.

The LRT result for the above pair of nested models indicated that the two models’ fit did not differ significantly supporting that the constraint did not significantly reduce model fit ( $LRT = 1.05, df = 1, p = 0.31$ ). This implies that fluid and verbal reasoning skills do not differ significantly in their mediation of spatial skills’ effect on mathematical skills.

This same test could also be accomplished by estimating the full model and directly testing the difference in the pair of product terms by constructing a confidence interval for $c\times d - e \times f$ . In the estimated full model, the path estimates were: $\hat {c}=0.51, \hat {d}=0.22, \hat {e}=0.33, $ and $\hat {f}=0.23$ . The LBCI for the difference in indirect effects of fluid reasoning skills ( $ \hat {c} \,\times\, \hat {d} = 0.11$ ) versus verbal reasoning skills ( $ \hat {e} \times \hat {f} = 0.08$ ) was ( $ -0.03, 0.07$ ), containing a value of zero. The same inference was thus obtained as when using the model comparison approach.

The research question might focus on more specific paths. For example, the question might focus on a comparison of the influence of the exogenous variable (spatial skills) on the mediator (paths c vs. e in Figure 3b) and whether that influence differs by type of general reasoning skill. Or, similarly, the researcher might wish to test whether the mediator’s impact on the distal outcome depends on the type of general reasoning skill (i.e., compare path d with path f). To investigate the comparison of the pairs of paths more specifically, a meta-analyst could test c versus e and/or d versus f. This approach would be more meaningful when the comparison of indirect effects is significant so that further examination of individual pairs is encouraged. Appendix B.1 shows the model specification using lavaan syntax, in particular, and the online repository includes the full analysis code and results to demonstrate exploration of the associated research questions.

5.2.1 Stage 0 for Example C: Specifying the model

A key advantage of SEM is its ability to incorporate latent variables and factor out measurement error through the latent variables’ measurement models. For example, Fong et al.Reference Fong, Lee and Krou ⁴³ investigated the measurement structure of scores on the learning and study strategies inventory (LASSI) using ten sub-scores. The authors compared four hypothesized models for LASSI’s factor structure and found support for the best-fitting measurement model that includes three correlated LASSI factors: affective/effort, anxiety-rousing, and cognitive activities. In ongoing work, Lee et al. (in progress)Reference Lee, Fong and Soleimani ⁴⁴ expanded their earlier work and added academic achievement as the outcome variable predicted by the three LASSI factors. We are using a subset of their model and data for demonstration purposes.

In our smaller model, we are using the LASSI’s cognitive activities factor, measured by three indicators (information processing, self-testing, and study aids) as a predictor of the observed academic achievement outcome. And suppose the meta-analyst is also interested in the extent to which the cognitive activities factor would differentially predict the two types of academic achievement measures as a moderator: here, GPA versus standardized tests. Thus, the hypothesis being tested is the extent to which the influence of cognitive activities on academic achievement is moderated by the type of academic achievement. Because some primary studies recorded academic achievement using both types of measures and provided multiple correlation estimates, the type of academic achievement variable is a within-study moderator. The model for the type of academic achievement moderator effect is depicted in Figure 4a with the two outcome variables, GPA and standardized test score. The moderator effect can be tested by comparing path ${g}$ with path ${h}$ in Figure 4a.

Table 4

Example C. The pooled correlation matrix

Table 4 Long description

The table consists of five rows and five columns of data representing a correlation matrix. The diagonal values are all 1.00, representing the correlation of each variable with itself.

* Row 1: Processing has a 1.00 correlation with itself.

* Row 2: Self-testing correlates with Processing at 0.57 and with itself at 1.00.

* Row 3: Study aids correlates with Processing at 0.48, with Self-testing at 0.57, and with itself at 1.00.

* Row 4: G P A correlates with Processing at 0.14, with Self-testing at 0.16, with Study aids at 0.13, and with itself at 1.00.

* Row 5: Test correlates with Processing at 0.07, with Self-testing at 0.03, with Study aids at minus 0.03, with G P A at 0.39, and with itself at 1.00.

Figure 4

Example C. Factor model.

Figure 4 Long description

A two-panel structural equation model.

Panel a, titled Model for Moderator Effect, features a central circle labeled Cognitive Activities. Three arrows point downward to rectangular boxes labeled Information processing, Self-testing, and Study Aids. Two arrows point to the right. The top arrow is blue and labeled g, pointing to a box labeled G P A. The bottom arrow is red and labeled h, pointing to a box labeled Test. A double-headed curved arrow connects the G P A and Test boxes.

Panel b, titled Path and Standard Error Estimates, replicates the structure of panel a with numerical values. The three downward arrows are labeled with path coefficients and standard errors in parentheses: 0.70 (0.01) for Information processing, 0.81 (0.01) for Self-testing, and 0.70 (0.02) for Study Aids. The blue arrow to G P A is labeled 0.20 (0.02). The red arrow to Test is labeled 0.05 (0.03). The double-headed arrow between G P A and Test is labeled 0.36 (0.04).

5.2.2 Stage 1 for Example C: Pooling the correlation estimates

With five variables in the moderator model, the associated target pooled correlation matrix is a $5\times 5$ matrix containing 10 unique pairs of correlations. The dataset includes 1,203 correlation estimates from 156 primary studies. There are 55 studies that reported more than one estimate of the same correlation, indicating both natural and additional dependencies in the data. The same model and procedure as described for Examples A and B were used to pool correlations and to obtain the robust covariance matrix for the pooled correlation estimates using the multilevel multivariate model and RVE. The resulting pooled correlation matrix is presented in Table 4. The overall averaged sample size for this example was 67,416.05.

5.2.3 Stage 2 for Example C: Structural equation model estimation

To test the extent to which the influence of the cognitive activities factor on academic achievement depends on the type of academic achievement (here, GPA versus standardized test score), we compared two models: (1) the full model that freely estimates the structural paths ${g}$ and ${h}$ and (2) a restricted model constraining the two paths linking cognitive activities to GPA and to test score to be equal. The process for comparing the two paths is the same as in Examples A and B. However, unlike in the previous two example models, the model includes a latent factor measured by three observed indicators. The associated code used to specify a latent variable using the RAM formulation can be found in Appendix C.

The results associated with the LRT comparing the two nested models supported a significant drop in model fit when the two paths were constrained to be equal $(LRT= 29.22,df=1,p < 0.05)$ . In other words, evidence was found that there was a statistically significant moderator effect whereby cognitive activities’ prediction of academic achievement differed significantly as a function of achievement measure type (i.e., GPA or standardized test).

Instead of using model fit comparison, the moderator effect could instead be tested by estimating the full model and using the results to test the different in the two paths directly. In the full model, the path ${g}$ and ${h}$ estimates were 0.20 and 0.05, respectively. The estimates and standard errors for the factor loadings and structural paths are presented in Figure 4b. The 95% LBCI for the difference in paths ( $ \hat {g}-\hat {h}=0.20-0.05=0.15$ ) was ( $ 0.10, 0.21$ ) indicating that the two paths differed significantly from each other. The conclusion from this path comparison is consistent with that from the model comparison approach supporting a statistically significant moderator effect.