Reanalysis of Study 2

Turnbull-Dugarte and López Ortega (Reference Turnbull-Dugarte and Ortega2024) conducted study 2 “to assess the external validity of the primary findings [ $ \dots $ ]” and “[ $ \dots $ ] to expand upon [those] findings by [asking whether] ethnic out-group opposition result in increased national pride in liberal ‘Western’ values” (1370). Given the analytical inconsistencies outlined above, I conduct a series of analyses that probe the consequences of the authors’ ad hoc choices. Study 2 does not support the authors’ stated conclusions.

For the sake of transparency, I first reproduce the original Table A9 from the published paper’s supplementary materials, which presents the key results for study 2, as Table SM1 in this note’s Supplementary Material. This includes four different analyses where the dependent variable is support for LGBT+ education in schools and the treatment is a binary variable for the vignette condition (1 if treatment, 0 if control):

1. 1. Base Model: Estimates the effect of the treatment, conditional on immigration sentiment.

2. 2. Interaction: Estimates the effect of the treatment, allowing the effect to vary linearly by immigration sentiment (higher value means more positive).

3. 3. ProImmigration: Estimates the effect of the treatment, only for the subset of individuals whose immigration sentiment is higher than the mean (in study 2, this is 7 or higher).

4. 4. AntiImmigration: Estimates the effect of the treatment, only for the subset of individuals whose immigration sentiment is lower than the mean (in study 2, this is 6 or lower).

I estimate these four tests with various combinations of weighting and standard error choices. Table 2 provides an overview of how these choices alter the statistical conclusions presented in the article. The table summarizes the key coefficients of interest in the regression, their corresponding standard errors, and statistical significance at conventional levels (0.1, 0.05, and 0.01). Results are shown for the four key tests in separate columns—the baseline test, the interaction test, the pro-immigration sample, and the anti-immigration sample. Note that for three of the four tests, there is only one coefficient of interest (treatment), while for the interaction test, there are two coefficients of interest (treatment and the interaction term, shown in that order).

Table 2.

Summary of Researcher Choices, Point Estimates, and Statistical Conclusions

Table 2. Long description

The table consists of five columns: Researcher choices, Base model, Interaction, Pro-immig. (pro-immigration), and Anti-immig. (anti-immigration). Standard errors are provided in parentheses below each point estimate.

* Weighted plus Classical S E: Base model 0.095 with three asterisks (0.025); Interaction 0.211 with three asterisks | minus 0.019 with two asterisks (0.062 | 0.009); Pro-immig. 0.111 with three asterisks (0.027); Anti-immig. 0.103 with two asterisks (0.044).

* Unweighted plus Classical S E: Base model 0.026 (0.023); Interaction 0.094 | minus 0.01 (0.065 | 0.009); Pro-immig. 0.037 (0.024); Anti-immig. 0.034 (0.043).

* Weighted plus Robust S E: Base model 0.095 with two asterisks (0.042); Interaction 0.211 with one asterisk | minus 0.019 (0.112 | 0.016); Pro-immig. 0.111 with two asterisks (0.051); Anti-immig. 0.103 (0.069).

* Unweighted plus Robust S E: Base model 0.026 (0.023); Interaction 0.094 | minus 0.01 (0.075 | 0.01); Pro-immig. 0.037 (0.024); Anti-immig. 0.034 (0.043).

* Weighted plus Survey-R S E: Base model 0.095 with two asterisks (0.042); Interaction 0.211 with one asterisk | minus 0.019 (0.11 | 0.015); Pro-immig. 0.111 with two asterisks (0.05); Anti-immig. 0.103 (0.068).

Statistical significance levels are indicated by asterisks: one asterisk for p less than 0.1, two asterisks for p less than 0.05, and three asterisks for p less than 0.01.

Note: * 0.1, ** 0.05, and *** 0.01. Standard errors in parentheses.

For each test, there are five possible specifications. The first row (“Weighted + Classical SE”) is a direct replication of the original analysis in the article (corresponding to Table SM1 in the Supplementary Material). To estimate robust standard errors, I use the HC3 option in estimatr::lm_robust(), and to estimate survey-robust standard errors, I use survey::svyglm(). Rows 2–5 show various combinations of weighting and standard error choices, corresponding to Tables SM2–SM5 in the Supplementary Material.

Table 2 shows that the results for study 2 are highly sensitive to researcher choices in terms of both magnitude and statistical significance. It is only when weights are used and classical standard errors estimated that a pattern of statistically significant results consistent with the authors’ theory emerges.

The use of weights materially changes the magnitude of the point estimates—these attenuate toward zero by around $ 1/2\times $ to $ 1/3\times $ when the weights are removed. Likewise, when using more appropriate variance estimators, the standard errors increase by around $ 2\times $ to $ 3\times $ across the board. It is quite striking that the weighted analyses in study 2 produce point estimates so close to the unweighted analyses in study 1. With weights applied to study 2 (but not to study 1), the interaction estimate is 0.019 in both studies, and the effect in the anti-immigration subsample is approximately 0.1 in both studies.

This fragility is not merely a lack of “robustness”—the results directly contradict the authors’ theoretical argument. First, in none of the alternative specifications is the interaction term between treatment and anti-immigration sentiment statistically significant, despite this being the authors’ key theoretical prediction. This is not only a question of statistical inference; Table 2 shows that without weights, the magnitude of the interaction term is halved and substantively close to zero. Second, in all alternative specifications, there is no statistically significant result for specifically the anti-immigration sample, the very sample for which the authors’ theory predicts an effect would most likely emerge. Again, Table 2 shows that when weights are removed, the estimated treatment effect for this subsample is one-third the magnitude of the original estimate, and substantively close to zero.

Similar conclusions can be drawn for the ancillary analysis of Western liberal values. In the Supplementary Material, I reproduce the published Table A11 (which includes robust standard errors in the published paper) from the published paper’s supplementary materials as Table SM6. Without weights, there is again neither a statistically significant first-order effect of treatment, nor a statistically significant interaction effect. Substantively, both point estimates attenuate toward zero and are of approximately the same magnitude as many of the “placebo” estimates included in the same table. In sum, study 2 does not support the authors’ theory.

Heterogeneous Effects by Weights

The sensitivity to the use of weights is likely exacerbated by their unusual bimodal distribution. This has implications for both the differences in the point estimates between the unweighted and weighted analyses (the estimates are prone to change a lot), and for the standard errors of the estimates (the standard errors are prone to increase due to the relatively high variance of the weights). To probe this, I segment the data into three bins—one for those with low weights under 0.01, one for those with high weights greater than or equal to 3, and one for those with weights in-between. The exact choice of the middle bin is arbitrary, but given the bimodal distribution of the weights, it matters little.

I focus on the anti-immigration and pro-immigration subsamples in turn. Table 3 shows that even among those with high anti-immigrant sentiment, the treatment effect is only nonzero and statistically significant for those with high weights. The medium bin captures only a handful of observations so the result there can likely be set aside, but the point estimate for those in the low bin—which captures almost three-fifths of the data in the anti-immigration subsample—is essentially zero. The exact same pattern emerges, with point estimates that are very similar, in Table 4, despite these respondents being those for whom the authors’ theory predicts minimal treatment effects.

Table 3.

Study 2 (Spain): Heterogeneous Effects by Weight Bin for Anti-Immigrant Subsample

Table 3. Long description

The table consists of four columns: an unlabeled first column for variables, followed by Low weights, Mid weights, and High weights.

Row 1, Treatment: Low weights is minus 0.018 with standard error 0.055. Mid weights is minus 0.182 with standard error 0.169. High weights is 0.132 asterisk with standard error 0.074.

Row 2, Intercept: Low weights is 0.687 triple asterisk with standard error 0.041. Mid weights is 0.682 triple asterisk with standard error 0.104. High weights is 0.463 triple asterisk with standard error 0.052.

Row 3, Number of observations: Low weights is 294. Mid weights is 38. High weights is 184.

Row 4, R 2: Low weights is 0.000. Mid weights is 0.034. High weights is 0.018.

Row 5, Weighted: No for all three columns.

Row 6, H C 3 robust S Es: Yes for all three columns.

Row 7, Survey robust S Es: No for all three columns.

Note: Asterisk indicates p less than 0.1, double asterisk indicates p less than 0.05, and triple asterisk indicates p less than 0.01. Standard errors are provided in parentheses.

Note: * 0.1, ** 0.05, and *** 0.01. Standard errors in parentheses.

Table 4.

Study 2 (Spain): Heterogeneous Effects by Weight Bin for Pro-Immigrant Subsample

Table 4. Long description

The table consists of four columns: Variable, Low weights, Mid weights, and High weights.

- Treatment: Low weights show 0.007 with standard error 0.027. Mid weights show minus 0.006 with standard error 0.077. High weights show 0.128 with standard error 0.055, marked as statistically significant at the 0.05 level.

- Intercept: Low weights show 0.904. Mid weights show 0.950. High weights show 0.766. All intercept values are statistically significant at the 0.01 level. Standard errors are 0.019, 0.051, and 0.044 respectively.

- Number of observations: Low weights have 474. Mid weights have 38. High weights have 188.

- R-squared: Low and Mid weights are 0.000. High weights is 0.029.

- Model Specifications: All three columns are not weighted, use H C 3 robust S E s, and do not use survey robust S E s.

Note: Asterisks indicate significance levels where one asterisk is p less than 0.1, two asterisks is p less than 0.05, and three asterisks is p less than 0.01.

Note: * 0.1, ** 0.05, and *** 0.01. Standard errors in parentheses.

A similar analysis using all the data from study 2, presented in Table SM7 in the Supplementary Material, reveals that in none of the weight bins is the estimated interaction term statistically significant. Indeed, it is only in the mid and high weight bins that the interaction term is nonzero. In the low-weight bin, the interaction term is essentially zero. Together, these results again provide evidence contrary to the authors’ hypothesis that the treatment effect is conditional on immigration attitudes. Any conditionality in study 2 is in fact with respect to the weights, and not immigration attitudes.

Who Is Being Up-Weighted?

The authors state in the published paper that they devised the weights to match Spanish population parameters in terms of gender, age, education, and region. Perhaps it is the case that the weights happen to target those with high levels of anti-immigration sentiment, which would explain the heterogeneous effects in a way consistent with the authors’ theory. In Table 5, I assess which characteristics are being up-weighted by examining the characteristics of a range of covariates across the weight bins.

Table 5.

Study 2 (Spain): Covariates by Weight Bin

Table 5. Long description

The table consists of four columns: Variable, Low weight bin, High weight bin, and Medium weight bin.

* Age: Low 26.17, High 39.64, Medium 31.11.

* Gender: Low 0.48, High 0.51, Medium 0.54.

* Edu: Low 3.27, High 3.45, Medium 3.57.

* Child: Low 0.07, High 0.37, Medium 0.2.

* Foreignborn: Low 0.21, High 0.25, Medium 0.21.

* Queer: Low 0.3, High 0.16, Medium 0.3.

* ImmigrationAttitudes: Low 6.9, High 6.16, Medium 6.36.

Essentially, the weight distribution appears to be driven by age. The Prolific sample skews young compared to the Spanish population, and so those older respondents receive high weights while those who are younger (the bulk of the data) are down-weighted. Respondents in the high weight bin are much older, are much more likely to have children, and are far less likely to identify as queer. Gender, foreign born status, and education appear reasonably well balanced across the bins. Importantly, those in the high weight bin are only weakly more likely to be anti-immigration than those in the low weight bin, further evidence that the dependence of the treatment on weights has little to do with anti-immigration sentiment. I present an in-depth analysis of treatment effects by age group in both the UK and Spain study in the Supplementary Tables SM8 and SM9, and Supplementary Figures SM1–SM3.

Figures 3 and 6: The Interaction Plots

In the body of the article, the authors state that they “estimate the CATE via [ $ \dots $ ] a linear estimation of the conditionality of moderator values” using the following specification (equation 1, p. 1367):

$$ \begin{array}{rll}{Y}_i=\alpha +{\delta}_1treat+{\beta}_1 ImmigrationAttitudes+{\beta}_2treat* ImmigrationAttitudes+{\epsilon}_i.& & \end{array} $$

Given the experimental design, the variable treat represents the vignette treatment and ImmigrationAttitudes, an 11-point scale variable, the moderator. Note that there is likely a typographical error in the equation in the published paper (which I have corrected here), as one would typically model an interaction as a new parameter, rather than the product of both the parameters and the variables. However, it is clear in the text and the formalization that a linear regression with an interaction term underpins the analysis.

Figures 3 and 6 in the published paper purport to summarize the results of this analysis, and the authors direct readers to supplementary material Tables A7 and A9, respectively, for “full regression output” in the figure notes. Each figure includes two different panels which convey roughly the same information. The top panel shows how the predicted value of Y changes as a function of immigration attitudes, for each level of the treatment variable—there are thus two curves, one in dashed red (for the treated group) and one in solid blue (for the control group). The reader is asked to interpret the gap between the curves as the conditional average treatment effect for a given level of ImmigrationAttitudes. The bottom panel of the figures presents that difference as a point for each discrete level of the moderating variable, with confidence intervals around each point.

Neither figure is based on the stated empirical specification or the regression tables to which readers are directed. Inspecting the replication code, the figures are instead generated with the jtools::interact_plot() function in R on the basis of an underlying logistic regression, not a linear regression. This discrepancy between the stated regression specification and the actual implementation is never noted in the article or in the figures. The published Tables A7 and A9 to which readers are directed present the results of linear regressions, not the logistic regressions that underpin the figures. While logistic regression is a reasonable (and perhaps even commendable) choice for interacted specifications with binary dependent variables, the discrepancy between the text, the figures, and the supplementary tables remains.

The visible nonlinearity in both panels in the published Figures 3 and 6 is thus a function of the use of logistic regression. Heterogeneous treatment effects are not separately estimated with a fully-saturated regression, including first-order and interaction terms for each level of the moderator. Only four parameters are estimated: the intercept (where the treatment and immigration are both 0), the treatment effect (the mean shift in Y for a change in treatment status), the first-order coefficient for immigration (how Y shifts for a 1 unit change in immigration sentiment), and the interaction term (any additional shift in Y for a 1 unit change in immigration sentiment for those who are treated only). Because these analytical choices are not explained in the article, and because the authors do not indicate that their specification is a logistic regression, the visible nonlinearities in the published Figures 3 and 6 are misleading—readers may erroneously believe that the treatment effect is estimated for each level of the moderator, or that there is some more complex heterogeneity across the span of ImmigrationAttitudes.

Somewhat to this point, the authors report the results of a “linearity test” styled after Hainmueller, Mummolo, and Xu (Reference Hainmueller, Mummolo and Xu2019) in their supplementary materials (the published Figures A6 and A7). Unfortunately, these tests are implemented in a confusing fashion using the jtools::interact_plot() function. The authors appear to mistakenly switch the predictor value (which should be treat) for the moderator value (which should be Immigration Attitudes)—the tests (and the published Figures A6 and A7) thus reveal a roughly linear relationship between immigration sentiment and the outcome variable, but do not directly test for linearity in the effect of treatment over the span of the moderator. Neither the purpose of this test as specified, nor the result of this implementation, are explained in the article or the Supplementary Material. Fortunately, more appropriate tests of linearity do suggest that linearity is not unreasonable in this case (not reported in this note).

Finally, it is worth contemplating the presentation of uncertainty in the published Figures 3 and 6. First, the authors do not include confidence bands for the top panel of the figure—the reason for this omission is unclear. However, as noted earlier, in the lower panel, the authors present only 90% confidence intervals based on classical standard errors. This detail is not mentioned anywhere in the article or the figures, and was only discernible by examining the hard-coded critical values used to generate the confidence intervals in the replication materials. While there is certainly no hard and fast rule about which confidence intervals one should present, and any particular level of $ \alpha $ is ultimately an arbitrary choice, 95% confidence intervals are expected and assumed as default and to present 90% intervals with no indication of that fact is misleading.

I reproduce the published Figures 3 and 6 below, with two corrections, as Figures 2 and 3, respectively. First, I correct the code to use the specification described in the article—linear regression estimated with ordinary (weighted) least squares. To reiterate, there is nothing necessarily wrong with logistic regression, but it is not the analysis proposed by the authors in the article. Second, I correct the confidence intervals to be 95% confidence intervals based on robust standard errors, and include confidence bands based on robust standard errors in the upper panel. I do not reproduce the published Figure 8, but the same issues apply. Additionally, in the Supplementary Material, I reproduce the published Figure 6 for study 2 as Supplementary Figure SM6, which includes the above corrections but removes the weights.

Figure 2.

Study 1 (UK): Corrected Reproduction of Figure 3

Note: This figure shows results from an interaction analysis for the UK sample. The top panel shows the predicted support for dichotomized LGBT+ education in schools for the control group (blue) and the treated group (red), across the span of the moderator, pretreatment attitudes toward immigration (higher values imply higher support for immigration). The bottom panel shows the conditional average treatment effect (CATE) for each level of the moderator. All results come from a linear regression of the outcome on a treatment indicator, the moderator, and the interaction of these two variables. All confidence intervals are 95% confidence intervals.

Figure 2. Long description

The multi-panel graph is titled Conditional average treatment effect Study 1 U K. Both panels share a horizontal X axis labeled Pre-treatment attitudes towards immigration, ranging from 0.0 to 10.0.

Top Panel:

The Y axis is labeled Predicted support for L G B T plus education in schools, ranging from 0.00 to 0.75. Two lines with 95 percent confidence interval ribbons are shown. A dashed red line, labeled Slope for treated group, starts at approximately 0.40 on the Y axis and shows a steady linear increase to about 0.85. A solid blue line, labeled Slope for control group, starts lower at approximately 0.25 and shows a steeper linear increase, eventually crossing the red line at an X value of approximately 7.5.

Bottom Panel:

The Y axis is labeled Conditional A T E, ranging from negative 0.2 to 0.2. A horizontal dashed blue line marks the 0.0 baseline. A solid red line with vertical error bars representing 95 percent confidence intervals shows a downward linear trend. It starts at a positive C A T E of 0.15 at X equals 0.0 and gradually descends, crossing the 0.0 baseline at approximately X equals 7.5, ending at a slightly negative value at X equals 10.0.

Figure 3.

Study 2 (Spain): Corrected Reproduction of Figure 6 (Weights Used)

Note: This figure shows results from an interaction analysis for the Spain sample. The top panel shows the predicted support for dichotomized LGBT+ education in schools for the control group (blue) and the treated group (red), across the span of the moderator, pretreatment attitudes toward immigration (higher values imply higher support for immigration). The bottom panel shows the conditional average treatment effect (CATE) for each level of the moderator. All results come from a weighted linear regression of the outcome on a treatment indicator, the moderator, and the interaction of these two variables, using the provided weights. All confidence intervals are 95% confidence intervals.

Figure 3. Long description

A multi-panel figure with a shared x-axis labeled Pre-treatment attitudes towards immigration, ranging from 0.0 to 10.0.

Top Panel: Predicted support for L G B T + education in schools.

- The y-axis ranges from 0.00 to 1.00.

- A solid blue line represents the Slope for control group, showing a linear increase from approximately 0.18 at x=0 to 0.92 at x=10. It is surrounded by a light blue 95% confidence interval band.

- A dashed red line represents the Slope for treated group, showing a linear increase from approximately 0.39 at x=0 to 0.95 at x=10. It is surrounded by a light red 95% confidence interval band.

- The gap between the lines is widest at the left and narrows as immigration support increases.

Bottom Panel: Conditional A T E.

- The y-axis ranges from -0.1 to 0.4, with a dashed horizontal reference line at 0.0.

- A red line with circular data points shows the treatment effect decreasing linearly as the moderator increases.

- At x=0, the effect is approximately 0.21. At x=10, the effect drops to approximately 0.02.

- Vertical error bars represent 95% confidence intervals. The error bars cross the 0.0 reference line when pre-treatment attitudes towards immigration are approximately 7.5 or higher, indicating the effect is no longer statistically significant at those levels.

Figures 4 and 7: The Dot-Plots

The second set of visualisations, Figures 4 and 7 in the published paper, are jittered (noise-applied) dot-plots with group means and confidence intervals, generated with modified code based on jtools::effect_plot() in R. These figures appear to be styled after Coppock (Reference Coppock, Druckman and Green2021), who advocates for such visualizations because they clearly communicate not only the treatment effects of interest, but also the underlying research design that motivates the statistical analysis, the data used to estimate the treatment effect, and statistical uncertainty in any estimates. These figures are again misleading.

The points in the published Figures 4 and 7 are not the underlying data on which the analyses are based. Each data point’s value on the y-axis is not that data point’s value on the binary outcome variable Y or the continuous variable on which the dichotomization was based. Instead, they are the fitted value $ \widehat{Y_i} $ of the binary outcome for each observation, given the results from the underlying regression of the outcome on the treatment indicator. For each of the two subsamples in each experiment (pro-immigration and anti-immigration), there are, of course, only exactly two unique fitted values: one for the treated units and one for the control units. As such, almost all of the visualized variation in the plots comes exclusively from the jittering of these four unique values in “R.” This is misleading and is not explained in the article, either in the text or the figures.

Further, the published versions of the plots once again present 90% confidence intervals, a fact documented in neither the figure caption nor the article. Finally, as noted earlier, while in the published Figure 4 the confidence bands are based on robust standard errors, the superimposed text-based representation of significance is not. In Figure 7, neither the confidence bands nor the text representation is based on robust standard errors.

I reproduce the published Figures 4 and 7 as Figures 4 and 5, respectively, with a number of corrections. The figures below show the actual (jittered) value of Y for each observation, and the confidence intervals are the 95% confidence intervals based on robust standard errors. The text representations of point estimates and statistical significance are also updated to reflect estimation with robust standard errors. In study 1, the corrected figures do not much alter the substantive conclusions in the published paper, though the confidence intervals widen and the p-values do increase across the board. In study 2, the changes are more notable.

Figure 4.

Study 1 (UK): Corrected Reproduction of Figure 4

Note: Each dot represents the value of the dependent variable for each individual respondent in the UK sample, jittered on both axes for visibility, red for treated and blue for control. Black point estimates and overlayed text come from a linear regression of dichotomized support for LGBT+ education in schools on the treatment indicator. Lines represent 95% confidence intervals. The left panel includes only those respondents who scored higher than the mean on immigration sentiment, and the left those who scored lower than the mean. Lines are 95% confidence intervals. Treatment group outcome statistically distinct at p < 0.1 (*), p < 0.05(**), p < 0.01(***).

Figure 4. Long description

The Y-axis is labeled Support for L G B T plus education in schools (0-1) with increments of 0.2. The X-axis for both panels has two categories: Control and Treatment. Individual responses are shown as jittered dots clustered at the top (value 1.0) and bottom (value 0.0) of the scale. Blue dots represent the control group and red dots represent the treatment group.

* Left Panel: Pro-immigrant voters (N = 595). The control group has a black point estimate near 0.83. The treatment group has a black point estimate near 0.77. A bracket connects the two estimates with the text A T E = -.06*.

* Right Panel: Anti-immigrant voters (N = 556). The control group has a black point estimate near 0.45. The treatment group has a black point estimate near 0.55. A bracket connects the two estimates with the text A T E = .10**.

Vertical black lines through the point estimates represent 95% confidence intervals.

Figure 5.

Study 2 (Spain): Corrected Reproduction of Figure 7 (Weights Used)

Note: Each dot represents the value of the dependent variable for each individual respondent in the Spain sample, jittered on both axes for visibility, red for treated and blue for control. Black point estimates and overlayed text come from a linear regression of dichotomized support for LGBT+ education in schools on the treatment indicator, using the provided weights. Lines represent 95% confidence intervals. The left panel includes only those respondents who scored higher than the mean on immigration sentiment, and the left those who scored lower than the mean. Lines are 95% confidence intervals. Treatment group outcome statistically distinct at p < 0.1 (*), p < 0.05(**), p < 0.01(***).

Figure 5. Long description

The Y-axis represents Support for L G B T + education in schools on a scale from 0.0 to 1.0. The X-axis categorizes respondents into Control (blue dots) and Treatment (red dots) groups. Individual responses are jittered at the top (near 1.0) and bottom (near 0.0) of the scale.

* Left Panel: Pro-immigrant voters (N = 700). The Control group has a black point estimate at approximately 0.78. The Treatment group has a higher point estimate at approximately 0.89. A bracket connects the two, labeled A T E = .11 with two asterisks indicating statistical significance.

* Right Panel: Anti-immigrant voters (N = 516). The Control group has a black point estimate at approximately 0.48. The Treatment group has a point estimate at approximately 0.58. A bracket connects the two, labeled A T E = .10.

In both panels, vertical black lines through the point estimates represent 95 percent confidence intervals. The treatment effect is visually larger and statistically significant only in the pro-immigrant voter group.

The updated figures reveal a broader point of concern, most acutely in the Spain sample but also in the UK sample: there are far fewer respondents who take a zero value on the dependent variable in either the treatment or control conditions. In study 1, across both conditions, the proportion of the pro-immigrant subsample that takes a zero value on the dependent variable is 0.2. In study 2, that proportion is just 0.11.

This raises concerns about potential ceiling effects that might produce artificial heterogeneous effects. If those who are pro-immigration are mostly already pro-LGBT+ education in schools, then there is not much scope for a positive treatment effect—there are few people who can be moved toward favouring LGBT+ education. By contrast, if those who are anti-immigration are more split on LGBT+ education, then there is unsurprisingly more scope for a treatment effect to emerge. This may generate heterogeneity that is observationally equivalent with the authors’ theory, when really it is driven by a ceiling effect. Indeed, this provides a useful example of the value of the approach to visualizing treatment effects proposed by Coppock (Reference Coppock, Druckman and Green2021), but this value is of course conditional on the visualizations being appropriately implemented.