2. What we (mostly) know: where to place moderators

Researchers have given considerable attention to evaluating whether the placement of moderators can bias treatment effects. The options for where to place moderators are: well before the treatment (i.e., on an earlier panel), on the same survey but before the treatment, or on the same survey but after the treatment. While there is a general consensus that measuring moderators well before treatment, such as in an earlier panel, is ideal, this is often not feasible given resource constraints and attrition concerns. When it comes to moderator placement on a single survey, two main issues have risen to the fore: post-treatment bias and priming effects.

Studies have demonstrated that moderating on variables that are measured post-treatment can bias treatment effects (Acharya et al., Reference Acharya, Blackwell and Sen2016; Montgomery et al., Reference Montgomery, Nyhan and Torres2018). This is especially likely for moderators that are potentially influenced by the treatment itself. On the other hand, measuring moderators pre-treatment runs the risk of priming effects. Priming can occur when moderators bring particular concerns to mind for all respondents in a way that goes on to influence how they respond to the treatment (Mendelberg, Reference Mendelberg2008; Klar et al., Reference Klar, Leeper and Robison2020).

Researchers have studied the placement of sensitive moderators, like racial attitudes, in particular. By and large, studies have found that measuring racial attitude and/or identity moderators pre- or post-treatment does not introduce substantial bias. Valentino et al. (Reference Valentino, Neuner and Vandenbroek2018) find no difference when symbolic racism, as a moderator, was placed pre-treatment versus post-treatment versus in an earlier wave. Similarly, Albertson and Jessee (Reference Albertson and Jessee2023) find no difference between pre- or post-treatment measurement of racial resentment. Sheagley and Clifford (Reference Sheagley and Clifford2025) largely uncover no difference in pre- or post-treatment moderator placement, though they do note that measuring racial resentment pre-treatment, in the same survey, versus an earlier wave led to somewhat stronger treatment effects (though not statistically significantly stronger at a 95% confidence level). This finding is actually the opposite of the typical concern, that priming would wash treatment effects out (Sheagley and Clifford, Reference Sheagley and Clifford2025). Finally, across four separate experiments, d’Urso et al. (Reference d’Urso, Bonilla and Bogdanowicz2025) uncover no effects of moderator placement on the measurement of the moderators, the measurement of experimental outcomes, or experimental outcomes based on treatment conditions. However, they do find some inconsistent effects for moderator placement on the dependent variables when placement is interacted with the treatment condition and the moderator itself. Even in these instances, though, d’Urso et al. (Reference d’Urso, Bonilla and Bogdanowicz2025) note that the differences were minimal, and conclusions unlikely to change. Nevertheless, they advise researchers in most cases to follow Montgomery et al. (Reference Montgomery, Nyhan and Torres2018)’s recommendations to place moderators pre-treatment when possible, to limit or avoid any potential bias. Thus, while the bulk of the literature on the placement of sensitive items indicates that measuring moderators pre- or post-treatment has no or minimal effects on the experiment, the most conservative approach is still to measure the items pre-treatment.

3. Our question: which measure of racial attitudes to use pre-treatment?

While existing studies suggest that placing moderators pre-treatment is preferred, less is known about whether the particular type of moderator used to measure racial attitudes matters for avoiding bias. Though many studies have demonstrated that experimental results hold regardless of the particular racial attitude battery used (e.g., Valentino et al., Reference Valentino, Neuner and Vandenbroek2018; Stephens-Dougan, Reference Stephens-Dougan2020), none to our knowledge have directly tested whether the potential for biasing treatment effects is greater for some batteries than others. Further, most of the evidence that tests placement of racial attitude moderators uses one common battery: the racial resentment scale (Albertson and Jessee, Reference Albertson and Jessee2023; d’Urso et al., Reference d’Urso, Bonilla and Bogdanowicz2025; Sheagley and Clifford, Reference Sheagley and Clifford2025) or its close variant, symbolic racism (Valentino et al., Reference Valentino, Neuner and Vandenbroek2018).Footnote ¹ This makes sense, given that racial resentment is the dominant measure of racial attitudes in American politics research, but it leaves uncertainty as to whether some racial attitude measures have more potential for bias (via priming effects) than others.

Given the emerging consensus in favor of pre-treatment measures of racial attitudes as moderators, it is important to evaluate whether any specific measures of racial attitudes are more or less likely to bias treatment effects downstream. The reason why the particular racial attitude battery may matter is because some are subtler than others—and the more subtle the measure, the less likely that priming effects will emerge. Thus, the goal is to choose a racial attitude battery that is least likely to prime racial considerations for the entire sample, which would run the risk of washing out any treatment effects (especially if the treatment is related to race).

But deciding how to measure racial attitudes is tough for both methodological and theoretical reasons. The ideal measure is direct enough to accurately capture the racial attitudes held by respondents, but subtle enough that respondents are not primed to think about race in a way that changes their response to treatment or control conditions. As a result, a plethora of measures for racial attitudes have emerged over the years.

One commonly used direct measure of racial attitudes asks respondents about the degree to which they associate different groups with characteristics that reflect certain negative stereotypes. By rating groups in terms of a particular stereotype, such as the proclivity of group members to be “hard-working” or “lazy,” respondents’ answers to these stereotype endorsement items are designed to directly capture racial animus (Huddy and Feldman, Reference Huddy and Feldman2009).Footnote ² A potential benefit to the stereotype-endorsement measures is that they can be asked about a variety of groups, including non-racial groups. As a result, it may be the case that the stereotype-endorsement measures could be made subtler if racial groups are embedded within other, non-racial social groups, such as age groups or professions. Thus, while this measure of racial attitudes is direct, embedding the racial groups within a larger pool might increase its subtlety.

Another common measure of racial animus is the racial resentment scale. Racial resentment conceives of contemporary racism as a combination of anti-Black bias and proindividualist attitudes—a form of “new racism” that Kinder and Sanders (Reference Kinder and Sanders1996) argue emerged in response to the decline of biological racism (Schuman et al., Reference Schuman, Steeh, Bobo and Krysan1998) in the wake of the Civil Rights Movement. The common four-item battery asks respondents to report their level of agreement with four statements that purport to explain racial inequality in contemporary American society: (1) The Irish, Italians, Jews, and many other minorities overcame prejudice and worked their way up. Blacks should do the same without any special favors. (2) Generations of slavery and discrimination have created conditions that make it difficult for blacks to work their way out of the lower class. (3) It is really a matter of not trying hard enough; if blacks would only try harder, they could be just as well off as whites. (4) Over the past few years, blacks have gotten less than they deserve.

Some have critiqued the racial resentment scale, arguing that it measures conservatism rather than racial attitudes (Feldman and Huddy, Reference Feldman and Huddy2005; Carmines et al., Reference Carmines, Sniderman and Easter2011). However, recent work has demonstrated that racial resentment is a powerful measure of racial attitudes (Stephens-Dougan, Reference Stephens-Dougan2020), even after accounting for ideological commitments (Enders, Reference Enders2021). It is included on most large-scale national surveys, like the American National Election Study (ANES). The Cooperative Election Study (CES) uses a shortened version of the racial resentment scale, including only the first two items listed above. This reduced battery could make the racial resentment scale, which is already intended to be subtler than more direct measures of racial animus, even less likely to prime racial considerations.

When it comes to experimental research on race, being able to moderate by racial attitudes is extremely important, but it is also important that we limit the priming of racial considerations in ways that might bias treatment effects or contaminate the control. On the one hand, as the preceding discussion suggests, racial resentment, as a measure of “new racism,” aims to be subtler than more direct measures, including stereotype endorsement. While stereotype endorsement measures are more direct, stereotype batteries that include items not only about racial groups but also non-racial ones could reduce potential priming effects. Thus, to better understand whether different measurement approaches might bias treatment effects, this study evaluates the stereotype endorsement approach against two versions of the racial resentment scale, all measured pre-treatment.

4. Experimental design

Our core aim is to determine whether different types of racial attitude measures are more or less likely to bias treatment effects when asked pre-treatment. Note that we are measuring racial attitudes early in the survey, then providing additional questions to buffer before the treatment. So, while we are including racial attitudes measures as well as an experiment on the same survey, there is space between the two. We are not testing moderator placement or advocating measuring racial attitudes immediately prior to the experiment.

We fielded a pre-registered survey experiment with a 4 × 2 design. Respondents were randomized into one of four measurement categories: (1) no racial attitudes measured, (2) stereotype endorsement items, (3) four-item ANES racial resentment scale, and (4) two-item CES racial resentment scale. Later in the survey, respondents were again randomized into one of two conditions: (1) experimental control and (2) a race-related experimental treatment. We use this 4 × 2 design to test whether the way that we measure racial attitudes pre-treatment has effects on an experiment that comes later in the survey. For the stereotype-endorsement measure, we asked about how hardworking or lazy the following groups are (randomized): Old people, Young people, White people, Black people, School teachers, and Police officers.

The survey was in the field with Lucid Marketplace from May 11 to 12, 2021.Footnote ³ Respondents first consented to the survey and answered a few basic demographic questions. Then, they were randomized into one of the four measurement conditions. Afterward, respondents were again asked some demographic questions to take their minds away from race—before being randomized into either the experimental treatment or control. Finally, all respondents answered the key dependent variables, a manipulation check, and were debriefed before exiting the survey.

The subsequent survey experiment tested whether mentioning racial inequality changes support for race-based public policies. Respondents in both conditions read news articles about the COVID-19 pandemic. The control article, titled “COVID-19 Reducing Life Expectancy in U.S.: Vaccines Aim to Help,” reported on the pandemic and vaccine roll-out without any mention of race. The treatment article, titled “COVID-19 Reducing Life Expectancy in U.S.—Especially for Blacks: Vaccine Inequities Don’t Help,” reported on racial disparities in COVID-19 deaths and vaccine access. We present the full articles in the appendix.

6. Discussion and conclusions

This paper set out to provide scholars interested in measuring racial attitudes in survey experiments with guidance about how to do so. We provide new evidence about the appropriateness of measuring racial attitudes prior to treatment in the same survey. Specifically, using the racial resentment scale pre-treatment did not bias downstream treatment effects, but using stereotype endorsement measures did, in some cases. As such, our findings suggest that measuring racial attitudes before the treatment is possible, but that it is better to do so with the racial resentment scale than stereotype endorsement measures.

As scholars consider whether and how to measure racial attitudes as moderators in experiments, it is essential to examine what may be driving the bias induced by measuring stereotypes prior to treatment. In our study, measuring stereotype endorsement pre-treatment often biased substantive treatment effects. Specifically, when exposed to the substantive treatment containing information about racial inequality, respondents exposed to the stereotype endorsement measures became more supportive of race-conscious policies than those who saw no racial attitude measures. This indicates that the respondents may have been primed by the stereotype endorsement measures in a way that shaped their response to the treatment as compared to the control.

It is important to think through exactly what might be driving this result. The patterns here suggest that the more direct nature of the stereotype endorsement items makes them prone to introducing bias; whereas the racial resentment batteries, which are designed to be less direct and subtler, do not. Even embedding the stereotype items about racial groups alongside parallel items about non-racial groups does not sufficiently obscure the direct racial content of the stereotype endorsement approach. Mechanistically, the results we observe could appear if one or both of the following scenarios occur: (1) the scores on the post-treatment outcomes of those exposed to the substantive treatment condition decrease (i.e., become less opposed to racial-egalitarian policy) when stereotype endorsement is measured pre-treatment relative to not measuring racial attitudes pre-treatment, and/or (2) the scores on the post-treatment outcomes of those exposed to the substantive control condition increase (i.e., become more opposed to racial-egalitarian policy) when stereotype endorsement is measured pre-treatment relative to not measuring racial attitudes pre-treatment.

Why might these patterns emerge? The first scenario above (in which those who were both exposed to a stereotype endorsement battery pre-treatment and treated with exposure to information about racial inequality become more supportive of racially egalitarian policy) might be driven by social desirability. Answering the stereotype endorsement battery may have reminded respondents that negative judgments about racial groups are normatively unacceptable, which then pushes them to respond in a more egalitarian way to the message highlighting disparities between racial groups in the treatment. The second scenario above (in which the control condition is contaminated by measuring stereotype endorsement pre-treatment) might occur if measuring stereotype endorsement brings racial considerations to mind and reduces support for racial egalitarian policy, generally, for those who are not treated with exposure to information about racial inequality. Race is brought to mind by the stereotype endorsement battery, so support for potentially racialized policies declines.

Further research would be needed to fully test the mechanism by which measuring stereotype endorsement pre-treatment introduces bias in treatment effects. If we are correct, it is essential for experimental researchers to think carefully about the outcomes they are assessing. In contexts where priming racial considerations could shift attitudes on outcomes, which is often the case when studying how the racialization of policy domains shapes egalitarian attitudes, then measuring stereotype endorsement prior to treatment in the same study may be especially risky. Additionally, our study was not designed to assess whether the measure of racial attitudes utilized biases any conditioning effects in which people with different degrees of racial animus respond differently to treatments with racial content.

Our results are based on a specific treatment and measurement of specific types of racial attitudes and outcomes. The finding that some racial attitudes can be measured pre-treatment without biasing treatment effects, while others cannot, suggests that future research could more broadly assess the effects of different approaches to pre-treatment racial attitudes measures. For instance, future work could evaluate whether the presence or degree of bias introduced by different pre-treatment measures of racial attitudes varies according to the substantive focus of the treatment itself or by the outcomes of interest. Here the treatment focused on racial inequalities in vaccine access, and our outcome measures considered support for a series of different egalitarian policies. Though we found similar patterns across a range of outcome measures, studies that varied the substance of the treatment or considered other dependent variables would further extend our understanding of the nature of bias that stereotype endorsement items produce. Finally, future work could also provide evidence about precisely how far in advance of the treatment racial attitudes that do bias treatment effects in our analysis must be measured in order to avoid the bias uncovered here.