1. Introduction
How we measure racial bias matters. Given the problems inherent in merely asking people whether they are prejudiced, researchers have developed a number of techniques in the last half-century to recover unbiased estimates of racial bias. Randomized response, endorsement experiments, list experiments, and conjoint and factorial experiments, among others, are examples of commonly used strategies. However, despite the common usage of these techniques, we have relatively limited empirical validation of which of these techniques works best to recover unbiased estimates of prejudice, in what contexts they perform best, and the extent to which they measure the same types of bias. I investigate these methodological questions in the context of racial motivation for opposition to new housing development, an area where the methodological stakes are particularly high. If our tools systematically understate the role of race, then both scholars and policymakers may be mischaracterizing the nature of housing opposition, treating what may be racially motivated opposition as legitimate concern about traffic, density, or neighborhood character.Footnote 1
In recent years, conjoint and factorial experiments in particular have become predominant. This is not without good reason. These experiments are easy to implement, well powered, tend not to confuse respondents, and allow researchers to assess the relative importance of several factors at once. However, there is limited empirical validation confirming that these experiments truly work to reduce social desirability bias. Horiuchi et al. (Reference Horiuchi, Markovich and Yamamoto2022) attempt to experimentally validate conjoint analysis as effective for reducing social desirability bias. They find encouraging results, but caution that their results are exploratory and exhibit sensitivity to alternative model specifications. Likewise, their test case is not racial bias, but willingness to vote for a candidate embroiled in a scandal. It may well be the case that measuring race, as perhaps the most fraught topic in contemporary American politics, works differently.
Similarly, there is an ongoing debate among methodologists as to which of text-based treatments or image-based treatments is better at achieving the goal of correctly measuring bias. Those in the image camp argue that they are a more realistic method of treatment delivery that more accurately approximates how people get information in the real world. Likewise, they argue that image treatments are less likely to cue a defensive reaction in respondents. Those in the text camp argue that clear information delivery is paramount, and that images may fail to fully deliver this information. In the real world, they argue, what matters is not how someone acquired information, but that they have it and how they act on it.
Recent research presenting text and visual candidate conjoints reveals a methodological quandary but no definitive answers. Abrajano et al. (Reference Abrajano, Elmendorf and Quinn2018) find that text cues elicit much stronger negative biases toward Latino candidates in the United States than do visual cues, but López Ortega and Radojevic (Reference López Ortega and Radojevic2024) find exactly the opposite in a candidate study in Germany. Their results suggest that visual indications of group membership elicit more discriminatory preferences than do text cues, especially against visible social minorities. So far, these differences have only been explored with reference to candidates and using conjoint experiments.
It also remains to be seen whether this is the case for non-conjoint treatments, or with regard to policy issues rather than candidates. Thus, the present study examines the suitability of conjoint analysis’ cousin, the factorial experiment, in reducing social desirability bias with reference to an important topic in contemporary American politics: racial motivation in opposition to housing development.
The multiple experiment design thus serves both to shed light on the substantive role of race in housing opposition and to help scholars understand how different methodological choices impact the conclusions we draw about racial motivations in public opinion surveys.
2. Why housing?
The U.S. is in the midst of a housing crisis. The growth of home prices and rents in the country’s largest metro areas has far outpaced the growth of incomes, pricing out many of the lowest-income residents (Bailey, Reference Bailey2022) or rendering them homeless (Colburn and Aldern, Reference Colburn and Aldern2022; Kushel et al., Reference Kushel, Moore, Birkmeyer, Dhatt, Duke, Knight and Ponder2023). Two of the primary culprits of this rapid escalation in prices are zoning rules and permitting intransigence in city governments (Glaeser and Gyourko, Reference Glaeser and Gyourko2018; Mast, Reference Mast2024).Footnote 2 These zoning codes, and in particular single-family zoning, constrain development, increase rent, and exclude non-White residents (Menendian et al., Reference Menendian, Nawaz and Gambhir2024).
Zoning codes and levels of development are directly influenced by ordinary citizens (Sahn, Reference Sahn2025a; Dynes et al., Reference Dynes, CF and JQ2022; Einstein et al., Reference Einstein, Glick and Palmer2019; Anzia, Reference Anzia2022; Ellickson, Reference Ellickson2022; Gray, Reference Gray2022), increasing housing costs by constraining supply growth (Asquith et al., Reference Asquith, Mast and Reed2019; Liu et al., Reference Liu, McManus and Yannopoulos2022; Xiaodi, Reference Xiaodi2022; Freemark, Reference Freemark2023; Mast, Reference Mast2023; Ryan and Phillips, Reference Ryan and Phillips2023). If new housing will help alleviate the affordability crisis, yet new development consistently faces determined opposition, then it is important for us to understand the factors that motivate this opposition and undermine new housing. Concerns around traffic, parking, density, property values, “neighborhood character,” and environmental hazards abound (Pendall, Reference Pendall1999; Matthews et al., Reference Matthews, Bramley and Hastings2015; Trounstine, Reference Trounstine2018; Einstein et al., Reference Einstein, Glick and Palmer2019; Marble and Nall, Reference Marble and Nall2021; Anzia, Reference Anzia2022), as do concerns that are less about new housing itself than about related beliefs and affect. Monkkonen and Manville (Reference Monkkonen and Manville2019) find that opposition to new development increases by as much as 20% when respondents are informed that the developer of a proposed project would make a large profit, while David et al. (Reference David, Elmendorf and Kalla2024) find that negative affect toward cities is highly predictive of lower support for upzoning.
When it comes to race, however, we have less research. Whittemore and BenDor (Reference Whittemore and BenDor2019) offer an observational study suggesting that increasing diversity is associated with higher levels of concern about new development. However, this work only draws from one central Virginia county and is unable to draw conclusions about whether diversity causes a change in attitudes toward development. Luttig et al. (Reference Luttig, Federico and Lavine2017) similarly find that Trump supporters react more negatively to housing assistance policy proposals when their support is being solicited by a Black man as opposed to a White man. However, this is not about housing per se so much as social assistance.
Put another way, despite widespread belief that race plays a role in opposition to new housing in much popular discourse, there is a dearth of literature which convincingly demonstrates that this is true. The best causally identified work specifically measuring racial opposition to new housing of which I am aware is Trounstine (Reference Trounstine2021). Trounstine employs a conjoint experiment and finds little effect of expected racial composition on the likelihood that respondents will choose one profile over another. However, given the lack of strong evidence of the suitability of conjoints for this purpose, Trounstine’s null findings may be more a result of respondents not wishing to appear racist as opposed to a genuine ambivalence about the race of their neighbors (Clingermayer, Reference Clingermayer2004).
Moreover, even if conjoints provided some degree of plausible deniability for respondents, they face a fundamental measurement challenge that list experiments do not: the canceling-out problem. In conjoint designs, if 20% of respondents discriminate against Black neighbors but 30% signal support for diversity (whether genuine or performative), the AMCE (Average Marginal Component Effect) will show a positive or null race effect—not because bias does not exist, but because at least nominally pro-diversity responses outnumber discriminatory ones. The list experiment, in contrast, detects the prevalence of the prejudice, regardless of what others do.
To this point, those who stonewall the development of new housing are often quite adamant that race has nothing to do with it. While we know that racial animus has historically played an important role in housing discrimination (Rothstein, Reference Rothstein2017; Reny and Newman, Reference Reny and Newman2018; Sahn Reference Sahn2025b), we have little convincing causal evidence of this bias in contemporary America. To credibly study how contemporary racial attitudes might influence opinion toward development, we need a research design that avoids cuing a defensive reaction from respondents.
Whether race matters for housing opposition is thus both substantively important and empirically unresolved. As I argue in the next section, the reason it remains unresolved may have as much to do with how we measure it as with whether the bias exists.
3. Methodological debate and experimental design
This study examines the effectiveness of visually- and text-presented factorial experiments relative to a list experiment at recovering unbiased estimates of discriminatory preferences against non-White neighbors. The rationale for employing a factorial experiment is straightforward: a conjoint is not the most aptly tailored setup for studying opposition toward new housing.Footnote 3 Conjoints are an ideal experiment for candidate selection as, after all, voters are choosing to vote for one candidate or the other. However, this is not the choice that faces people when it comes to housing. People are not choosing whether to support one development over another. They are choosing whether to support development at all. I do not mean to suggest that conjoint cannot yield useful insights about housing preferences; they clearly can. Rather, my choice of a factorial design is motivated by the fact that the real-world decision facing residents is whether to support or oppose a given development, not which of two developments they prefer. A factorial experiment better mirrors this choice structure.
Thus, to better mirror the choice that respondents face in the real world, I conduct two factorial experiments: one that is text-based and one that is image-based. Following the factorial experiment, I also conduct a list experiment, which is a more validated technique (see Rosenfeld et al., Reference B, K and JN2016) for uncovering respondents’ discriminatory preferences.
Further, as mentioned previously, a wrinkle of these designs that has been overlooked in previous research is that these techniques may not measure the same type of prejudice. In fact, I would argue it is useful to think of these three different methods as measuring different dimensions of prejudice. The image-based factorial experiment most directly measures embedded biases, those with which a person goes through their everyday life, and that are more likely to be captured following the subtler visual treatment. Meanwhile, the text-based factorial experiment, with its explicit statements about expected racial makeup, most directly measures clearly explicit, actively cued biases. Lastly, list experiments most directly measure these activated biases, but are simultaneously designed very intentionally to eliminate the mediator of social desirability bias by giving respondents cover, as the survey enumerator cannot say for certain whether any given individual displays racial bias.
Likewise, as mentioned briefly in the last section, it is important to consider mechanical features of factorial/conjoint experiments as opposed to list experiments. Even in a world without the aforementioned concerns around social desirability bias, a factorial or conjoint experiment will, mechanically, only detect discrimination if the discriminatory position leads more people to select that choice than non-discriminatory positions because these experiments allow respondents to take non-discriminatory positions. Even if there is a sizable minority of respondents who discriminate against non-White candidates, for instance, if there are more people who genuinely prefer a non-White candidate, the results of the experiment will fail to reflect the fact that the discriminatory preferences exist. List experiments do not suffer from this mechanical problem. They cannot tell you whether the discriminatory preferences are winning out, but they can more clearly demonstrate whether those preferences exist at all. As such, we need to think about these techniques as recovering different estimands. In a perfect world, the factorial experiment would yield the net marginal effect of race on support for development, while the list experiment would show the prevalence of a specific racial concern. If we are interested in whether racial bias may motivate opposition to new housing, the factorial experiment offers more challenges than the list experiment from the off.
Lastly, a remaining important limitation of this study, and of factorial/conjoint research on race more broadly, is the difficulty of examining heterogeneity by racial attitudes. I am not aware of prior research that has successfully tested whether race cues in these experiments differentially affect respondents based on racial resentment or similar constructs. This gap likely stems from contamination concerns: asking racial attitude batteries pretreatment risks priming respondents, while conditioning on post-treatment attitudes creates inferential problems. Accordingly, my subgroup analyses focus on partisanship rather than direct measures of racial attitudes. Future methodological research should explore designs that can cleanly isolate these heterogeneous effects.
4. Factorial experiment
The factorial design tests the impact of development type, estimated cost, and expected racial makeup of the new development on the likelihood that a respondent will choose to support that development. Each attribute of the development profiles is randomly and independently varied so that all 12 combinations of the attributes are equally likely to be shown to any given respondent. I present this experiment in two ways. First, through a standard text profile (Figure 1). Second, through a visual treatment conveying the same information, but in a more realistic way—two examples of which appear below (Figure 2 and Figure 3).
Sample text treatment.
Sample visual treatment.
Figure 2 Long description
The flyer promotes a home for sale by St. Clair Homes, priced at 1500 dollars per month. It features images of a house with solar panels and a family sitting in a living room. The property details include 3 bathrooms, 2 bedrooms, a living room, a modern kitchen and all new appliances. The text encourages potential buyers with phrases like ′This listing won′t last!′ and ′Experience luxury living at affordable prices for your family.′ A call to action reads ′Inquire today!′.
Sample visual treatment.
Figure 3 Long description
The flyer promotes Apartments for rent at 3000 dollars per month. It highlights features such as 3 bedrooms, 2 bathrooms, 2 living areas, a modern kitchen and all new appliances. The flyer includes images of the apartment building, a family moving in and a furnished living room. The text encourages experiencing luxury living at affordable prices for families and invites inquiries today.
As previously mentioned, there is a growing literature that holds that image treatments are more realistic than their text-based counterparts. Moreover, images have become one of the main tools used to identify implicit racial biases. To provide just one example, Implicit Association Tests (IATs), which are widely used in social psychology and, to a lesser extent, in clinical, cognitive, and developmental psychology research (Nosek et al., Reference Nosek, Greenwald and Banaji2007), rely on images of faces to gauge respondents’ implicit biases (see also Malhotra et al., Reference Malhotra, Y and Mo2013 and Pérez Reference Pérez2010) for examples of IATs in political science and the importance of implicit biases more generally). Similarly, in addition to well-known concerns about social desirability bias, “virtue signaling” may impact White respondents’ answers when racial issues are involved. White study participants may use the chance to take pro-Black stances as an opportunity to establish themselves as non-racist (see Salerno et al., Reference Salerno, Kulak, Smalarz, Eerdmans, Lawrence and Dao2023 for a recent example).
Each respondent was randomly presented with one of the above types of vignettes, either in text form or image form. The information that is randomly varied can be seen in Table 1.
Treatment factors
Table 1 Long description
The data lists price by dwelling type and racial makeup. Three rows are shown under the columns Dwelling Type, Racial Makeup, and Price. Row 1 lists Single Family Home, Black, and $1500 per month. Row 2 lists Apartment, White, and $3000 per month. Row 3 lists a price of $4500 per month, with the dwelling type and racial makeup cells left blank.
In the same survey item as these treatments, respondents were then asked the following: “Based just on the above information about this development, if it were put to a public vote, would you support it being built in your neighborhood?” to which they could respond either “Yes” or “No.” The following pre-registered hypotheses relate to this first outcome measure.Footnote 4
H1: Respondents receiving the Black family treatment will be less likely to say that they would support the project being built in their neighborhood.
H2: This dispreference will be stronger among those respondents receiving the image version of the treatment.
5. List experiment
Following the main treatment and first outcome measure, I also implement a list experiment. This is important for three reasons. First and foremost, it offers a clear comparison to the factorial experiment using a technique that is explicitly designed and well validated for uncovering bias when the experimental assumptions hold true (Blair and Imai, Reference Blair and Imai2012). While both forms of the factorial experiment are still, in theory, susceptible to social desirability bias, a list experiment is less prone to this kind of confounding. Second, the list experiment allows me to uncover whether race itself is a concern about housing development, rather than inferences respondents are making about the effects of diversity. And third, to the best of my knowledge, the present study is the first to use a list experiment to study reasons for opposition to new housing development, despite this situation being a clear use case for such an experiment.
I present respondents with a list of reasons why people are often worried about new housing and only include an explicitly racial motivation in the second of the two lists. This concern is bolded and italicized in the list below.Footnote 5 Respondents are only presented with one of the two lists, chosen at random and not influenced by which version of the factorial experiment they received.Footnote 6
• I am concerned that new housing would make my neighborhood louder because of construction
• I am concerned that new housing would bring more traffic to my neighborhood
• I am concerned that new housing would bring crime to my neighborhood
• I am concerned that new housing would affect the character of my neighborhood
• I am concerned that new housing would impact the property values of my neighborhood
5.1. I am concerned that new housing would cause non-White people to move into my neighborhood
The basic premise of list experiments is simple: if the average number of items selected is higher in the list containing the sensitive item, then the only plausible reason is that at least some respondents are answering affirmatively to the sensitive item.Footnote 7
If race is a primary motivating factor, two things should be true. First, the average number of items selected in the second list should be higher than in the first list. Second, because the list experiment comes after the factorial experiment, the average number of items that respondents agree to in the second list should be higher for those in the Black treatment condition relative to those in the White treatment condition. This intuition is formalized in the following pre-registered hypotheses.
H3: Respondents receiving the second list will select a higher number of items on average than those receiving the first.
H4: Respondents in the Black family treatment condition who receive the sensitive list will select a higher number of items on average than those in the White family treatment condition who receive the sensitive list.
6. Subsidiary mechanism test
Lastly, as a subsidiary mechanism test, I present respondents with a battery of topics—crime, school quality, property values, traffic, neighborhood character, and affordability—and ask whether new housing would worsen, have no impact, or improve each of those dimensions of the neighborhood.
There is a substantial body of literature that suggests that perceptions of crime, school quality, and neighborhood character are racialized (Chiricos et al., Reference Chiricos, Welch and Gertz2004; Maria et al., Reference Maria, Couper, Farley and Forman2009; Lewis et al., Reference Lewis, Emerson and Klineberg2011; Goyette et al., Reference Goyette, Farrie and Freely2012; Pickett et al., Reference Pickett, Chiricos, Golden and Gertz2012; Ghandnoosh, Reference Ghandnoosh2014; Moye and Thomas, Reference Moye and Thomas2023; Joshua et al., Reference Joshua, Hull, Pathak and Walters2024). As a result, if race is a motivating factor, then those receiving the racial treatment ought to be more likely to say that new housing is likely to worsen crime, school quality, and neighborhood character. There is also a chance that respondents will report anticipated negative effects on property values as well. It is a demonstrated phenomenon that the real estate industry, banks, and other actors undervalue property owned by Black individuals and families (see e.g. Perry, Reference Perry2020). It is less clear whether this is directly due to White neighbors believing that Black families reduce property values as a rule. As a result, I do not have clear theoretical expectations as to the sign and magnitude of this coefficient. Respondents’ expectations about the impact of development on affordability and traffic should not be worsened by receiving the racial treatment. These expectations are formalized in the following pre-registered hypotheses.
H5: Respondents in the Black family treatment condition will be more likely to say that new housing will worsen crime.
H6: Respondents in the Black family treatment condition will be more likely to say that new housing will worsen school quality.
H7: Respondents in the Black family treatment condition will be more likely to say that new housing will worsen neighborhood character.
7. Data
The data in this paper come from an original survey fielded to 3,496 White respondentsFootnote 8 using the Qualtrics survey engine and fielded between October 23 and October 30, 2024, on the online platform Cint, a commonly used sample provider for both academic and market research (Alexander and McClellan, Reference Alexander and McClellan2019).Footnote 9
My sample mirrors the White population well. In terms of political party, 39.6% are Democrats or lean toward the Democratic Party, 46.3% are Republicans or lean toward the Republican Party, and 11.4% are independent or belong to a third party and do not lean toward one party or the other. Ideologically, 12.9% identify as Very Conservative, 20.6% as Conservative, 40.5% as Moderate, 15.6% as Liberal, and 10.4% as Very Liberal. They are also well distributed geographically, in 2,912 ZIP codes in all 50 states, and largely tracking population density, as shown in the Supplementary Appendix.
This is also largely a suburban, residentially stable sample, with 55.2% saying they live in an All Single Family Home neighborhood,Footnote 10 with another 24.5% living in neighborhoods with Mostly Single-Family Homes. Those who live in neighborhoods that are mostly or all apartments comprise 13.4% and 6.8%, respectively. On average, they have lived in their current neighborhood for 13 and a half years, though the median is lower: 9 years of residential stability, with 65% owning their home, while 35% are renters. However, this sample is older than the White population—54 in the sample vs. 44 in the population, most likely because the sample is limited to adults.
When it comes to their views on housing development and the housing crisis, they largely acknowledge that the U.S. has a housing affordability problem,Footnote 11 but they are split on its causes and what the solution is. Of those who see an affordability problem, only 47.3% see a shortage of housing as its primary cause. Meanwhile, 23% believe it is primarily the rich and corporations purposefully holding housing off the market, 20.9% blame people trying to live in places they cannot afford rather than places they can, and the remaining 9.7% offer various additional reasons. Likewise, back in the full sample, only a bare majority, 51.4%, say that they support “infill development”Footnote 12 when presented with a very favorable definition of the term.
In short, my respondents largely do see the housing crisis before them, but they are divided on its causes, and on whether they support the most evidence-backed policy for addressing it.
Lastly, consistent with previous research, when asked how important various neighborhood topics are to them, they consistently say that they are all important. All topics are rated at or above four out of five on a scale from “Not at all Important” to “Very Important”, as shown in Table 2.Footnote 13
Summary statistics for importance measures
Table 2 Long description
Mean and St. Dev. are reported for six measures labeled Importance of Crime, Schools, Property, Traffic, Character, and Affordability. Importance of Schools has mean 4.10 and standard deviation 0.95. Importance of Property has mean 4.05 and standard deviation 0.92. Importance of Traffic has mean 3.93 and standard deviation 0.92. Importance of Character has mean 4.01 and standard deviation 0.87. Across measures, means range from 3.93 to 4.49, and standard deviations range from 0.75 to 0.95.
This is important for substantive purposes, of course, but also useful for my investigation of methodological differences. The fact that there is substantial viewpoint diversity among my respondents indicates that there is enough variation for me to capture the effect of racial bias if such a bias exists and the methods I employ work as they are supposed to.
8. Results
Before presenting the full results, Table 3 summarizes the outcomes of all seven pre-registered hypotheses. The central pattern is immediately apparent: the factorial experiments (H1–H2) find no evidence of racial bias (in fact, the point estimates are in the opposite direction) while the list experiment (H3) reveals meaningful racial concerns among a substantial share of respondents. Two of the subsidiary mechanism tests (H6–H7) show effects only in the text condition, where the racial treatment is most salient.
Summary of pre-registered hypotheses and results
Table 3 Long description
Seven hypotheses labeled H1 through H7 are listed with columns for Hypothesis, Supported?, and Key Result. H1 is marked No with opposite direction, with a positive pooled coefficient of 0.041 with one asterisk. H2 is marked No, with no significant difference between text and image conditions in the pooled sample. H3 is marked Yes, with an estimated effect of 0.185 with two asterisks, described as about 20 percent of respondents, and a stronger effect in the image condition of 0.342 with three asterisks. H4 is marked Directional only, with a pooled coefficient of 0.144 with p greater than 0.1 and a coefficient of 0.230 plus among Republicans. H5 is marked No, not significant in pooled or subgroup analyses. H6 is marked Marginally, marginal in the text condition with p less than 0.1 and null after multiple testing correction. H7 is marked Marginally, significant in the text condition with p less than 0.05 and attenuated after correction.
Results reported from primary pre-registered analyses. Holm-adjusted p-values reported in Supplementary Appendix Table 12. H3 remains significant after adjustment (p Holm = 0.015). The reversal in H1 is only marginally significant after adjustment (p Holm = 0.089).
I explore each of these results in detail below and then turn to exploratory analyses that help adjudicate between competing interpretations of this divergence.
9. Different methods yield different results
Consistent with the aforementioned research on racial bias with regards to candidates, the text and visual treatment conditions yield different results. Similarly, consistent with Trounstine’s (Reference Trounstine2021) conjoint approach, the factorial experiments fail to recover evidence of racial bias. However, the list experiment, as well as an exploratory manipulation check, confirms that this is likely a design failure rather than a sincere belief among my respondents. Lastly, exploratory analyses reveal substantial partisan differences. Republicans and Democrats respond differently from each other in both the factorial and list experiments.
10. Factorial experiments show minimal evidence of racial bias
In Table 4, we can see the treatment effects in the pooled, text, and image conditions for the full study population. Clearly, apartmentsFootnote 14 and higher-priced housing elicit more opposition. This is consistent with previous research. People do not want apartments, nor more expensive housing, being built in their neighborhoods. However, on the main variable of interest, the coefficient is actually in the opposite direction from the expectation in Hypotheses 1 and 2. In the factorial experiment, respondents are actually more likely to say that they would support a given housing development when Black residents are depicted as moving in. Further, if a researcher were to run just one of these experiments, image or text, they would come to a different conclusion about whether race produced a significant response.Footnote 15
Factorial experiment results
Table 4 Long description
Three model columns, pooled, text, and image, report coefficients with standard errors in parentheses for Apartment, Black Family Treatment, and two price levels, plus fit statistics. Apartment is negative in all columns: minus 0.082 with standard error 0.016 in pooled, minus 0.070 with 0.023 in text, and minus 0.097 with 0.023 in image, each with significance markers. Black Family Treatment is positive: 0.041 with 0.017 in pooled, 0.037 with 0.023 in text, and 0.046 with 0.023 in image, with significance markers on pooled and image only. Price at 3000 per month is negative: minus 0.134 with 0.021 in pooled, minus 0.116 with 0.029 in text, and minus 0.152 with 0.028 in image, each with significance markers. Price at 4500 per month is negative: minus 0.159 with 0.020 in pooled, minus 0.119 with 0.029 in text, and minus 0.193 with 0.028 in image, each with significance markers. Num. Obs is 3491 pooled, 1706 text, and 1785 image. AIC is 4737.8 pooled, 2260.2 text, and 2397.0 image; BIC is 4768.6 pooled, 2287.4 text, and 2424.4 image; RMSE is 0.49 pooled, 0.48 text, and 0.49 image.
+p < 0.1, *p < 0.05, **p < 0.01, ***p < 0.001.
11. The list experiment suggests substantial racial hangups
These results contrast with the list experiment, where, as we can see in Table 5, the results cleanly align with the expectations in Hypothesis 3,Footnote 16 and are directionally consistent with Hypothesis 4, significantly so (at the 10% level) for Republicans, as seen in Table 9 below.
List experiment: Hypotheses 3 and 4
Table 5 Long description
Regression results are reported for two models, ICT Regression full sample and OLS Regression sensitive list only. In ICT Regression full sample, Sensitive Item is 0.185 with two asterisks and its standard error is 0.060. In the same column, Control List is 2.518 with three asterisks and its standard error is 0.040. In OLS Regression sensitive list only, Constant is 2.631 with three asterisks and its standard error is 0.063. In that OLS column, Black Treatment is 0.144 and its standard error is 0.090. The number of observations is 3494 for ICT Regression full sample and 1737 for OLS Regression sensitive list only.
+p < 0.1, *p < 0.05, **p < 0.01, ***p < 0.001.
Recall that, when presented with the factorial experiment, on average, respondents reported they would be more likely to support development in the Black treatment condition. In contrast, when presented with a list of reasons why someone might be concerned about new housing in their neighborhood, on average, they report a higher number of concerns if they get the list with the explicit racial concern. This suggests that the positive effects for the Black treatment in the factorial experiment may be driven by social desirability or virtue signaling. Once given cover, many respondents readily admit to being concerned about non-White people moving into their neighborhood if more housing is built.
The dependent variable is whether respondents answered “Yes” when asked if they would support the development in the profile if it were put to a public vote, reported as the marginal effect on the probability of voting yes.
The sensitive item read “I am concerned that new housing would cause non-White people to move into my neighborhood.”
To provide a bit of intuition as to how substantively meaningful the result here is, it is equivalent to nearly 20% of respondents choosing a higher number of concerns when the only plausible reason for doing so is the inclusion of the explicit racial item. Remarkably, even with the purpose of the list experiment being to give respondents cover because they can answer without explicitly admitting to racial bias, just under 10% of those presented with the explicit racial list reported that they had all six concerns.
12. The results of the factorial experiment likely reflect social desirability
The fact that the point estimates are positive for the Black family treatment has two plausible explanations. It may reflect genuine preference for Black neighbors among my respondents, or it may reflect social desirability bias. The results of the full list experiment suggest that genuine preference is unlikely, but on its own this result is not dispositive. An additional falsification test provides me more purchase.Footnote 17
The logic here is straightforward. It cannot simultaneously be the case that respondents are, in the aggregate, sincerely more likely to approve of a development when there is a Black family moving to the neighborhood, and then also report that they are concerned about Black people moving to the neighborhood when given cover to do so. If the positive point estimates were reflective of genuine preference for Black neighbors, then those who voted yes in the factorial experiment—which includes those who saw the Black family treatment—should not be expressing the same level of racial concern in the list experiment. After all, they just said they would approve of housing that would bring Black neighbors. Put differently, if genuine preference is driving the positive factorial coefficients, then the evidence of racial bias that we observe in the list experiment should be localized only to those who said they would vote against the proposed development, or at the very least the coefficient for the “Yes’s” should be markedly smaller than that for the “No’s.” In contrast, if social desirability is what’s driving the positive coefficients, then we should see racial concerns in the list experiment among both groups, since the list format gives everyone cover.
We observe results consistent with the latter case. As we can see in Table 6,Footnote 18 the estimates are virtually identical. Unsurprisingly, those who reported they would vote no on the proposed development reported a higher number of concerns on average. But the relevant result here, that both groups appear to have identical racial hangups, provides strong evidence that the positive effects we see on the Black family treatment in the factorial experiments do not reflect genuine preference.
List experiment: yes vs. no
Table 6 Long description
ICT Regression results are reported for two columns labeled, Yes and No. For Sensitive Item, the coefficient is 0.177 with one asterisk in Yes and 0.176 with one asterisk in No; the standard errors are 0.085 for Yes and 0.084 for No. For Control List, the coefficient is 2.256 with three asterisks in Yes and 2.759 with three asterisks in No; the standard errors are 0.055 for Yes and 0.057 for No. Num. Obs is 1638 for Yes and 1853 for No.
+p < 0.1, *p < 0.05, **p < 0.01, ***p < 0.001.
Additionally, as I discuss further below, the positive coefficient is driven specifically by Democrats in the text condition—a pattern consistent with progressive Whites using the transparent factorial to signal non-racism, as documented by Salerno et al. (Reference Salerno, Kulak, Smalarz, Eerdmans, Lawrence and Dao2023). Critically, this Democratic effect differs substantially between the text and image conditions (see Supplementary Appendix Table 10). If it reflected genuine policy belief, we would not expect such divergent results across treatment modes. Third, the list experiment results are substantially stronger in the image condition (Supplementary Appendix Table 11), where respondents were less primed on race. This pattern largely rules out priming as an alternative explanation and instead suggests that the text condition may cue social desirability that partially suppresses even the list experiment results. The lack of a significant effect in the list experiment for Democrats does admit the possibility of genuinely pro-diversity preferences among this group, but as noted below, this may reflect a lack of power in the subgroup analysis rather than a true lack of any discriminatory preferences.
All this said, this is a limitation of the present study as I lack direct measures of social desirability propensity such as self-reported comfort with racial cues or process measures like individual item response times. Without this direct evidence, this remains an unresolved question. While these convergent findings provide substantial, if indirect, evidence for social desirability bias, I cannot definitively rule out genuine belief as a partial explanation, particularly among Democrats. Future research employing direct measures of deliberative processing or self-monitoring would help further validate these mechanisms.
13. Information equivalence
We might also be concerned about information equivalence and treatment receipt in the factorial experiments. In order to further investigate the differences between the text- and image-based factorial experiments, I ran a second surveyFootnote 19 to check the success of the manipulation for the visual treatments. This survey presented 430 White respondents with one of the visual treatments and asked several questions to gauge whether the visual treatment worked as desired. The results suggest that many respondents are not receiving the treatment as intended.
Only 68.14% of respondents correctly answered the question asking them to identify the racial group of the family in the treatment. Moreover, this is likely a high estimate of just how many respondents actually noted the racial group properly, as some of the respondents in the manipulation check who got this question correct undoubtedly got it correct by chance.
Similarly, as Figure 4 shows, the proportion of respondents who believed that the treatment was “Likely” or “Very Likely” meant to convey that Black families were going to live in the development was less than 50%. Even adding in those who thought that the likelihood of such a meaning was “50-50” still gets us less than 75%. As a result, even using these high watermarks, only about 50% of respondents presented with the image version of the factorial experiment actually received the intended information. Therefore, in addition to the aforementioned theoretical differences suggested by the literature, it may be the case that visual treatments do not mechanically convey racial treatments as effectively as researchers would like.
Reasons for inclusion of Black family.
Figure 4 Long description
Reasons for Inclusion of Black Family. A) A bar chart with the title, Abstract from Black Family. The horizontal axis label is, Response. The vertical axis label is, Percent. The vertical axis range is 0 percent to 40 percent. The bars, from left to right, are labeled, Very Unlikely, Unlikely, 50 50, Likely, Very Likely. Bar heights are approximately 5 percent, 10 percent, 35 percent, 40 percent and 15 percent. B) A bar chart with the title, Avoid Taxes. The horizontal axis label is, Response. The vertical axis label is, Percent. The vertical axis range is 0 percent to 30 percent. The bars, from left to right, are labeled, Very Unlikely, Unlikely, 50 50, Likely, Very Likely. Bar heights are approximately 10 percent, 20 percent, 30 percent, 30 percent and 15 percent. C) A bar chart with the title, Heighten Density. The horizontal axis label is, Response. The vertical axis label is, Percent. The vertical axis range is 0 percent to 40 percent. The bars, from left to right, are labeled, Very Unlikely, Unlikely, 50 50, Likely, Very Likely. Bar heights are approximately 0 percent, 5 percent, 15 percent, 40 percent and 35 percent. D) A bar chart with the title, No Meaning. The horizontal axis label is, Response. The vertical axis label is, Percent. The vertical axis range is 0 percent to 30 percent. The bars, from left to right, are labeled, Very Unlikely, Unlikely, 50 50, Likely, Very Likely. Bar heights are approximately 15 percent, 25 percent, 30 percent, 25 percent and 15 percent. E) A bar chart with the title, Open to Black Families. The horizontal axis label is, Response. The vertical axis label is, Percent. The vertical axis range is 0 percent to 30 percent. The bars, from left to right, are labeled, Very Unlikely, Unlikely, 50 50, Likely, Very Likely. Bar heights are approximately 5 percent, 10 percent, 25 percent, 30 percent and 30 percent. F) A bar chart with the title, Signal Race Integration. The horizontal axis label is, Response. The vertical axis label is, Percent. The vertical axis range is 0 percent to 30 percent. The bars, from left to right, are labeled, Very Unlikely, Unlikely, 50 50, Likely, Very Likely. Bar heights are approximately 10 percent, 20 percent, 30 percent, 25 percent and 15 percent.
Of course, this does not mean that all visual treatments do not work, just that the treatment I included was too subtle. The same follow-up survey (full n = 1,231) also included two more overt versions of the image treatment. In the more overt cases, 73.9% and 81% of respondents correctly identified the race of the family in the fliers. The difference between the worst-performing and best-performing treatment is significant at the α = 0.0001 level. These two additional versions appear in the Supplementary Appendix.Footnote 20 Nevertheless, my results still indicate that researchers must take care to ensure that their treatments actually convey the intended information. Further research is needed in order to understand this area more completely.
14. Subsidiary mechanism tests also suggest racialized issue salience
The subsidiary mechanism testFootnote 21 yields interesting results, shown in Figure 5 below, both from a substantive and methodological point of view. Consistent with previous work (see e.g. Trounstine, Reference Trounstine2021; Elmendorf et al., Reference Elmendorf, Nall and Oklobdzija2024), apartments are seen as worsening almost every element of the neighborhood. Similarly, unsurprisingly, more expensive housing is seen as worsening affordability. Interestingly, those assigned to the Black treatment also report that they believe new housing will improve affordability, while those assigned to the apartment condition do not (this is also consistent with the developing literature on this question; see Clayton et al., Reference Clayton, Elmendorf and Oklobdzija2024). This is despite the fact that there is substantial evidence of the latter and little evidence, if any, of the former.
Subsidiary mechanism tests by treatment mode.
Figure 5 Long description
Dot-and-whisker plots illustrate estimated effects on various neighborhood aspects across different treatment modes: Black Treatment, Apartment, Price: 4500 dollars and Price: 3000 dollars. The x-axis is labeled ′Estimated effect′ with no specific unit, ranging from -0.3 to 0.6. The y-axis lists categorical conditions. Each plot includes three series: pooled (black circles), text (blue triangles) and image (orange circles), with error bars indicating confidence intervals. The zero-effect baseline is marked, showing where intervals overlap or exclude zero. Key findings include: Crime shows mixed effects with intervals crossing zero; School effects are mostly negative; Property values have positive effects for higher price treatments; Traffic effects are generally negative; Neighborhood character shows mixed results; Affordability is positively impacted by Black Treatment but negatively by apartment conditions. The plots provide a visual comparison of treatment impacts, highlighting areas of uncertainty and significant effects.
The more interesting piece however, is that pooled effects would once again mask substantial heterogeneity between the image and text conditions. As shown in Figure 5, the text condition yields significant results that are largely consistent with hypotheses five through seven on School Quality, Neighborhood Character, and also Property Values. These are the patterns that we would expect to see if race was playing a role in other observable mechanisms, as well as if the text condition were more explicitly priming race than the image condition. Though I did not have clear theoretical expectations for the last measure on property values, it is likewise consistent with the kind of “statistical discrimination” we would expect to see if Whites are indeed inferring that the mere presence of non-White neighbors will reduce the value of their property. This is consistent with what we know was true about White reactions to non-White newcomers during the mid-20th century: they did not want non-White neighbors (see e.g. Trounstine, Reference Trounstine2018, Reference Trounstine2020; Rothstein, Reference Rothstein2017; Ananat, Reference Ananat2011; Wang, Reference Wang2024; Reny and Newman, Reference Reny and Newman2018 for discussions of various dimensions of these historical phenomena). It would seem that many people still have an aversion to non-White neighbors even half a century later.
These results once again demonstrate that the choice of a text- vs. image-based treatment yields clearly different results, not only on the main outcome measure but also on the tests that we use to determine mechanisms. They also further indicate that the results from the list experiment are closer to the ground truth than the results from the factorial experiment. In the text condition, where the racial treatment is strongest, those receiving the Black treatment report that they believe new housing will worsen only those things which are racialized.
15. Republicans and Democrats react differently to all experimental conditions
The Average Treatment Effects of the Factorial Experiment above also mask substantial heterogeneity between Republicans and Democrats.Footnote 22 In both the text condition and the visual condition, Republicans and Democrats answer differently when presented with the same kinds of profiles.
Similarly, disaggregating Republican and Democratic respondents (shown in Table 7) shows the only remaining positive and significant coefficient is for Democrats in the text condition. This is precisely what we would expect to see if progressive Whites are taking advantage of the factorial experiment to prove their non-racist credentials.
Republicans and Democrats respond to the factorial experiment differently
Table 7 Long description
Four columns report coefficients and standard errors in parentheses for Republican, text; Republican, image; Democrat, text; and Democrat, image. Apartment coefficients are minus 0.121 with three stars and standard error 0.033 for Republican, text; minus 0.104 with two stars and 0.034 for Republican, image; minus 0.070 with a plus and 0.037 for Democrat, text; and minus 0.110 with two stars and 0.036 for Democrat, image. Black Treatment coefficients are minus 0.041 with 0.033 for Republican, text; 0.048 with 0.034 for Republican, image; 0.109 with two stars and 0.037 for Democrat, text; and minus 0.007 with 0.036 for Democrat, image. Price: \$3000/m coefficients are minus 0.125 with two stars and 0.042 for Republican, text; minus 0.100 with one star and 0.043 for Republican, image; minus 0.142 with two stars and 0.047 for Democrat, text; and minus 0.202 with three stars and 0.043 for Democrat, image. Price: \$4500/m coefficients are minus 0.096 with one star and 0.042 for Republican, text; minus 0.171 with three stars and 0.042 for Republican, image; minus 0.190 with three stars and 0.045 for Democrat, text; and minus 0.186 with three stars and 0.044 for Democrat, image. Model statistics list Num. Obs as 792, 826, 689, and 695; AIC as 1011.2, 1124.5, 933.6, and 906.3; BIC as 1034.6, 1148.1, 956.2, and 929.1; and RMSE as 0.47, 0.49, 0.49, and 0.48, in the same column order.
+p < 0.1, *p < 0.05, **p < 0.01, ***p < 0.001.
Moreover, whatever the reason behind these differential effects for Republicans and Democrats, it is readily apparent that the type of treatment has an effect on how both respond to the Black treatment condition. Table 10 in the Supplementary Appendix demonstrates that the difference between the coefficients on the Black Treatment variable in the text condition and the visual condition is statistically significant for both Republicans and Democrats.
These inter-party differences are also apparent in the list experiment (Table 8). Republicans not only choose more items on average when presented with the second list while Democrats do not (at least not significantly so), but they also report a higher number of baseline concerns on average. This suggests that Republicans are both more likely than Democrats to be motivated by concerns about non-White neighbors, and that they have more concerns about new housing development in general than do Democrats. This is also mirrored in the pure averages of the choice to support new housing. Republican respondents approved of the development they were presented with 43.04% of the time, while Democrats approved of 54.44% of their presented developments.
List experiment: Republicans vs. Democrats
Table 8 Long description
ICT Regression reports coefficients and standard errors for Republicans and Democrats. For Sensitive Item, Republicans have 0.239 with a star and a standard error of 0.087, while Democrats have 0.119 with a standard error of 0.095. For Control List, Republicans have 2.771 with three stars and a standard error of 0.059, while Democrats have 2.278 with three stars and a standard error of 0.063. Num. Obs is 1618 for Republicans and 1384 for Democrats.
+p < 0.1, *p < 0.05, **p < 0.01, ***p < 0.001.
Hypothesis 4—Republicans vs. Democrats
Table 9 Long description
Regression coefficients are reported for three columns labeled, Pooled, Republicans, and Democrats, with standard errors shown on the next line in parentheses for each row. For Constant, the coefficients are 2.631 with three stars in Pooled, 2.900 with three stars in Republicans, and 2.317 with three stars in Democrats; the standard errors are 0.063, 0.088, and 0.104. For Black Treatment, the coefficients are 0.144 in Pooled, 0.230 with a plus in Republicans, and 0.150 in Democrats; the standard errors are 0.090, 0.128, and 0.143. The number of observations is 1737 for Pooled, 803 for Republicans, and 683 for Democrats.
+ p < 0.1, *p < 0.05, **p < 0.01, ***p < 0.001.
Lastly, as stated earlier, Republicans alone are significantly more likely to select a higher number of items in the second list when they are in the Black treatment condition.
It is also important to note that neither Republican nor Democratic respondents reported that they agreed with a higher number of concerns in the control list when they were assigned to the Black treatment. This adds credence to the argument that the significant effects for Republicans in Table 8 are indeed driven by the explicit racial concern being present, rather than simply being more concerned in general if they are in the Black treatment condition.
All this said, we should take the results in this section with a grain of salt. The Democratic coefficient of 0.119 is positive and, if it were significant, substantively nontrivial. With a standard error of 0.095, the confidence interval is consistent with an effect comparable in magnitude to that found for Republicans. The nonsignificance here may reflect insufficient power in the subgroup rather than a genuine absence of racial concerns among Democratic respondents. Moreover, as discussed above, the list experiment is not mechanically designed to detect pro-Black preferences. It can only detect the expression of an anti-Black concern. The Democratic null in the list experiment therefore tells us that Democrats are not expressing anti-Black concerns at a detectable rate, but it cannot tell us whether their positive factorial coefficients reflect genuine pro-diversity preferences, social desirability, or some combination of the two. For that, we need to turn to the evidence in previous sections. This evidence suggests, in my view, relatively strongly, that social desirability is doing a substantial amount of the work, but it is not dispositive. This question remains an open area for future research.
16. Discussion
My results suggest that factorial experiments, in either text or visual form, may fail to accurately capture racial biases relative to more validated techniques like list experiments. And importantly, it is only the measures of racial bias that behave this way. The results for the price and apartment variables behave very similarly and as expected across the image and text conditions. At the very least, the fact that image-based and text-based treatments do indeed yield different effects for the racial treatment should be enough to give researchers pause. On the main choice outcome, for both Republicans and Democrats, the estimated coefficients for the racial treatment in the text and image models are significantly different from one another.Footnote 23 The same is true for the subsidiary mechanism test. Researchers need to be careful that they do not draw sweeping conclusions based on one technique.
Furthermore, my results suggest that the positive point estimates we observe are more likely than not the result of social desirability/virtue signaling rather than genuine preference. As noted above, it is plausible that Democrats are expressing true pro-diversity preferences in the factorial experiment, but the evidence consistent with this interpretation must be weighed against the (in my view stronger) evidence that indicates social desirability is more impactful. For Republicans, the null factorial effects are difficult to reconcile with their significant racial concerns in the list experiment, and the discrepancy between the two is also most parsimoniously explained by social desirability. The Table 6 falsification test—showing identical racial concerns among respondents who voted yes and no in the factorial—provides additional evidence that, at least in the aggregate, the positive factorial coefficients do not reflect purely genuine preference.
Second, on the substantive elements, the more validated technique—the list experiment—suggests that there are indeed a nontrivial number of people whose opposition to housing is motivated by race. Lastly, the significant results from the subsidiary mechanism test suggest that, at least when the racial treatment is overt, those who are in the Black treatment condition are more likely to report that new housing will worsen their neighborhoods along the expected racialized dimensions.
This paper demonstrates that how we study opposition to housing matters just as much as what dimensions of that opposition we study. The results from these three experimental approaches reveal not only the presence of racial bias in housing opposition but also how methodological design can obscure or reveal these biases. Text- and image-based factorial experiments point to minimal evidence of racial bias, but the more validated list experiment uncovers meaningful racial hangups for a meaningful subset of respondents. These differences highlight the need for caution in interpreting results from any single method, particularly on racialized topics, where social desirability and virtue signaling may play a role.
Acknowledgements
I would like to thank Julia Payson and Michelle Torres for many helpful comments on previous versions of this article, as well as participants in the American Politics Research Group and Political Psychology Lab at UCLA, the 2025 Local Political Economy Conference, and Jessica Trounstine for providing thoughtful feedback on conference and lab drafts.
Author bios
Clayton Becker is a PhD Candidate in Political Science at The University of California, Los Angele.
Declaration of interests
I have no competing interests to declare in the production of this work.
Funding statement
I acknowledge support from the UCLA Graduate Division Graduate Summer Research Mentorship program as well as Julia Payson, Michelle Torres, Lynn Vavreck, and Chris Tausanovitch in helping to fund the survey data collection.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/psrm.2026.10131. To obtain replication material for this article, please visit https://url.uk.m.mimecastprotect.com/s/OeerCJqDDi8rRjQgtVf33iylqN0?domain=doi.org.
Open access
