1 Introduction
Names are common treatments when the goal is to estimate the effect of a characteristic—such as race or ethnic minority status—on a behavioral outcome, and name-based treatments have been used in observational studies (Barth, Mittag, and Park Reference Barth, Mittag and Park2019; Broockman and Soltas Reference Broockman and Soltas2020), survey experiments (Doherty, Dowling, and Miller Reference Doherty, Dowling and Miller2019), and field experiments (Butler and Broockman Reference Butler and Broockman2011; Costa Reference Costa2017). In audit studies, researchers use fictional aliases to manipulate a desired characteristic and measure discrimination based on the different service received (Butler and Homola Reference Butler and Homola2017). In this letter, we combine a name perception study and an audit study of state legislators to show that names can manipulate perceptions of multiple characteristics, such as minority status and socioeconomic status (SES). This implies that such a treatment does not meet the information equivalence assumption that no characteristic(s) other than the intended one is manipulated by the treatment (Dafoe, Zhang, and Caughey Reference Dafoe, Zhang and Caughey2018). If this assumption is not met, then the treatments change experimental participants’ beliefs about both intended and unintended characteristics of the hypothetical people in a name-based experiment, and, therefore, estimated treatment effects will not reflect solely the intentionally manipulated characteristic. The information equivalence assumption is a specific type of violation of the exclusion restriction that is well known from instrumental variable approaches to estimating treatment effects (Dafoe et al. Reference Dafoe, Zhang and Caughey2018). Prior research on whether names manipulate multiple characteristics of a hypothetical individual applied the instrumental variables framework (Butler and Homola Reference Butler and Homola2017), but we choose to use the more recent and specific information equivalence framework. In the context of experiments that choose names as a treatment, this approach more directly captures the key issue associated with names manipulating multiple factors and thereby complicating design of experiments and interpretation of estimated causal effects.
Names are used in research to signal minority status because certain names are more likely to be used by blacks (Cook, Logan, and Parman Reference Cook, Logan and Parman2014) and foreign-origin minorities (Abramitzky, Boustan, and Eriksson Reference Abramitzky, Boustan and Eriksson2019), and therefore, these names can signal minority status. Distinctive names may also convey information about socioeconomic status (Fryer and Levitt Reference Fryer and Levitt2004). Political scientists have examined if names modify multiple factors by using administrative data to determine characteristics associated with some of the distinctive names used in prior research and did not find evidence that characteristics other than minority status were associated with legislative reply rates (Butler and Homola Reference Butler and Homola2017). However, characteristics drawn from administrative data may not correspond with actual perceptions, and it is perceptions that will affect responses. To test if names manipulate perceptions of multiple characteristics and violate the information equivalence assumption, we would ideally measure the perceptions of those being studied, in our case legislators and their staffers, but this proved logistically difficult when we attempted to do so.Footnote 1 Instead, we measured nonelites’ name perceptions and found that names chosen to signal racial distinctives also vary in perceived SES and migrant status. Other research has also shown that names can manipulate factors other than race (Barlow and Lahey Reference Barlow and Lahey2018). We further show that these unintentionally manipulated characteristics are associated with an outcome of interest by combining a name perception study with an audit study of over 4,000 state legislators to examine if variation in perceptions of racial and nonracial characteristics is related to whether a hypothetical constituent receives a reply. Using several putatively white, Hispanic, and black names, we show that reply rates in our audit study are related to perceptions of a name’s SES and migrant status. These results suggest that name-based treatments are more multifaceted than previously recognized, and that these different aspects of the treatments are related to behavior. We illustrate the problem using an audit study, but the unintentional manipulation of multiple characteristics is a general problem for research using names to signal a single causal factor (Barth et al. Reference Barth, Mittag and Park2019; Doherty et al. Reference Doherty, Dowling and Miller2019). Our conclusions also raise issues more generally about the design of experiments in which an informational treatment manipulates respondents’ beliefs and suggest the need to test regularly if the treatment violates information equivalence.
2 Audit and Perceptions Studies
We fielded our two studies, a name perception study and an audit study, to coincide with the 2018 elections. Our study of name perceptions sheds light on how people perceive the SES, race and ethnicity, gender, and migrant status of the types of names used in audit studies. The audit study used the same names to examine how state legislative offices respond to the 12 names, and whether the response rates from legislators are related to perceptions of race and ethnicity, SES, or other characteristics.
We conducted the two studies at similar points in time to ensure perceptions of the names are not affected by differences in the timing of the studies. Ideally, we would measure the perceptions of the politicians and/or staffers who respond to constituent emails (Landgrave and Weller Reference Landgrave and Weller2020a), but eliciting these actors’ perceptions proved too difficult. Instead, we used U.S. MTurk respondents, which is common in name perception studies (Gaddis Reference Gaddis2017; Barlow and Lahey Reference Barlow and Lahey2018).Footnote 2
We used 12 distinctive male names—four each to signal white, Hispanic, and black identities. For black names, we relied on distinct first names to signal race, and, for Hispanics, we selected based on distinct last names. We used information from the U.S. Census Bureau and the Social Security Administration to identify distinctive names.
In the name perception study, MTurk respondents were asked about a name’s perceived gender, race and ethnicity, migrant status, and SES because these characteristics are likely to be related to each other and therefore might be manipulated by the names used in audit studies. Respondents were randomly assigned to rate six names, and each name received about 65 ratings from different respondents with 446 unique respondents rating at least one of the 12 aliases used in the audit study. See Appendix A for further details.
For the audit study, all U.S. state legislators with a valid public email address (n = 5,735) were randomly assigned to receive an email address from one of the 12 names used in the perception study. After excluding invalid emails, the audit presented here includes results from 4,460 state legislators in 43 states.Footnote 3 In the audit study, fictional college-age constituents asked state legislative offices for help registering to vote as in Butler and Broockman (Reference Butler and Broockman2011). See Appendix B for full email composition details.
3 Results
We focus on two types of results: the relationship between names and perceived characteristics and the relationship between perceived characteristics and legislative responses. In Table 1, we estimate the relationship between a name’s characteristics as perceived by MTurk respondents and a name’s racial category using ordinary least squares (OLS):
${\mathrm{Perceived}\ \mathrm{Characteristic}}_i={B}_0+{B}_1\mathrm{Purportedly}\ {\mathrm{Black}\ \mathrm{Name}}_i+{B}_2\mathrm{Purportedly}\ {\mathrm{Hispanic}\ \mathrm{Name}}_i$
.
Perceived name characteristics by purported race and ethnicity.
Standard errors in parentheses and are clustered by MTurk respondent.
The results in Table 1, column 1, show that as expected, both the black and Hispanic names are much less likely to be perceived as white compared to names chosen to represent white constituents (the excluded category).Footnote 4 In Table 1, column 2, we show that the putatively black and Hispanic names were respectively 12.9 percentage points (p-value = 0.001) and 13.6 percentage points (p-value = 0.001) more likely to be perceived as low SES compared to the white names, which is consistent with prior literature (Gaddis Reference Gaddis2017; Barlow and Lahey Reference Barlow and Lahey2018). Hispanic names are 22.5 percentage points (p-value = 0.001) more likely than white names to be perceived as belonging to someone who is a migrant, and black names are no more likely than white names to be perceived as migrants (p-value = 0.187) as shown in Table 1, column 3. The results in Table 1 show substantial differences in perceptions of race and ethnicity, socioeconomic status, and migrant status. These results indicate that names used in audit and other studies may manipulate multiple characteristics at the same time.
We turn next to the results from our audit study to investigate whether any of the perceived characteristics of the names are associated with differential reply rates from state legislators (Figure 1). Our first analysis looks at the proportion of emails from each constituent that receives a response from the legislative offices. We estimate the relationship via OLS:
${\mathrm{ReplyRate}}_i={B}_0+{B}_1\mathrm{Perceived}\ \mathrm{Low}\ {\mathrm{SES}}_i$
. Perceived low SES is a continuous variable indicating the portion of respondents who viewed the name as working class (the lowest rating category). Overall results are presented in Figure 1, and the results are also separated out by a name’s purported race and ethnicity. Across all names, perceived low SES is associated with a lower reply rate (
${B}_1$
coefficient: −0.29, p-value = 0.01), and this also holds true within each racial subcategory. Researchers have interpreted audit studies in the U.S. context as indicating that the minority status of distinctive names explains differences in reply rates (Butler and Homola Reference Butler and Homola2017; Gell-Redman et al. Reference Gell-Redman, Visalvanich, Crabtree and Fariss2018), but our results show that perceived SES also affects discrimination.
Reply rate by name’s perceived socioeconomic status (SES) and purported race and ethnicity.
Reply rate by name’s perceived characteristics.
Robust standard errors in parentheses.
As we showed earlier, there are differences across the names in perceptions of several descriptive factors. To understand the relationship between these factors and replies by legislative offices, we estimate the following linear probability model model using robust standard errors:
${\mathrm{Reply}}_i={B}_0+{B}_1\mathrm{Perceived}\ \mathrm{Low}\ {\mathrm{SES}}_i+{B}_2{\mathrm{Perceived}\ \mathrm{Whiteness}}_i+{B}_3{\mathrm{Perceived}\ \mathrm{Migrant}}_i$
. The dependent variable is whether an email receives a response, and perceived low SES, whiteness, and migrant status are continuous measures of the proportion of respondents who perceive a name as indicating low SES, whiteness, or migrant status based on the coding described earlier. In Table 2, column 1, we examine the standard treatment effect in audit studies about race. We do not find a difference between white and black names in response rates, but we do find that Hispanic names receive about 6 percentage points fewer responses than white names. In column 2, we show that there is a significant negative relationship between the perceived low SES of a name and replies from legislative offices (p-value < 0.001). We also examine the relationship between perceptions of Hispanic/black from the name study and response rates in column 3, and we find the same pattern when we use the purported race of the names—Hispanic names receive fewer responses than white names, but there is no difference between black and white names in response rates.Footnote
5
Columns 4 and 5 demonstrate that Hispanicity (both purported and perceived) and low SES continue to be related to fewer legislative replies even after we also adjust for perceptions of race, gender, and migrant status. Taken together, these results demonstrate that names used in audit studies can manipulate multiple factors associated with the outcome of interest and may cause us to make a mistaken inference about the relationship between a single characteristic (in this case, race) and the outcome of interest.
4 Conclusion
The results in this paper demonstrate that perceptions of socioeconomic status and migrant status vary with whether a name is purportedly white, black, or Hispanic, and, therefore, these treatments do not meet the information equivalence assumption making it difficult to attribute estimated name-based discrimination to a single factor (i.e., effect of race on reply rates in an audit study). We find that once we account for perceived SES and migrant status, the reply rate is no longer solely associated with minority names. Although perceived SES is an important explanatory variable, including when subset by race and ethnicity, it is highly correlated with perceived whiteness (pairwise correlation = −0.69), further demonstrating how names can manipulate multiple characteristics. Our results are not meant to downplay discrimination against minority constituents but rather emphasize that it is difficult to identify a single cause for the discrimination. Our findings corroborate the argument that identity is a bundle of characteristics that cannot be easily separated from each other (Sen and Wasow Reference Sen and Wasow2016).
Name-based treatments are common in observational and experimental studies that seek to understand the differential effect of identity on an outcome, and our findings suggest that names can violate the information equivalence assumption. Other studies that rely on name-based treatments may also suffer from this problem, and researchers should seek to understand if the assumption of information equivalence is violated by a name-based treatment and, if so, understand the various characteristics manipulated by the treatment. There are multiple solutions outlined in Dafoe et al. (Reference Dafoe, Zhang and Caughey2018) to address violations of the information equivalence assumption. One option in the context of name-based treatments is to develop measures of the informational equivalence violations and use the measures in a regression analysis, but this may not provide causal estimates for any given factor, given that the treatment manipulates all factors at once. Another option is to design the experiment such that the name only manipulates perceptions of a single factor by somehow fixing respondents’ perceptions of other factors or choosing an outcome that is only manipulated by the intended factor even if the treatment manipulates other factors. Regardless of which approach researchers take, the most important step is to attend carefully to the fact that names can manipulate multiple characteristics thereby causing violations of information equivalence, and as such studies need to address these issues in both their design choices and interpretation of estimated treatment effects.
More generally, these results speak to the challenge of designing informational treatments that manipulate the intended, and only the intended, causal factor, which is an important component of understanding the meaning of an estimated causal effect. Dafoe et al. (Reference Dafoe, Zhang and Caughey2018) demonstrate the informational equivalence problem in the context of survey experiments, and we show that it also exists in studies using names as treatments. McDermott (Reference McDermott1998) made a similar point in the context of low-information elections by pointing out that a candidate’s identity (race and/or gender) can signal ideology, suggesting that some candidate evaluation studies may violate the information equivalence assumption. Together, these results suggest that violations of the information equivalence assumption may be quite common and occur across substantive research areas. Moving forward, researchers should consider the frequency of information equivalence violations, account for it in the experimental design, and interpret results carefully considering this possibility.
Acknowledgement
We thank the University of California Institute for Mexico and the United States (UCMEXUS), for financial support for the experiments reported herein.
Data Availability Statement
Replication data are available at https://doi.org/10.7910/DVN/4LO1XE (Landgrave and Weller Reference Landgrave and Wellers2020b).
Supplementary Materials
To view supplementary material for this article, please visit http://dx.doi.org/10.1017/pan.2020.52.
