DATA AND RESULTS

We obtained the 2018 Florida and North Carolina voter registration files from the data and analytics firm The Data Trust, LLC. The Florida and North Carolina voter files contain the address and self-reported race of each registered voter in each state. We then combined this with demographic information for each voter’s census tract.

To benchmark the model against the results in Imai and Khanna (Reference Imai and Khanna2016), we begin by implementing the predicted race model on the 2018 Florida and North Carolina files and obtain the overall error rate by comparing the predicted ethnicity for each voter to the voter’s self-reported race.Footnote ³ While our 2018 files are more recent, the overall results are similar to those reported in Table 1 of Imai and Khanna (Reference Imai and Khanna2016). The model allows for the inclusion of additional variables beyond surname and census tract, including political party, age, and gender. We present the results of the model that uses surname, census tract, and party as that is the model used in Imai and Khanna (Reference Imai and Khanna2016). However, the results of models that include additional covariates are shown in A.1 in the Supplementary Material. The substantive conclusions reached do not depend on the BISG specification.

The BISG model has a 15.1% error rate in Florida and a 15.4% error rate in North Carolina.Footnote ⁴ However, this rate varies dramatically across race and ethnicity. In Florida and North Carolina, 6.7% and 6.9% of self-identified white voters are incorrectly classified as being non-white. On the other hand, 24.7% and 31.6% of voters who self-identify as non-white are incorrectly predicted to be white.Footnote ⁵ Future researchers are advised that in some applications, using predicted probabilities may be more appropriate than a deterministic racial classification based on highest probability.

Looking further down the BISG columns shows that in some areas, the model excels and in others it struggles significantly. The model is very good at avoiding false positives among minority groups (i.e., incorrectly classifying people as belonging to a minority group). For example, in Florida (North Carolina), the false positive rate is only 2.9% (5.1%) for Blacks, 3.7% (1.5%) for Hispanics, and less than 1% (0.8%) for Asians. However, the model has very high false negative rates among minority groups (i.e., incorrectly classifying a Black person as not being Black). This can be seen in the very high rates of false negatives for Blacks (33.5% FL, 34.1% NC), Hispanics (15.1% FL, 23.7% NC), and Asians (47.5% FL, 34.0% NC). In other words, among self-reported Black voters, roughly one-third of them are incorrectly classified as non-Black. These false-negative and false-positive rates are similar to those shown in Imai and Khanna (Reference Imai and Khanna2016).

Table 1 largely replicates results in Imai and Khanna (Reference Imai and Khanna2016); however, an unanswered question that remains is whether the misclassification rate is correlated with other factors. If the misclassification rates in Table 1 occur more or less at random, this would be less concerning than if the classification model is systematically wrong for particular types of individuals. Systematic misclassification could introduce significant bias into any analyses that use predicted race.

Table 1.

Replication and Extension of Imai and Khanna’s (Reference Imai and Khanna2016) Race Classification Model using 2018 Florida and North Carolina Voter Files

Note: Numbers in parentheses in the first column represent the proportion of voters in the voter file who self-identify with each racial category in FL and NC, respectively. The two columns labeled “BISG” show the classification error rates using the Imai and Khanna (Reference Imai and Khanna2016) wru model. The “Random forest” columns show the results using our proposed improvement. The random forest model is trained on 80% of the FL data, and FL results in in this table are predictions on a 20% test set. Results for NC are a true out-of-sample prediction, based on the FL-trained model.

Figure 1 shows four panels, one each for self-identified white, Black, Hispanic, and Asian individuals. In each panel, the x-axis is the median income of the individual’s census tract. The y-axis is the proportion of individuals whose race is misclassified by the BISG model. Each point shows the binned (by $1,000 increments) misclassification rate, and the red line is a weighted (by population size) lowess curve fit to those binned points. The pattern of misclassification varies dramatically by race and tract income. For whites, misclassification rates are relatively low overall and are also negatively correlated with income—that is, individuals who live in the poorest neighborhoods are the most likely to be misclassified by the model. Among Black individuals, the trend is strikingly large and in the opposite direction. Among Blacks who live in the wealthiest census tracts, the misclassification rate is nearly 100%, meaning that the model is inaccurately classifying nearly all Blacks living in wealthy census tracts. Furthermore, the misclassification rate is quite high even in modestly wealthy census tracts. The average misclassification rate approaches 50% around census tracts with a median income between $50,000 and $60,000. Among Hispanics, there is a positive correlation between census tract income and model misclassification; however, the degree of change across the figure is not as large as among Blacks. Finally, among Asian voters, there is no linear trend, but rather an S-shaped relationship, and the overall misclassification rate remains very high (approximately 50%) across all levels of income.Footnote ⁶

Figure 1.

Misclassification Rates and Census Tract Income

Note: Each panel shows the relationship between voters’ census tract income (x-axis) and the proportion of voters from each race that are misclassified by the wru model, using surname and census tract. Points, sized in proportion to number of observations, show average misclassification rate for each $1,000 increment. The line plots a lowess fit (span = 0.6) through those points, weighted by number of cases.

Whereas Figure 1 is plotting the false-negative rate (i.e., incorrectly predicting a Black person to be non-Black) across tract income, Figure A.3 in the Supplementary Material shows the false-positive rate (i.e., incorrectly classifying a non-Black person as Black) for each of the four racial groups. The results show that the high false-negative rate for non-white voters at high incomes (especially among Black voters) is due to a high false-positive rate in which the model incorrectly predicts a person to be white.

The intuition behind the correlation between socioeconomic factors and misclassification rates is quite simple: if a voter’s surname is relatively common across multiple ethnicities (i.e., Brown or Johnson for white and Black individuals), then the model will lean more heavily on inferring race from the distribution of ethnicities in the individual’s census tract. However, this will lead to errors for voters who are racially distinct from the majority of their neighbors—that is, “local minorities.” This pattern is especially true among white and Black voters because of the relative similarity of surnames between whites and Blacks, which causes the model to rely more heavily on geographic factors when making its predictions. This is less the case among Hispanics and Asian individuals for whom surnames tend to be more ethnically distinct. Figure A.6 in the Supplementary Material shows that people who live in areas where they are local minorities are much more likely to be misclassified.

Another consideration is the degree to which the racial prediction model misclassifies people’s race because of factors related to gender. The connection to gender is likely due to the fact that women are significantly more likely than men to change their surname after marriage. This, combined with the different propensities for interracial marriage across ethnicities, suggests that minority women are especially difficult to correctly classify. Figure A.10 in the Supplementary Material shows that this is the case.

IMPLICATIONS FOR RESEARCHERS AND PROPOSED IMPROVEMENT

The BISG method of predicting race from census and geographic data provides scholars a previously unavailable opportunity to incorporate racial considerations into important questions of political behavior, representation, accountability, and material well-being. However, this advancement comes with significant limitations. If racial misclassification is highly correlated with political and economic factors, estimation may be biased when incorporating these predictions into any study of political economy.

We propose an ensemble solution for improving the accuracy and minimizing misclassification biases of the BISG racial classification. We incorporate the Imai and Khanna (Reference Imai and Khanna2016) BISG probability scores for each racial class as inputs into a random forest model with other individual- and neighborhood-level political and socioeconomic predictors. The intention is to use a second model to adjust the BISG predictions so that they become less correlated with political and economic factors. The random forest is a straightforward and widely used machine learning tool for multiclass prediction. In brief, the random forest identifies potential splits in the data that maximize the accuracy of classification of observations into each racial subgroup. Because of class imbalance (the majority of Florida voters are white, whereas other racial groups, especially Asian and “Other,” are much smaller), we incorporate class weights into the model and use a macro-averaged F1-score rather than total prediction error to select the best model parameters. Both of these changes place an additional modeling emphasis on correct classification of racial and ethnic minorities, relative to correct classification of white voters. The model is trained and tuned using a randomly selected 80% subset of the Florida voter file and then “tested” on the remaining 20% of the voter file. Section A.4 of the Supplementary Material provides a complete description of training, tuning, and testing the random forest model.

We test the random forest approach to classification on two different datasets and show how it both improves classification and leads to more accurate inferences based on those predictions. First, we apply the parameters from the random forest model to the entire North Carolina voter file for a true out-of-sample prediction of each voter’s race. Second, we generate out-of-sample prediction on the held-out 20% of the Florida voter file. We focus here on the North Carolina data and present the results from the 20% Florida sample in the Supplementary Material. Racial classification predictions are of most value in contexts where racial data are not readily available, but when applying a trained model from one state to another, the researcher must make the assumption that the relationship between a person’s name, geographic location, political involvement, income, and other neighborhood characteristics and their racial identification is the same in both locations. Because no North Carolina data are used in training the random forest model, this provides a true out-of-sample test of the BISG + random forest model, which allows us to examine the performance of the predictions across state contexts given these assumptions. We find that the model performs well in both North Carolina and the withheld test data in Florida. Future researchers adopting this method should carefully consider whether this assumption continues to hold in their specific application.

While we test the accuracy of our solution using Florida and North Carolina, where self-reported race is available, we emphasize that the proposed solution, similar to the original BISG model, can be implemented anywhere that researchers have geographic location data and individual surnames, plus other desired covariates. To allow future researchers to implement our proposed improvements more easily, we have included the random forest model object in this article’s replication materials (Argyle and Barber Reference Argyle and Barber2023). In this way, future researchers with data that include the same set of publicly available predictor variables can use it for racial classification predictions without needing to retrain their own random forest model, as we demonstrate with the North Carolina predictions in the article.Footnote ⁷ Alternatively, if researchers have applications in a different domain where any different set of predictor variables are available and theoretically important, they can adapt our public replication code to train their own random forest model. This will require a set of training data in which self-reported race is available, in addition to the data for which the researcher intends to make racial classification predictions. This code uses straightforward commands in existing R packages.

We calculate the overall misclassification rates as well as the false-positive and false-negative rates for each race using the random forest improvement. Because such a high proportion of voters in Florida and North Carolina are white, overall accuracy of the BISG model can look quite good if it selects “white” for all uncertain cases. The class weights and macro-F1 optimization reduce the propensity to select the major class in uncertain cases, so that the largest and most important gains are expected to be in a reduction of false positives among white respondents and a larger reduction in false negatives among minority populations. The cost of this approach can be a reduction in the accuracy of predictions of the major class (white) because some uncertain cases that were “guessed” correctly are no longer accurately classified.

The misclassification, false-positive, and false-negative rates are reported next to the rates for the original BISG model in Table 1. The results are as expected: the BISG + random forest algorithm generates lower overall misclassification rates compared with the BISG model across all individuals of all races, but the improvements are relatively small in magnitude (around 1 percentage point). However, more dramatic improvements come when looking at the false-positive and false-negative rates for non-white respondents. For example, the random forest model in North Carolina reduces the false-negative rate among Blacks from 0.341 to 0.271, a 20.53% reduction, and the false positive rate for whites declines from 0.316 to 0.255, a 19.3% reduction. The improvements are even larger in Florida. We note that, as expected, there is a small increase in the false-negative rate of white respondents in both states. Improvements are quite small among Hispanic, Asian, and “Other” racial groups, which suggests that the ensemble method is unlikely to come at a cost of reduced accuracy for any racial group.

The improvements are even larger in certain ranges of census tract income. Figure 2 shows a reduction in the correlation between misclassification and census tract income, particularly among Blacks. The solid red lines show the same misclassification rates as Figure 1 for the BISG model. The dotted green line in Figure 2 shows the misclassification rate across census tract income for the BISG + random forest algorithm. Among Black individuals, across all income ranges, the random forest model produces lower misclassification rates than the original BISG model. In some areas (around $80–$100 k tract median income), the misclassification rate is reduced by nearly 50%. Figures A.11 and A.12 show the same results in the withheld 20% of the Florida voter file.

Figure 2.

Misclassification Rates and Census Tract Income with Random Forest Improvements

Note: Misclassification rates and census tract income for the BISG model and the BISG + random forest algorithm in North Carolina. The solid red line shows the average misclassification rate for the BISG model and the dotted green line shows the misclassification rate for the BISG + random forest algorithm using a lowess line (span = 0.6) fit to the data. The addition of the random forest algorithm dramatically decreases misclassification rates among Black voters.

Table 2 looks at several measures of economic and political inequity across races using self-reported race, the BISG model alone, and the BISG + random forest model. Each column of the table presents a different theoretically and empirically important variable from the full North Carolina voter file: income, home value, campaign donations, voter turnout, and geographic segregation.Footnote ⁸ In many cases, the differences in these factors using self-reported race and the BISG model are quite large, especially among Black individuals. The next row in the table shows these same estimates using our proposed BISG + random forest solution. Incorporating our proposed solution of using the BISG model with the random forest correction improves estimates (brings them closer to the estimates that use self-reported race) in nearly all cases. For example, the BISG model underestimates Black median tract income by 14.95% but has a much smaller underestimate of 4.13% using the BISG + random forest model. The same is true of median home value (−10.73% using BISG alone vs.−2.74% using BISG + random forest), the share of campaign donors who are Black (28.64% using BISG alone vs. 6.00% using BISG + random forest), voter turnout (−33.9% using BISG alone vs.−5.27% using BISG + random forest), and whether an individual is a minority in their own neighborhood (i.e., lives in a census tract where their race is not the most common racial group,−43.3% using BISG alone vs.−12.18% using BISG + random forest). Bold values indicate cases where the random forest model provides improvements over the BISG model alone.

Table 2.

Differences in Summary Statistics for Self-Reported Race versus Predicted Race Using BISG and BISG + Random Forest Models in North Carolina

Note: Among Black individuals, predicted race using the BISG model leads to estimates of lower median income, home value, rates of campaign donations, and rates of living as a minority in one’s census tract. The BISG + random forest ensemble improves estimates. The “Campaign donors” column measures the estimated proportion of donors who identify with each ethnicity. The “Minority in own tract” column measures the proportion of individuals from that racial group who live in a tract in which their race is not the largest group. Bold values indicate cases where the random forest model provides improvements over the BISG model alone.

Finally, we look more deeply at how the improvements generated by the random forest model alter the substantive conclusions a person might draw regarding voter turnout compared with the conclusions they would arrive at based solely on the BISG model. Scholars have long discussed the variation in turnout by race and income (Fraga Reference Fraga2018; Wolfinger and Rosenstone Reference Wolfinger and Rosenstone1980). In fact, a key component of the U.S. Supreme Court’s decision to strike down portions of the Voting Rights Act in Shelby County v. Holder relied on estimates of turnout rates by race (Ansolabehere, Fraga, and Schaffner Reference Ansolabehere, Fraga and Schaffner2020). Figure 3 shows the estimated turnout rate among white and Black voters in North Carolina and the income of their census tract. Calculating turnout rates requires a numerator (the number of people who voted) and a denominator (the number of people who are eligible to vote). We rely on the voter file to provide the number of individuals by race who turned out to vote in the 2016 general election, as other scholarship has argued that voter files provide the most accurate measure of how many voters turned out in an election (Fraga and Holbein Reference Fraga and Holbein2020). We also follow the scholarly convention of using the U.S. Census estimates of the citizen voting age population broken down by race to gauge the number of eligible voters by racial group (Fraga Reference Fraga2018).

Figure 3.

2016 Turnout Rates in North Carolina by Race and Census Tract Income for Self-Reported Race, BISG Model, and BISG + Random Forest Algorithm

Figure 3 illustrates how voter turnout estimates vary based on income and race for the different models. Among white voters (left panel), the BISG model overpredicts turnout by approximately 6 percentage points overall, whereas the BISG + random forest model is much closer (1.1 percentage points off overall) to the true turnout rate calculated using self-reported race. When looking across income levels, the differences between white voter turnout rates using the BISG estimates versus self-reported race grow in some cases to more than 10 percentage points. Among Black voters, the problem is even more concerning, both overall and when considered across income levels. Because the BISG model misclassifies so many Black voters in high-income neighborhoods, the associated turnout rate approaches zero, leading to an incredibly large underprediction of Black turnout overall. While a researcher using this method would hopefully recognize these implausible results at high levels of census tract income, they may not be so lucky if only considering Black turnout overall, and would, therefore, underestimate Black turnout by nearly 20 percentage points. This is a direct consequence of the BISG model’s inability to make accurate predictions for high SES Black individuals. The BISG + random forest model, while missing the mark by a wide margin in the wealthiest census tracts (but not nearly to the degree as the BISG model alone), is much more accurate (3.1 percentage points off overall).