STUDY 1: BIASES DURING THE PRIMARIES

There are several ways to search for social desirability bias and we use two different experimental methods in each of our two studies. The method in our first study has been used to detect social desirability with regards to race (Holbrook, Green, and Krosnick Reference Holbrook, Green and Krosnick2003) and partisan identification (Klar and Krupnikov Reference Klar and Krupnikov2016). Instead of asking respondents to provide their true responses, respondents are randomly assigned to either answer as if their goal was to make the best impression on others or to make the worst impression on others. If more people give a particular response in the Best Impression condition, then that response is judged to be socially desirable. If more people give a particular response in the Worst Impression condition, then the response is socially undesirable. If an equal percentage of people give a response in both conditions, then there is no social desirability bias associated with that response category.

In January 2016, a national sample of 810 respondents participated in this first study via Survey Sampling International (SSI)—a panel-based Internet survey company whose samples are commonly used in political science research (e.g., Kam Reference Kam2012; Krupnikov, Piston, and Bauer Reference Krupnikov, Piston and Bauer2015). Respondents were randomly assigned to either say who they would support in the Democratic primaries if they wanted to make the best impression or who they would support if they want to make the worst impression. Their choices were the three candidates seeking the nomination at the time: Hillary Clinton, Bernie Sanders, and Martin O’Malley.

The left side of figure 1 presents the results for all respondents, which gives a sense of the larger social desirability among the public. The right side displays how Democratic respondents in particular view the social desirability of voting for each candidate. As the figure shows, there is no difference in the percentage of people who say Clinton is the answer that makes the best impression and the people who say Clinton is the answer that makes the worst impression. Instead, the figure suggests participants may feel more social desirability pressures when it comes to stating that one is a Sanders supporter—especially among Democrats.

Figure 1

Impressions of Vote Choices in the Democratic Primaries

Error bars are for 95% confidence intervals.

Note that in both treatments a majority of Democrats respond, “Hillary Clinton.” As a result, there is no generalized social desirability bias among all Democrats because anyone who states they support Clinton only to avoid appearing sexist will be matched with someone who says they do not support Clinton for another reason related to socially desirability.

In addition to the simple analysis in figure 1, we attempted to see if the effect among Democrats was conditional on demographics that received a lot of attention during the primary: gender, race, and age. We found no differences in the level of social desirability for a Clinton candidacy between men and women or whites and minorities. However, among the youngest respondents—those under 25—53% said supporting Clinton gives the worst impression while only 27% said supporting Clinton gives the best impression (p<.05). This result goes against the conventional wisdom that respondents will hide negative gender attitudes by falsely stating they support a woman candidate. If there is social desirability bias in polls, this suggests polls would underestimate Clinton support among young people.

STUDY 2: BIASES IN THE GENERAL ELECTION

Our next measure relies on a technique initially implemented by the Obama campaign in 2008 to uncover voters who tell a pollster they support Obama, but in the privacy of the voting booth select McCain (Issenberg Reference Issenberg2012). After trying numerous alternatives, the campaign identified a question about whether one’s neighbors would be willing to vote for a well-qualified black candidate for president. This question worked well identifying people “who had high Obama-support scores but ended up backing McCain.” (Issenberg Reference Issenberg2012, 296). Issenberg writes, “Something in that response—perhaps a feeling of being liberated to publicly share an unpopular opinion—convinced [the campaign] that the people who acknowledged their neighbors’ racism might really be confessing a view of their own” (Issenberg Reference Issenberg2012, 296).

The “would your neighbor be willing to support this candidate” question that the Obama campaign used, was actually a question Gallup had fielded to measure implicit sexism in voting three years earlier. Only 1 in 10 survey respondents said they were unwilling to support a woman for president, while about 1 in 3 said their neighbors would be unwilling to do so (Feldmann Reference Feldmann2007). In fact, although Streb et al. (Reference Streb, Burrell, Frederick and Genovese2008) ultimately used a “list experiment,” they cite the Gallup version of the neighbor question as evidence of social desirability bias.

In the third week of April, 2016, as the nomination contests neared a conclusion, a new set of SSI respondents (N=964) was interviewed to see if the method the Obama campaign used to detect negative racial attitudes could uncover negative gender attitudes. We randomly assigned subjects to answer either the standard question about willingness to support women candidates or the neighbor version. We also randomly assigned subjects to both versions of a question asking about plans to vote for Hillary Clinton, Donald Trump, or not vote.

By having measures that could capture both gender bias in general and regarding a specific candidate, we can estimate with some precision how many, if any, percentage points Clinton’s sex is likely to cost her. This also will enable us to explore whether the social desirability problems Streb et al. (Reference Streb, Burrell, Frederick and Genovese2008) document regarding a hypothetical female candidate also affect voting plans towards the first actual female major party candidate. If there are hidden biases, we hope to uncover them using the “neighbor” question in an experiment.

The two versions of the “willingness to support a female candidate” question are below (neighbor version in brackets). Both questions had the same response options.

Figure 2 presents the basic results of these treatments. Figure 2A reveals that the different versions yield significantly different distributions with regards to the willingness to support a woman for president (χ²=82.93; 3 d.f.; p<.001). The difference is mostly movement from “doesn’t matter” to the “don’t know” category, but nevertheless the result is a distribution that is less congenial toward women candidates. So worries that social desirability bias may be causing underestimates of opposition to women candidates for president receive some validation from our experiment.

Figure 2

Effects of “Neighbor” Question Treatment on Willingness to Support a Female Candidate and Presidential Vote Intention

A. “Willingness” to Support Female Candidate

B. Presidential Vote Intention

Error bars are for 95% confidence intervals.

We hasten to add, however, that while the less congenial distribution could be problematic for Clinton, it may not necessarily translate into fewer votes. For example, if “don’t know” responses are no more predictive of voting plans than “doesn’t matter” responses, then the distribution differences wouldn’t matter much when it comes to vote intentions. We will investigate this possibility later in this article.

Before that, we look directly at the matchup between Clinton and Trump. Figure 2B graphs the distributional differences across the two versions—standard and neighbor—of the vote intention question. We can reject the null hypothesis that the distribution was statistically independent of question version (χ²= 8.73; 2 d.f.; p<.05). The candidates’ marginals are not dramatically different, and Clinton holds a 20 percentage point lead in both treatments. Respondents, however, are about 7% less likely to say their neighbors “won’t vote” compared to their reports about their own plans. In fact, the “won’t vote” difference occurs almost entirely among Republicans. With the neighbor version, more Republicans report voting for Trump and Clinton. This suggests that some Republicans felt it was socially undesirable to admit defecting or voting for their own candidate in this election. We find no evidence in the vote intention questions, however, that social desirability will lead pollsters to overestimate support for the Clinton candidacy.

Our final analysis estimates the effect of willingness to support women candidates for president, in general, on 2016 vote intentions in order to assess the potential penalties and rewards Clinton faces for being a woman. Because of our 2x2 experimental design, we have roughly one quarter of our sample in each of the two question version combinations. By looking at rewards and penalties in each combination we compute a range of possibilities that varies depending on the question version and the estimated effects. Ultimately, this analysis allows us to see if the Clinton campaign should worry about the large number of subjects who report they “don’t know” if their neighbors are willing to support a female candidate.

We examine rewards and penalties for Hillary Clinton using a logit model of vote intention among subjects who stated they would vote for either her (coded 1) or Donald Trump (coded 0). The key independent variables are: (1) a dummy variable capturing whether the subject received the neighbor vote choice question or the standard version; (2) a dummy variable capturing whether the subject received the neighbor “willingness” question or the standard version; (3) a series of dummy variables measuring how the subject answered the “willingness” question with Doesn’t Matter as the reference category; (4) interactions between the “willingness” treatment and the “willingness” answers. The model also controls for party, age, education, race, and respondent gender.

The coefficient estimates of the model are shown in table 1. The statistically significant “less likely” and “more likely” dummy variable coefficients reveal the existence of both positive and negative biases (relative to those that say “doesn’t matter”). The null interaction coefficients suggest similar effects regardless of question version—though we note the “don’t know” interaction is the most robust and, because it is nearly equal and opposite the “don’t know” dummy variable coefficient, it indicates the large number of “don’t know” respondents in the neighbor version are indistinguishable from “doesn’t matter” respondents and unlikely to penalize Clinton for being a woman.

Table 1

Logit Model of Support for Clinton by Question Treatment

Dependent variable coded 1 if subject supports Clinton and 0 if subject supports Trump. Subjects who state they will not vote are dropped from this analysis.

Turning to the substance of these effects, first we use the standard questions as a baseline to see if the various neighbor question combinations uncover any hidden biases. In all parts of table 2, the four columns show the values for each of the four question treatment combinations. Part A of table 2 shows the predicted probability of declaring an intent to vote for Clinton by the subject’s answer to the “willingness” question based on the model reported in table 1. Part B displays the distribution of willingness responses within each of the four question treatment combinations. Part C of the table multiplies part A by part B of the table to obtain the proportion of people who are both in that “willingness” category and answer the vote intention question with “Hillary Clinton.” The sums of the columns in the final part of the table are the estimated vote for Clinton in each question treatment combination.

Table 2

Calculating Support for Clinton in Each Treatment Combination

Estimates based on table 1’s model and computed using the “margins” command in STATA.

The first column, which displays the result if the respondent is asked the standard version of both questions, is a baseline of support for Clinton that one would see in most polls. This baseline support includes the penalty she suffers because some percentage of people does not want a woman president. The other columns look for the hidden biases. Here, we see that Clinton can lose up to 5 percentage points off her lead, but she still maintains a large lead in all treatment combinations. In the case where the subjects receive only the neighbor version of the vote intent question, she suffers no penalty at all. These results suggest there could be some hidden gender bias against a woman seeking the presidency, but, in this particular matchup, it is electorally inconsequential.

In figure 3, we also examine how each category of “willingness” affects Clinton’s vote compared to the baseline of “doesn’t matter” within each treatment. This allows us to see the penalties and rewards Clinton suffers and gains based on the distribution of subjects who give an answer other than “doesn’t matter.” To get the values in figure 3, we subtract the probability of voting for Clinton in each category from the probability in “doesn’t matter” (see table 2 part A) then multiply the difference by the proportion of respondents in that category (see table 2 part B).

Figure 3

Rewards and Penalties for Clinton by willingness to Support Answer within Each Treatment Condition

Estimates based on Table 1’s model. Values greater than 0 indicate Clinton is rewarded for her gender; values less than 0 indicate Clinton is punished for her gender.

Not surprisingly, the largest differences in probabilities from the “doesn’t matter” category are in the “less likely” category. Yet, as figure 3 shows, the penalties Clinton suffers are largely canceled out by the rewards she receives from subjects in the “more likely” category. Interestingly, while the substantive effect is smaller among “more likely” voters—because most subjects for whom it does not matter support Clinton—there are more people in the “more likely” category than the “less likely” category, which bolsters the reward. Again, because rewards and penalties tend to balance one another in figure 3, it indicates polls will not overestimate Clinton’s support dramatically and that biases are unlikely to undermine polling leads (assuming they are maintained).