2.1 The list experiment setup

To clarify notation and to demonstrate concretely how a list experiment works, I use as a running example the list experiment conducted by Kuklinski, Cobb and Gilens (Reference Kuklinski, Cobb and Gilens1997). In this study, respondents were divided at random into control and treatment groups and those in the control group assigned the following question and list of items:

• ∙ The federal government increasing the tax on gasoline

• ∙ Professional athletes getting million-dollar contracts

• ∙ Large corporations polluting the environment

Respondents in the treatment group were presented with the same question and a list of items that included those received by the control group in addition to the sensitive item of interest:

• ∙ The federal government increasing the tax on gasoline

• ∙ Professional athletes getting million-dollar contracts

• ∙ Large corporations polluting the environment

• ∙ A black family moving in next door

The logic underlying a list experiment such as this is both simple and powerful. Suppose first that a random sample of $N$ respondents $i=1,\ldots ,N$ , is drawn from a population of interest and respondents are assigned at random into treatment and control groups, where $T_{i}\in \{0,1\}$ denotes treatment status. Those in the control group ( $T_{i}=0$ ) are then assigned a list of items. The number of items in a typical control list is 3 or 4, which I denote by the integer $J$ . In the list experiment described above, for example, the control list contains three items. Those in the treatment group ( $T_{i}=1$ ) are assigned the same list of $J$ items in addition to the sensitive item, which in the example above is “A black family moving next door.” Each respondent therefore receives $J+T_{i}$ items in total. Finally, respondents are asked how many of the items in the list would make them upset, how many they agree with, or an analogous question that is consistent with a “how many” question formulation.

The reason that list experiments are able to elicit truthful responses is because respondents are not asked about each of the $J+T_{i}$ items individually. This design property is powerful because the response to the sensitive item for any respondent in the treatment group cannot be known unless all $J+1$ items are answered affirmatively, or unless none of them are. The goal of the list experiment is therefore to ensure that survey responses cannot be deconstructed by survey enumerators and researchers per item.Footnote ² This question design property provides respondents with the opportunity to express their beliefs safe in the knowledge that researchers cannot determine their response to the sensitive item (or any other item).

To see how analysis of a list experiment proceeds, let $Y_{i}\in \{0,\ldots ,J+T_{i}\}$ denote a respondent’s answer to the list experiment question. This variable can be decomposed into the sum of two latent variables as follows:

(1) $$\begin{eqnarray}\displaystyle Y_{i}=\left\{\begin{array}{@{}ll@{}}Y_{i}^{\ast }\quad & \text{if }T_{i}=0,\\ Y_{i}^{\ast }+Z_{i}^{\ast }\quad & \text{if }T_{i}=1,\end{array}\right. & & \displaystyle\end{eqnarray}$$

where $Y_{i}^{\ast }\in \{0,\ldots ,J\}$ is the latent response to the $J$ control items, and $Z_{i}^{\ast }\in \{0,1\}$ is the latent response to the sensitive item. As is clear from Equation (1), responses to the list experiment differ depending on treatment status only by the response to the sensitive item. Imai (Reference Imai2011) demonstrates that, given a set of assumptions,Footnote ³ the difference in the mean of $Y$ for the treatment and control groups provides an unbiased estimate of the proportion of the population that would truthfully answer affirmatively to the sensitive item.

For much research in the social sciences, it will also frequently be of interest to model the predictors of responding affirmatively to the sensitive item. Imai (Reference Imai2011) therefore develops and implements a maximum likelihood estimator to enable the sensitive item in a list experiment to be modeled within a multivariate regression context. The method that I propose below builds on the framework introduced by Imai (Reference Imai2011), which I detail further in the online appendix (see also Blair and Imai Reference Blair and Imai2012).

2.2 The direct question and misreporting

To identify the response pattern consistent with misreporting, I begin by introducing an assumption that I use in conjunction with those for the standard list experiment model (Imai Reference Imai2011, 408–409). To demonstrate this assumption concretely, I return to the list experiment conducted by Kuklinski et al. (Reference Kuklinski, Cobb and Gilens1997) for illustration. Respondents in this study were not asked the sensitive question directly, but for the sake of concreteness we can assume that it would have been asked as follows:

Let $D_{i}\in \{0,1\}$ denote an individual’s response to the direct question, with response categories “No” ( $D_{i}=0$ ) and “Yes” ( $D_{i}=1$ ). To denote “misreporting,” let $U_{i}^{\ast }\in \{0,1\}$ be an indicator variable representing whether the response to the direct question is different from that given to the sensitive item in the list experiment: $U_{i}^{\ast }=\mathbf{1}(Z_{i}^{\ast }\neq D_{i})$ , such that an individual is said to be misreporting if $U_{i}^{\ast }=1$ .

Monotonicity assumption. The monotonicity assumption states that individuals who hold the socially undesirable belief may or may not misreport it when asked to express that belief openly. Those who do not hold the socially undesirable belief are assumed to state so truthfully.Footnote ⁴ This assumption follows naturally from our general understanding of social desirability bias, such that those who do not hold a socially undesirable belief are not expected to feel social pressure to misreport it by answering in a socially undesirable manner. More formally, this assumption can be stated as follows:

(2) $$\begin{eqnarray}\displaystyle U_{i}^{\ast }=\left\{\begin{array}{@{}ll@{}}0\quad & \text{if }Z_{i}^{\ast }=0,\\ 0\text{ or }1\quad & \text{if }Z_{i}^{\ast }=1.\end{array}\right. & & \displaystyle\end{eqnarray}$$

In our running example, this assumption means that only those who would be upset if an African–American family were to move next door may or may not claim otherwise when asked to reveal that attitude openly. Respondents who would not be upset, on the other hand, are assumed to respond truthfully. Put differently, if this were not the case, it would mean that some respondents who are not prejudiced would misreport by claiming to be prejudiced nonetheless.Footnote ⁵

2.3 Respondent types

Given the monotonicity assumption, we can identify three response patterns to the list experiment and direct question ( $Z_{i}^{\ast }$ , $D_{i}$ ). Substantively, these define the following respondent types: (1) those who hold the sensitive belief and misreport it when asked directly ( $Z_{i}^{\ast }=1$ , $D_{i}=0$ ); (2) those who hold the sensitive belief and do not misreport it ( $Z_{i}^{\ast }=1$ , $D_{i}=1$ ); and (3) those who do not hold the sensitive belief and do not misreport it ( $Z_{i}^{\ast }=0$ , $D_{i}=0$ ). These response patterns and their respective descriptions are presented in Table 1. By contrast, the standard list experiment model (Imai Reference Imai2011) identifies two respondent types: those who hold the socially unacceptable belief and those who do not. The method introduced here therefore differs in that it identifies two sub-types of respondents among those who provide the socially unacceptable response: those who admit openly to holding the socially unacceptable belief, and those who misreport it.

Table 1.

Respondent types defined by the response ( $Z_{i}^{\ast }$ , $D_{i}$ ).

Note that the respondent type for the response ( $Z_{i}^{\ast }=0$ , $D_{i}=1$ ) is undefined by the monotonicity assumption.

2.4 Maximum likelihood estimator

To model these responses within a regression-based framework, I define three sub-models to model the joint distribution ( $Y_{i}^{\ast }$ , $Z_{i}^{\ast }$ , $U_{i}^{\ast }$ ). First, the sub-model used to model the response to the sensitive item, $Z_{i}^{\ast }$ , is defined as follows:

(3) $$\begin{eqnarray}\displaystyle g(x;\unicode[STIX]{x1D6FF})=\text{Pr}(Z_{i}^{\ast }=1|X_{i}=x;\unicode[STIX]{x1D6FF}), & & \displaystyle\end{eqnarray}$$

where $X_{i}$ is a vector of covariates and $\unicode[STIX]{x1D6FF}$ is a vector of parameters to be estimated.

The second sub-model models the probability that a respondent’s answer to the sensitive item in the list experiment question is different from that given to the direct question:

(4) $$\begin{eqnarray}\displaystyle j(x,z,t;\unicode[STIX]{x1D6FE})=\text{Pr}(U_{i}^{\ast }=1|X_{i}=x,Z_{i}^{\ast }=z,T_{i}=t;\unicode[STIX]{x1D6FE}), & & \displaystyle\end{eqnarray}$$

where $\unicode[STIX]{x1D6FE}$ is a vector of parameters to be estimated. Note that, by the monotonicity assumption, $j(x,0,t;\unicode[STIX]{x1D6FE})=0$ . In other words, the probability that a respondent misreports if that respondent does not hold the socially unacceptable belief is 0 by assumption.

In the setup described here, survey respondents are presumed to answer the direct question some time after providing a response to the list experiment question. Although researchers may also ask the direct question before the list experiment, it is recommended that the list experiment question be asked first. Doing so permits researchers both to straightforwardly test whether receiving the treatment list affects the response to the direct question, and if such a relationship exists, to account for this by modeling it in the misreport sub-model (Equation (4)).Footnote ⁶ Theoretically, such a relationship might exist because those in the treatment group recall their previous response to the sensitive item in the list experiment and provide the same response to the direct question for reasons of cognitive ease or to be consistent on principle.Footnote ⁷ To avoid reliance on modeling this relationship, it is recommended that researchers aim to minimize or eliminate this potential effect through conscientious survey design. It is suggested, therefore, that the list experiment and direct questions be separated from one another in the survey by other sets of unrelated questions when possible.

Lastly, the third sub-model models the response to the control items, $Y^{\ast }$ , as follows:

(5) $$\begin{eqnarray}\displaystyle h(y|x,z,u;\unicode[STIX]{x1D713})=\text{Pr}(Y_{i}^{\ast }=y|X_{i}=x,Z_{i}^{\ast }=z,U_{i}^{\ast }=u;\unicode[STIX]{x1D713}), & & \displaystyle\end{eqnarray}$$

where $\unicode[STIX]{x1D713}$ is a vector of parameters to be estimated. For implementation, binomial regression models with a logistic link are used for each of these sub-models.

Given the models $g(x;\unicode[STIX]{x1D6FF})$ , $j(x,z,t;\unicode[STIX]{x1D6FE})$ , and $h(y|x,z,u;\unicode[STIX]{x1D713})$ , I now derive the observed-data likelihood. Due to the likelihood function’s length, the components of the likelihood for each response pattern are provided in Table 2. The observed-data model likelihood is the product of the relevant individual likelihoods as given in the last column of the table.

Table 2.

Observed-data likelihood.

Table 3.

Weights, $w_{i}^{(\cdot )}$ , as calculated in the E-step.

2.5 Model optimization

For optimization, I implement an expectation-maximization (EM) algorithm (Dempster, Laird and Rubin Reference Dempster, Laird and Rubin1977). The EM algorithm requires the complete-data likelihood function, which is the likelihood function for the case in which the latent variables $Z_{i}^{\ast }$ , $Y_{i}^{\ast }$ , and $U_{i}^{\ast }$ are fully observed. This function is given as follows:

(6) $$\begin{eqnarray}\displaystyle & & \displaystyle L(\unicode[STIX]{x1D6FF},\unicode[STIX]{x1D6FE},\unicode[STIX]{x1D713};\{T_{i},Y_{i},X_{i},Z_{i}^{\ast },U_{i}^{\ast }\}_{i=1}^{n})\nonumber\\ \displaystyle & & \displaystyle \quad =\mathop{\prod }_{i=1}^{n}\{\{1-g(X_{i};\unicode[STIX]{x1D6FF})\}\{1-j(X_{i},0,T_{i};\unicode[STIX]{x1D6FE})\}h(Y_{i}|X_{i},0,0;\unicode[STIX]{x1D713})\}^{\mathbf{1}(Z_{i}^{\ast }=0\wedge U_{i}^{\ast }=0)}\nonumber\\ \displaystyle & & \displaystyle \qquad \times \,\{g(X_{i};\unicode[STIX]{x1D6FF})\{1-j(X_{i},1,T_{i};\unicode[STIX]{x1D6FE})\}h(Y_{i}-T_{i}|X_{i},1,0;\unicode[STIX]{x1D713})\}^{\mathbf{1}(Z_{i}^{\ast }=1\wedge U_{i}^{\ast }=0)}\nonumber\\ \displaystyle & & \displaystyle \qquad \times \,\{g(X_{i};\unicode[STIX]{x1D6FF})j(X_{i},1,T_{i};\unicode[STIX]{x1D6FE})h(Y_{i}-T_{i}|X_{i},1,1;\unicode[STIX]{x1D713})\}^{\mathbf{1}(Z_{i}^{\ast }=1\wedge U_{i}^{\ast }=1)}.\end{eqnarray}$$

The EM algorithm is iterative, consisting of two steps. The expectation step calculates the conditional expectation of the response ( $Z_{i}^{\ast }$ , $D_{i}$ ), given the model parameters $\unicode[STIX]{x1D6FF}$ , $\unicode[STIX]{x1D6FE}$ , and $\unicode[STIX]{x1D713}$ from the most recent iteration of the maximization step. The weights calculated from this step represent the posterior predicted probabilities that respondents are each of the three types as presented in Table 1. The expressions used to calculate these weights are given in Table 3.

The maximization step then computes the parameters $\unicode[STIX]{x1D6FF}$ , $\unicode[STIX]{x1D6FE}$ , and $\unicode[STIX]{x1D713}$ using the most recent values of the weights $w_{i}^{(\cdot )}$ from the E-step that maximize the complete-data log-likelihood, which is given as follows:

(7) $$\begin{eqnarray}\displaystyle & & \displaystyle Q(\unicode[STIX]{x1D6FF},\unicode[STIX]{x1D6FE},\unicode[STIX]{x1D713};\{Y_{i},X_{i},T_{i},w_{i}^{(\cdot )}\}_{i=1}^{n})\nonumber\\ \displaystyle & & \displaystyle \quad =\mathop{\sum }_{i=1}^{n}w_{i}^{\text{(nonsensitive)}}\{\log \{1-g(X_{i};\unicode[STIX]{x1D6FF})\}+\log \{1-j(X_{i},0,T_{i};\unicode[STIX]{x1D6FE})\}+\log h(Y_{i}|X_{i},0,0;\unicode[STIX]{x1D713})\}\nonumber\\ \displaystyle & & \displaystyle \qquad +\,w_{i}^{\text{(truthful sensitive)}}\{\log g(X_{i};\unicode[STIX]{x1D6FF})+\log \{1-j(X_{i},1,T_{i};\unicode[STIX]{x1D6FE})\}+\log h(Y_{i}-T_{i}|X_{i},1,0;\unicode[STIX]{x1D713})\}\nonumber\\ \displaystyle & & \displaystyle \qquad +\,w_{i}^{\text{(misreport sensitive)}}\{\log g(X_{i};\unicode[STIX]{x1D6FF})+\log j(X_{i},1,T_{i};\unicode[STIX]{x1D6FE})+\log h(Y_{i}-T_{i}|X_{i},1,1;\unicode[STIX]{x1D713})\}.\end{eqnarray}$$

The EM algorithm iterates between the E-step and M-step until convergence. Because the algorithm is not guaranteed to converge to a global maximum and can depend on the initialization of the starting parameters, the algorithm can be run multiple times using randomized starting values and the parameters from the run that results in the highest log-likelihood used as the model solution. Following optimization, standard errors are calculated through numerical approximation.

3.1 Simulation study 1

In this study, I demonstrate through simulation how researchers can increase efficiency in the sensitive-item sub-model by incorporating information from responses to a direct question. This is an important secondary benefit to the proposed method because it lowers the bias-variance tradeoff inherent in using a list experiment in place of a direct question: using both a list experiment and direct question simultaneously provides the benefits of lower bias without sacrificing the same degree of efficiency as using a list experiment alone.Footnote ⁹

To conduct the study, I simulate data from a list experiment containing four control items, with the intercept and sensitive-item parameters in the control-items sub-model set to 0 (on the logistic scale). Responses to a direct question are simulated from an intercept-only misreport sub-model where the population proportion of those who misreport is varied from 0.1 to 0.9 at 0.1 intervals. With this setup, 10,000 datasets are simulated, each containing 3,000 respondents. Models for the proposed and standard estimators are then fit to each dataset.

Figure 1.

Simulation study 1 results. These figures present a comparison by root mean squared error (RMSE), bias, and confidence interval coverage of the standard list experiment estimator (dashed line) (Imai Reference Imai2011) to the proposed estimator (solid line) in populations containing different proportions of those who misreport.

Results from these models are presented in Figure 1. As shown in the first panel, the proposed estimator substantially improves on the standard estimator in terms of RMSE across the range of populations containing different proportions of those who misreport. These gains in efficiency result from the additional information provided by the direct question response: respondents who openly admit to holding the socially unacceptable belief through the direct question are, by the monotonicity assumption, assumed to provide the same response to the sensitive item in the list experiment, reducing estimation uncertainty. In the second and third panels of Figure 1, both the standard and proposed estimators appear unbiased, with confidence interval coverage close to the nominal rate.

3.2 Simulation study 2

In this study, I examine the properties of the proposed and standard methods in the more typical scenario in which a researcher wishes to model responses to the list experiment and direct question with covariates. To do so, I simulate two variables for use as model covariates: the first covariate is a binary variable drawn from a binomial distribution with probability 0.5; the second, a continuous variable drawn from a univariate normal distribution with mean and variance 1. The parameters for the intercept and the continuous and binary covariates are set respectively to $(-0.25,0.25,-0.5)$ for the control-items sub-model; $(0,0.25,-0.5)$ for the sensitive-item sub-model; and $(0.5,-0.5,0.25)$ for the misreport sub-model. The parameter for the sensitive item in the control-items sub-model ( $Z^{\ast }$ ) is set to 0.5 and that for whether someone is misreporting ( $U^{\ast }$ ) is set to 0 to permit estimator comparison. Given the parameters as defined above, 10,000 datasets are simulated with covariates drawn from the appropriate distributions for samples of size 2,000, 3,000, and 5,000. The proposed and standard estimators are then fit to each simulated dataset.

Figure 2 presents results from this procedure for the sensitive-item sub-model. Figures for the control-items and misreport sub-models are presented in the online appendix. As Figure 2 shows, in line with results from the first simulation study, RMSE is lower for the proposed estimator compared to the standard estimator for all parameters. Parameter estimates appear unbiased and confidence interval coverage is close to the nominal rate, improving as expected with sample size. As this study shows, researchers who are only interested in the sensitive-item sub-model can nevertheless benefit from the incorporation of information from a direct question in the form of lower error in parameter estimates.

Figure 2.

Simulation study 2 results. These figures show a comparison of the standard estimator (black) (Imai Reference Imai2011) and proposed estimator (white) by root mean squared error (RMSE), bias, and confidence interval coverage for the sensitive-item sub-model of a list experiment with covariates. Results for the control-items and misreport sub-models are available in the online appendix.

3.3 Simulation study 3

In the final simulation study, I show how error in parameter estimates in the misreport sub-model varies as a function of both the proportion of the population that hold the sensitive belief and the proportion that misreport it. Both of these properties will be important considerations for applied researchers. First, as the simulations will show, investigation into misreporting requires substantial sample sizes when very few members of a population hold the socially unacceptable belief. This occurs because the misreport sub-model is used to distinguish between those who misreport and those who tell the truth within the sub-population that holds the sensitive belief: when very few respondents hold the sensitive belief, there will be little data from which to precisely estimate model parameters. Second, the simulations show that when very few members of a population misreport, estimation error in the misreport sub-model will be higher compared to populations containing higher rates of misreporting. To demonstrate these properties, I run two complementary simulation studies, varying the proportions of the population that hold the sensitive belief in the first, and the proportion of those who misreport in the second.

To conduct these studies, I simulate 10,000 datasets with sample sizes of 2,000, 3,000, and 5,000 respondents. For the first part of the study, the proportion of the population that hold the sensitive belief is varied from 0.1 through 0.9 at 0.1 intervals, and the proportion of the population that misreport is set to 0.5. For the second part of the study, the proportion of the population that misreport is set to 0.1 through 0.9 at 0.1 intervals, and the proportion of those who hold the sensitive belief is set to 0.5. For both studies, parameters for the control-items sub-model are set to 0 on the logistic scale for both the intercept and $Z^{\ast }$ . In the first study, therefore, interest is in error as it varies with the proportion of the population that hold the sensitive belief; in the second, interest is in error as it varies with the proportion of population that misreport.

Figure 3.

Simulation study 3 results. These figures show root mean squared error (RMSE), bias, and confidence interval coverage for two Monte Carlo simulation studies that examine the properties of the misreport sub-model. Panel A presents results for the misreport sub-model from a simulation study in which the proportion of those holding the sensitive belief is varied from 0.1 to 0.9 at 0.1 intervals; Panel B presents results from the misreport sub-model in which the proportion of those who misreport is varied from 0.1 to 0.9 at 0.1 intervals.

Results from the first and second parts of the simulation study are presented in panels A and B of Figure 3. In the first panel (A), RMSE in estimates of the population proportion of those who misreport is shown to decrease as the proportion of those in the population who hold the sensitive belief increases. Similarly, in the second panel (B), RMSE is decreasing in the proportion of those who misreport. In both cases, therefore, error in parameter estimates are shown to be larger when either very few members of a population hold the sensitive belief, or when very few members of a population misreport it. In each case, estimates appear slightly biased when very few members of a population hold the sensitive belief or misreport it, but appear unbiased elsewhere, with confidence interval coverage close to the nominal rate. The upshot is that when the misreport sub-model is of interest to researchers, care should be taken in selecting a sensitive item during the survey design stage: when researchers expect that very few respondents will hold the socially unacceptable belief of interest or few will misreport it, larger sample sizes will be required to estimate parameters in the misreport sub-model with precision.

4.1 Does the left misreport their prejudices more than those on the right?

Since the 1990s, the term “political correctness” has frequently been used to describe adherence to the set of social norms that prohibit the expression of prejudiced beliefs and the use of stigmatizing language (Hughes Reference Hughes2010, 46). These social norms are now relatively pervasive across the Western world and apply to a wide variety of beliefs regarding gender, race, ethnicity, religion, and sexuality. As a consequence, those who hold beliefs that society deems prejudiced are, in effect, expected either to misrepresent them or to refrain from voicing them altogether.

The strength of the social norms governing expressions of real or perceived prejudiced beliefs appear, however, to vary strongly along ideological lines. This has been highlighted recently by a variety of public debates and commentaries regarding the validity and effects of these norms. For example, political and social commentators have questioned whether those on the left are unduly apprehensive in their criticisms of illiberal practices by some new immigrants and refugees (The Telegraph 2016); whether university students are being “coddled” by the use of sanitized language and the suppression of sensitive topics (Lukianoff and Haidt Reference Lukianoff and Haidt2015); whether politically correct social and political norms are “perverting” the values of liberalism (Chait Reference Chait2015); whether liberal students are overly sensitive to controversial subjects (Schlosser Reference Schlosser2015; Taub Reference Taub2015); whether anti-racism protesters should be sheltered from media attention and criticism (Starr Reference Starr2015; Friedersdorf Reference Friedersdorf2015); and whether political appointments that achieve equality in gender representation are being made for the sake of appearances rather than merit (Coyne Reference Coyne2015). Opinions on these issues, furthermore, have recently been expressed by Western political leaders (e.g. Obama Reference Obama2015; Cameron Reference Cameron2015; Cross Reference Cross2015).Footnote ¹¹ As prominent commentators on the issue have argued, the episodes that spark these debates have “created a culture in which everyone must think twice before speaking up, lest they face charges of insensitivity, aggression, or worse” (Lukianoff and Haidt Reference Lukianoff and Haidt2015). As another commentator writes, difficulty in determining what is or is not insensitive language or behavior has led to the elaboration of complicated sets of norms and terminology within communities on the political left (Chait Reference Chait2015).

To investigate whether the strength of these norms varies along ideological lines, I apply the proposed methodology to a large-scale list experiment and direct question regarding beliefs about whether women and men are equally competent in politics. Substantively, this question is likely to be highly sensitive, in particular due to the underrepresentation of women in political office worldwide (Inter-parliamentary Union 2015) and discrimination against women in, among other areas, encouragement to serve in politics, political socialization, and elected and politically appointed office (Fox and Lawless Reference Fox and Lawless2014). Empirically, recent studies suggest that women who run for office are in fact more likely to be competent than men (Lawless and Pearson Reference Lawless and Pearson2008) and that those who are elected to office are as or more effective lawmakers than their male counterparts (Jeydel and Taylor Reference Jeydel and Taylor2003; Volden and Wiseman Reference Volden and Wiseman2011; Anzia and Berry Reference Anzia and Berry2011).Footnote ¹² However, setting aside the empirical question of whether gender differences in competence indeed exist, we can nevertheless presume that those who disagree that women are as competent in politics as men will feel strong pressure to claim otherwise. In the following, I test whether this pressure—as measured by whether respondents misreport their beliefs concerning gender equality—varies along ideological lines.

4.2 Data and survey design

To test the hypothesis that those on the left are more likely than those on the right to misreport their beliefs about women’s and men’s competence in politics, I use survey data from a list experiment and direct question that were embedded in a Canadian postelection study fielded after the 2015 federal election. The study was fielded by the research firm Vox Pop Labs to all of the members of its national online panel, resulting in an exceptionally large sample size (n  $=$  24,020). Because the survey was sent to all members of the research firm’s panel, the sample was not prestratified to match population characteristics. To address differences in sample and population characteristics, survey weights from the data are used to poststratify estimates.Footnote ¹³

To gauge attitudes toward gender equality in politics, respondents were asked whether they agreed or disagreed with the statement “women are as competent as men in politics.” Respondents were first asked to provide an answer regarding this statement through a list experiment question and later in the survey asked to provide an answer openly in response to a direct question.

The list experiment question included four control items and was asked as follows:

• ∙ There should be more funding for the arts

• ∙ The government should raise taxes on gasoline

• ∙ Corporations are taxed too much

• ∙ Unions have too much power

Respondents in the treatment group received the same question and a list that included both the control items and the sensitive item:

• ∙ There should be more funding for the arts

• ∙ The government should raise taxes on gasoline

• ∙ Corporations are taxed too much

• ∙ Women are as competent as men in politics

• ∙ Unions have too much power

The direct question was asked later in the survey as follows:

• ∙ Women are as competent as men in politics

The list experiment and direct question were separated by a large set of unrelated questions to obviate potential response bias, whereby the direct question response is not independent of treatment status.

Because list experiments are cognitively taxing, a question was included in the survey to filter out inattentive respondents. Rather than screen for inattentiveness in the survey as a whole (Berinsky et al. Reference Berinsky, Margolis and Sances2014), a question was included to address this concern for the list experiment directly. The screener question was displayed immediately after the list experiment question and asked respondents to identify, among a new list, the topic of one of the control items from the previous page (see online appendix for complete question text). Seven percent of respondents either did not answer this question correctly or answered “don’t know.” These respondents were dropped from the dataset analyzed in the results section.

Before analyzing the results, a series of tests were run on the data to check for violations of the list experiment’s design assumptions. These tests do not find strong evidence of any violations, the results of which are detailed in the online appendix.