Conjoint experiment

Consistent with our interest in how individuals think about local public policy problem solving and with recent trends in the drivers of local economic performance, the conjoint experiment is framed in terms of how the MSA should respond to globalization and technological change and implement policies that will generate economic growth and good jobs. The introduction to the conjoint experiment emphasized that increased spending would require tax increases or spending reductions in other areas. Appendix C contains the specific wording of the conjoint introduction.

Respondents were presented with pairs of hypothetical plans for local development. Each plan was composed of six attributes corresponding to six critical areas of local development policymaking: Investment and Taxes, Workers and Entrepreneurs, Local Services, Governance, Education, and Higher Education. For each issue area, a possible value was randomly drawn from an underlying set of potential values that included alternative reform or spending priorities as well as the status quo in that policy area. Table 1 lists each possible value for each dimension.

Table 1.

Conjoint dimensions and attribute values for local development plans

This table reports the attribute values for each dimension of the conjoint experiment.

Respondents were presented with randomly generated pairs of potential policies for their MSA to adopt as a local development plan and were asked to choose which plan they would prefer to see implemented.Footnote ⁶ Using this style of forced-choice design, we are able to evaluate the direction and relative weight individuals place on each dimension of local development. Respondents were presented with five sets of local development plan pairs. For our analysis, we constructed a binary measure Local Development Plan Support that equals one if a respondent selected a particular policy proposal as their preferred choice, and zero otherwise.

We estimate an ordinary least squares regression of Local Development Plan Support on dichotomous indicator variables for all treatment categories, with the exception of the baseline for each conjoint dimension.Footnote ⁷ For the sake of consistency, we take the value that expresses the status quo as our baseline for each dimension. This estimation yields the average marginal component-specific effect (AMCE) for each treatment group relative to the baseline.Footnote ⁸ Standard errors are clustered at the respondent level.

Intuitively, the coefficients give the average change in probability of selecting a development plan with the specific feature over a development plan that contains the status quo policy in that issue domain. Positive coefficients thus indicate that a given feature makes a plan more popular, relative to the status quo.

One of the main advantages of conjoint survey experiments is the ability to investigate multidimensional phenomena in an efficient way. This feature makes this approach attractive for studying polarization in policy opinions over local policies, where there may be different patterns of heterogeneity across different policies. Even if Democrats and Republicans have different preferences over the size of government or differential willingness to pay for public goods, we expect the magnitude of those differences to vary across policies. The conjoint allows us to study many policies without focusing arbitrarily on just a few aspects of economic development policy. A related advantage of conjoints for studying local politics is that because there are fewer representative studies of public opinion about local issues, it is less well-known what tradeoffs individuals will make between issues, or to put it differently, which issues they weigh more heavily when faced with tradeoffs.

Experimental conjoint estimates

Because our primary goal in this paper is assessing the extent to which attitudes toward development vary by party identification, we present estimates that pool across all respondents in Appendix Figure A-4. We find that on average, citizens support active policies to support businesses in their communities. They are also supportive of greater investments in human capital. While there is some variation across MSAs, the general pattern of estimates is quite similar across communities.

Partisanship and local development

Next, we examine the extent of sorting and partisan polarization about local development policy. Given the design of our conjoint experiment, we define sorting as a policy alternative having a different effect on the probability that Democrats and Republicans choose a development plan relative to the status quo. Polarization is defined as the policy alternative having an opposite effect on the two groups, increasing the probability of selecting a plan for one party and decreasing the probability for the other party. This definition is particularly compelling in this setting for which status quo policies are the baseline. Our approach requires that we estimate the conditional average marginal component effect (CAMCE) for Strong Democrats and Strong Republicans (Hainmueller et al., Reference Hainmueller, Hopkins and Yamamoto2014). Our initial CAMCE estimates are based on a split-sample approach in which we estimate the same regression for the AMCE separately for Strong Democrats and Strong Republicans.

Figure 2 presents our split-sample CAMCE estimates for Strong Democrats and Strong Republicans. Generally, these results show that Democrats and Republicans have broadly similar attitudes toward many of these proposals.Footnote ⁹ The point estimates for Strong Democrats and Strong Republicans are neither statistically nor substantively different from each other on a wide range of policies that we study, indicating an absence of pervasive partisan sorting. Moreover, even when such differences exist, the point estimates have the same rather than opposite signs as we would expect if opinions were polarized. The exceptions to these patterns tend to be a subset of issues relating to primary and secondary education and, to a lesser extent, labor.

Figure 2.

Split-sample results. CAMCE estimates show differences between Strong Democrats and Strong Republicans. Points show coefficient estimates from separate OLS regressions, and bars show robust 95 percent confidence intervals.

Beginning with the top panel, we present the CAMCEs for the Investment and Taxes factor. First, we find that the probability that Strong Republicans and Strong Democrats select a plan is increased if that plan includes subsidies and tax breaks to stimulate existing companies as opposed to the status quo. The CAMCE for Strong Republicans is 3.4 percentage points higher than for Strong Democrats, but this difference is not statistically significant.Footnote ¹⁰ Next, the same general pattern holds for a policy of using subsidies and tax breaks to attract new businesses. Democrats and Republican alike support this policy. The estimate for Strong Republicans is slightly higher but again this difference is not significant. On the other hand, respondents from both parties are less enthusiastic about encouraging investments by charities: the estimate is indistinguishable from 0 for both groups and there is little evidence of partisan sorting or polarization, again underscoring the partisan consensus that appears on each policy related to Investment and Taxes.

The next panel shows the results for the Workers and Entrepreneurs factor. Here, we find mixed evidence of sorting and polarization. There is almost no difference in the point estimates for the policy of giving tax breaks to entrepreneurs that start new businesses—in line with the previous results, this policy is popular across the board. However, when it comes to labor, there is more evidence of partisan differences: Democrats are more supportive of unions than Republicans. Both are opposed to limiting unions’ power relative to the status quo, though the effect is larger for Democrats. When it comes to expanding unions’ power, there is a large negative effect for Republicans but essentially zero effect for Democrats. Partisans are clearly sorted in this case, and there is some evidence of polarization. When it comes to providing vouchers for workers to get training, there is again evidence of sorting and some evidence of polarization.

Next, we turn to the results for the Local Services factor. Here, we see evidence of partisan sorting for all three alternatives to the status quo and some evidence for polarization in the area of public transportation. Republicans are less supportive than Democrats of affordable housing, though there is only weak evidence of polarization. Republicans are more supportive of spending more money on public safety and crime prevention, though respondents of both parties mostly support this policy proposal relative to the status quo. Finally, there is mild division on the issue of public transit. We will revisit these results in the next section: after controlling for other characteristics of respondents besides partisanship, we find smaller partisan differences over affordable housing and no polarization for any of these issues.

The next factor we consider are two Governance reforms: giving more power to the state and consolidating local governments. Both of these policies, on average, garnered neither support nor opposition relative to the status quo. There is no evidence of partisan differences.

Now, we consider the Education factor. This issue domain is where we see the greatest evidence of partisan polarization. We start with an issue that has received attention in national media: issuing vouchers that citizens can use to attend private schools. This policy has a large positive effect on the probability that Republicans select a development plan (relative to the status quo) but a negative effect for Democrats. There are similar divisions when it comes to charter schools and free pre-school—with Republicans supporting the former and opposing the latter, and vice versa for Democrats. Finally, the last education policy proposal, paying teachers more, indicates partisan sorting but not polarization. Inclusion of this policy increases the probability a platform is supported by both Democrats and Republicans, though the effect is stronger for Democrats.

The final factor we examine, in the bottom panel of Figure 2, is related to Higher Education policy. There is little evidence of partisan sorting in these estimates and no evidence of polarization. Increased investment in community colleges garners roughly equal support from both Democrats and Republicans. We see the same pattern for expanding technical vocational training. For investing in local public universities, the estimates are nearly zero for both Strong Democrats and Strong Republicans. Finally, for a proposal to expand student grant programs, the estimate is positive for both groups, though about 5.6 percentage points larger for Democrats—suggesting sorting on this policy proposal.

On the whole, we take our split-sample results to indicate that citizens exhibit similar preferences over local policies aimed at spurring economic development. Before discussing the substantive implications of the results, we raise two questions about these estimates. The definitions of sorting and polarization used in this analysis are specific to the experimental conjoint research design that we employ. One might wonder whether this method is well-suited for detecting partisan sorting. Is the absence of evidence of sorting and polarization due to a lack of partisan differences over local issues, or is it due to an inability of this tool to uncover such differences? Two observations suggest that it is the former rather than the latter. First, our estimates do detect partisan differences for some labor and education policies. Second, previous experimental conjoint studies of national policies have uncovered large partisan differences in AMCEs for policies where other survey methods would also find partisan differences (e.g., Hansen et al., Reference Hansen, Olsen and Bech2015; Ballard-Rosa et al., Reference Ballard-Rosa, Martin and Scheve2017). Another potential concern about these estimates is whether the presence or absence of partisan differences in the AMCEs is because Strong Democrats and Strong Republicans also have other characteristics which lead them to react similarly or differently to various policy attributes. We investigate this possibility in the next section.

A hierarchical model for estimating CAMCEs from experimental conjoint data

Our approach unifies two separate tasks: first, the preference measurement task that traditional OLS analysis of conjoint data enables; and, second, fitting a regression of the estimated preferences on individual-level characteristics.

To motivate the method, consider the more familiar setting of measuring preferences via a standard survey question. For instance, we might directly ask whether respondents support or oppose expanding charter schools. To investigate the correlates of support for charter schools, we could then regress responses on respondent-level covariates.Footnote ¹¹ In conjoint experiments, this exercise is not as straightforward, because we must first measure preferences from the choices made in the conjoint tasks and typically respondents do not complete enough tasks to nonparametrically identify individual-level marginal component effects. As such, the conjoint literature has typically focused on estimating AMCEs or simple CAMCEs that can be estimated via split-sample approaches. We refer to this approach as “complete pooling” because it does not explicitly model individual-level heterogeneity in parameter estimates across respondents (Gelman and Hill, Reference Gelman and Hill2007).

Alternatively, one could nonparametrically estimate individual-level marginal component effects (IMCEs) if each respondent completed a large enough number of conjoint tasks. In that case, we could run separate OLS regressions for each respondent. These estimates would converge to the true IMCEs as the number of tasks grows large. In the limit, we could perfectly measure individual-level parameters, then regress these preference parameters on individual-level covariates—just as we would with traditional survey questions. This “no pooling” approach does not share information between respondents in estimating parameters. While theoretically possible, this strategy is typically not feasible in practice because each respondent completes only a relatively small number of tasks.

Our proposed method provides an intermediate between the complete-pooling and no-pooling approaches. We use a hierarchical model that allows for individual-level heterogeneity in the way that conjoint levels affect the probability that the respondent prefers a particular profile. We then model these individual coefficients in a second-level regression as a function of respondent-level covariates. We estimate the model using a random-effects framework that allows for partial pooling between similar observations.

We briefly describe the setup here. For more details and further discussion, see Appendix E. Let $i$ index respondents ($i = 1\comma\; \dots \comma\; N$) and let $j$ index conjoint profiles ($j = 1\comma\; \dots \comma\; J$). If respondent $i$ preferred profile $j$ to the alternative, then we observe $y_{ij} = 1$; otherwise $y_{ij} = 0$. Let $X_{ij}$ denote a vector of dummy variables that specifies the conjoint levels that respondent $i$ saw for profile $j$.

The first-level regression models conjoint responses as a linear function of the conjoint levels:

(1)$$y_{ij} = \alpha_i + X_{ij}' \beta_i + \epsilon_{ij}.$$

In this equation, $\alpha _i$ is an intercept term, which may vary at the individual level, that indicates the probability respondent $i$ chooses a profile that features the baseline level of each factor. $\beta _i$ is a parameter vector that relates the conjoint profile features to the probability of choosing that profile. Finally, $\epsilon _{ij}$ is a mean-zero error term. Under the complete-pooling approach, we set $\alpha _i = \alpha$ and $\beta _i = \beta$ for all respondents, and estimate Equation 1 via OLS. Under randomization, $\beta$ represents the vector of AMCEs.

Instead, we allow for some heterogeneity, allowing elements of $\beta _i$ to vary as a function of individual-level covariate vector $Z_i$ (which includes a column of 1's as an intercept). In particular, we specify the following linear functional form for element $k$ of $\beta _i$:

(2)$$\beta_i^{k} = Z_i' \gamma_k + \eta_{ik}.$$

The coefficient vector $\gamma _k$ indicates how the expected individual-level marginal component effect varies as a function of respondent-level covariates, and $\eta _{ik}$ is a mean-zero error term. Because $Z_i$ may contain several variables, it allows us to characterize how some variable of interest—such as partisan identification—covaries with conjoint preferences after adjusting for other covariates. We set up an analogous model for $\alpha _i$, the individual-level probability of selecting the status quo as the preferred policy.

It is useful to consider Equation 2 as analogous to the approach taken when modeling answers to traditional survey responses. In that case, we would replace $\beta _i^{k}$ on the left-hand side with the actual survey response, and the $\gamma$ coefficients would be the usual linear regression coefficients. In our case, we are jointly estimating the preference parameter $\beta _i$, along with the second-level coefficients $\gamma$.

Finally, we place several distributional assumptions on $\epsilon _{ij}$ and $\eta _{ik}$—namely, that they are normally distributed. We estimate the model in a Bayesian framework using Stan (Carpenter et al., Reference Carpenter, Gelman, Hoffman, Lee, Goodrich, Betancourt, Brubaker, Guo, Li and Riddell2017). Estimation in a Bayesian framework is useful for several reasons. It provides a simple method of estimating IMCE parameters and associated uncertainty—a task that is more difficult with maximum likelihood estimation. Additionally, hierarchical models in a Bayesian framework have built-in regularization that helps deal with the multiple comparisons problem (Gelman et al., Reference Gelman, Hill and Yajima2012)—an attractive feature for our application, where we have many parameters to estimate and many quantities of interest. For more details on estimation, including methods used to assess convergence, see Appendix E.

Our approach allows a richer description of heterogeneity in conjoints, enabling us to make statements of the form, “On average, Democrats are $x$ percentage points more likely than demographically similar Republicans to support a plan that includes expanding union power, relative to the status quo.” However, there are several potential limitations, which we discuss in more detail in Appendix E. Most importantly, we can only control for observable individual-level characteristics; standard caveats about omitted variables bias apply. The second-level coefficients cannot be interpreted as causal without stronger assumptions. This approach is primarily a method for richer description—not for causal inference.

Multivariate estimates

Our main specification models the conditional average marginal component effect as a function of a seven-point party identification scale (with indicators for each response option), along with extensive sociodemographic control variables including age, race, sex, education, income, employment status, homeownership, length of time living in the region, and MSA indicators. The coefficients on partisanship therefore capture differences between Democrats and Republicans, after adjusting for other observable characteristics.

First, to demonstrate the advantage of the hierarchical model, we plot the distribution of estimated individual marginal component effects—in the notation of the previous section, the distribution of the $\beta _i$'s—in Figure 3. The lines in this figure show kernel density estimates of the posterior means across all 7800 survey respondents, while the points show the mean of the distributions—the model-based equivalent of the AMCEs presented earlier in the paper. This figure visualizes the variation in the effect of individual policies’ inclusion in a policy bundle on respondents’ probability of preferring that bundle.

Figure 3.

Estimated individual-level marginal component effects. The lines show a kernel density estimate of the posterior means of the IMCEs, while the dots show the average of the posterior means. Estimates are from the hierarchical model with sociodemographic covariates predicting IMCEs.

Next, we can use the second-level regression estimates to investigate the nature of this heterogeneity, especially as it pertains to partisan sorting and polarization. Table 2 shows the estimated difference between the CAMCEs for Strong Democrats compared to Strong Republicans on each policy proposal, after adjusting for observable sociodemographic variables. The table reports the posterior means, posterior standard deviation, and central 95 percent credible intervals.

Table 2.

Comparison of partisan CAMCEs

Estimated second-level coefficients on the indicator for Strong Republican, relative to Strong Democrat. The first two columns specify the policy proposal. “Mean” refers to the posterior mean of the coefficient, while “post. SD” is the posterior standard deviation. The final column shows the 0.025 and 97.5 quantiles of the posterior distribution, i.e., the central 95 percent credible interval. Asterisks indicate that the 95 percent credible interval excludes 0.

This table is especially useful for understanding sorting, which again we define in our context as a significant difference in CAMCEs. The patterns are broadly similar to what we saw previously. There are minimal differences between Democrats and Republicans across all Investment and Tax policy proposals, giving tax breaks to entrepreneurs, Governance policy proposals, and several Higher Education proposals. We find that the disagreements over unions, worker training vouchers, and Education policy that we documented previously are all robust to inclusion of sociodemographic covariates.

The main differences from our previous results arise in the Local Services dimension. In the split-sample approach, we found some mild differences across all three of these policy proposals. However, after adjusting for covariates, the only significant difference between Democrats and Republicans is on the proposal to spend more on public safety and crime prevention. Strong Republicans have a 6 percentage point higher CAMCE for this proposal, on average, than Strong Democrats. On affordable housing and public transportation, Democrats still have larger CAMCEs but these differences are small and the 95 percent credible interval includes 0.

We also see a difference in the results for investing in local public universities. There was no evidence of partisan sorting over this issue in the split-sample results but once we control for the demographic characteristics of respondents, Strong Republicans have a 4 percentage point lower CAMCE for this policy, on average, compared to Strong Democrats. This difference in the conditioned and unconditioned partisan comparisons is consistent with Republicans having demographic characteristics associated with greater support for higher education but once we control for these characteristics, Republicans may be ideologically more skeptical about higher education investments.

To investigate the extent of partisan polarization, we need a measure of whether Democrats and Republicans stand on opposite sides of an issue—not just whether they have a significant difference in CAMCEs, on average. To approach this question, we use the model to generate predictions of IMCEs for each respondent in the sample under two assumptions. First, we create two modified individual-level covariate matrices, $\tilde {Z}^{dem}$ and $\tilde {Z}^{rep}$, which are identical to the actual $Z$ matrix except we set partisanship to Strong Democrat or Strong Republican, respectively. Second, we predict the marginal component effects for each individual under these counterfactual covariate values. Third, we summarize the resulting distributions. The amount of overlap of the two resulting distributions, as well as whether they tend to fall above or below 0, tell us about the extent of partisan sorting and polarization on each issue.

Additionally, we compute the posterior probability that the means of the respective distributions have opposite signs. To do this, we simply compute the counterfactual AMCEs for $\tilde {Z}^{dem}$ and $\tilde {Z}^{dem}$ for each sample drawn from the posterior distribution of parameter values (i.e., the average of the predicted IMCEs). We then compute the proportion of times that the counterfactual AMCEs have opposite signs.

These results are presented in Figure 4, which plots the predicted distribution of conditional average marginal component effects under the assumption that everyone is a Strong Republican (red line) or a Strong Democrat (blue line). The points show means of the respective distributions, and percentages give the probability that the means have different signs.

Figure 4.

Simulated partisan marginal component effects. Distribution of predicted individual-level marginal component effects, assuming everyone is a Strong Democrat or a Strong Republican, holding other covariates fixed at their observed values. Lines show posterior means of kernel density estimates applied to 500 samples from the posterior; points show posterior means. Percentages refer to the posterior probability that the partisan means have different signs. The omitted default category for each dimension is the status quo.

There are several issues on which partisans are sorted—in the sense of some partisans having different CAMCEs—but not polarized. On the issue of limiting unions’ power, while Democrats are much less amenable to this policy than Republicans, there is still substantial overlap in the distributions, and the means of the distributions are both negative (with only a 9.5 percent probability of having different signs). A similar pattern exists for spending more on public safety, paying teachers more, and expanding student grant programs—all of which have less than 10 percent probability of the average partisans being on opposite sides of the issue. In these cases, partisans have different CAMCEs in magnitude, but the sign is the same on average. Partisans, then, may differ in the strength of their opinions on these issues relative to the status quo but they do not find themselves in opposition.

There is substantial polarization, however, on three of the primary and secondary education policy proposals: school vouchers, free pre-school, and charter schools. On these issues, the average difference between Democrats and Republicans is so large that shifting the distributions results in almost no common ground. On all three of these proposals, the probability of partisans being on opposite sides of the issue is greater than 95 percent.

Finally, there is also some evidence of polarization when it comes to labor policies, though it does not meet the strict 95 percent threshold. For worker training vouchers and expanding union power, the probability of partisans standing on opposite sides is roughly 93 percent and 91 percent, respectively.