METHODS

MRP is one of many methods that estimates an opinion as a function of demographic covariates and then simulates the opinion at a certain geographic unit of interest by multiplying the resulting coefficients by the share of the population falling into each covariate cell (post-stratifying). The innovation of MRP is in how it incorporates geographic information, providing more accurate estimates in sparsely populated cells.

A common implementation of MRP models individual i’s outcome y as a function of her characteristics x ₁ and x ₂ and her geographic area of residence geo:

(1)$$\Pr \left( {{y_i} = 1} \right) = {\rm{logi}}{{\rm{t}}^{ - 1}}\left( {{\beta _0} + \alpha _{j[i]}^{{x_1}} + \alpha _{k[i]}^{{x_2}} + \alpha _{g[i]}^{geo}} \right).$$

The individual-level effects α _j,k are drawn from a normal distribution, with a mean value of 0 and a standard deviation $\sigma _{j,k}^2$ estimated by the model. The geography-level effect α _g is modeled as a function of some geographic-level covariate(s) G ₁ and a larger geographic random effect region.

$$\alpha _g^{geo} \sim N\left( {\alpha _m^{region} + {\beta _1}{G_1},\sigma _{geo}^2} \right),$$

where

$$\alpha _m^{region} \; \sim\; N\left( {0,\sigma _{region}^2} \right).$$

The multilevel model allows for flexible joint estimation of individual and geography-specific correlations, yielding superior predictive accuracy by partially pooling respondents. This ensures that (1) all individuals contribute to the estimation of the individual-covariate model and that (2) geographic differences that persist after modeling individual-level predictors are not discarded. The resulting model is then used to predict opinions for each cell in census data, and geography-level estimates are calculated as the weighted average of these cells’ predictions with population shares as weights.

Despite numerous validation studies that demonstrate MRP’s superior performance over simple disaggregation (Ghitza and Gelman Reference Ghitza and Gelman2013; Lax and Phillips Reference Lax and Phillips2009; Warshaw and Rodden Reference Warshaw and Rodden2012), a comprehensive review by Buttice and Highton (Reference Buttice and Highton2013) introduces a note of caution. Across 89 surveys, the authors document evidence of substantial variation in MRP performance, finding that the correlation between MRP predictions and true state values is below 0.50 in 33 cases. On the one hand, this is an unfair test of MRP since the authors apply a naive combination of covariates uniformly across 89 surveys. But on the other hand, it may not always be the case that the researcher can easily identify and collect prognostic covariates, or that she can specify the correct functional form a priori.

Nonparametric methods can relieve the researcher of correctly determining the functional form and provide better regularization for estimating relationships in sparsely populated cells. Although these methods are not a silver bullet, I demonstrate that they can do more with less. In this letter, I focus on Bayesian additive regression trees (BART or, when combined with post-stratification, BARP) because of its well-documented predictive capabilities (see Chipman, George, and McCulloch Reference Chipman, George and McCulloch2010; Linero Reference Linero2017), intuitive connection with the post-stratification stage, and rich descriptive results on covariate importance and partial dependence. I compare BARP’s performance with several alternative regularization methods in Section 7 of the Supporting Information, finding that BARP is consistently the best-in-class method in terms of accuracy and is among the best methods in terms of correlation across geographic units.

BART estimates a function f that predicts an outcome using covariates: y = f(x). The unknown function f is approximated by h(x), which is a sum of decision trees. Each decision tree T_j divides the data based on a splitting rule designed to separate observations according to the outcome. The tree proceeds to move recursively across the covariate space and continues until a stopping rule is satisfied, leaving small groups of observations or “leafs.” Each leaf has a parameter value μ _b which captures E(Y|x), and are collectively denoted as M. These parameters are assigned to x via g(x; T, M). The full expression that approximates the unknown function f is

(2)$$y = {{\limits \sum_____{\hskip-7pt j}} {{\hskip-2pt g_j}} \left( {x;{T_j},{M_j}} \right) + \varepsilon ,\quad \quad \varepsilon \; \sim\; N\left( {0,{\sigma ^2}} \right).$$

BART uses Bayesian priors on the structure of the model and on the parameters in the terminal nodes. These priors ensure that no tree is unduly influential, thereby avoiding the overfitting problems facing more brittle tree-based methods. Estimation proceeds through a backfitting algorithm that generates a posterior distribution for all parameters. Draws from this posterior first propose a change to the structure that further protects BART from overfitting.

BARP provides two improvements over MRP. First, BARP allows for deep interactions between prognostic covariates and additive effects without requiring the researcher to specify these functional forms ex ante. Second, BARP provides superior regularization via the ensemble of weak trees that better insulates predictions from the brittleness associated with sparse data. The resulting rich model—built by identifying break points in the covariate space that most cleanly separate individuals by the outcome—maps intuitively on to the share of the population falling into each covariate bin in the post-stratification stage.^{Footnote 1} In addition, BART provides partial dependency estimates and variable inclusion proportions which may be of substantive interest to applied researchers. BARP can be modified to predict binary, categorical, and continuous outcomes. More details can be found in Section 8 in the Supporting Information.