3.1 Distributions

The observed number of respondents $y_{ikt}$ offering an affirmative opinion (e.g., in support of democracy) for each country $i$ , year $t$ , and survey item $k$ , is modeled as a binomial distributed count:

(1) $$\begin{eqnarray}y_{ikt}\sim \text{Binomial}(s_{ikt},\unicode[STIX]{x1D70B}_{ikt}).\end{eqnarray}$$

There are then two ways of proceeding. In the simpler binomial specification I model the probability $\unicode[STIX]{x1D70B}_{ikt}$ of offering an affirmative opinion directly, as a function of item and country–time effects (e.g., Linzer Reference Linzer2013; Caughey and Warshaw Reference Caughey and Warshaw2015). However one could also follow McGann (Reference McGann2014) in utilizing a beta prior on the probability parameter. This allows for some additional dispersion in the observed survey responses, which captures sources of error over and above simple sampling error. Indeed, since public opinion survey data are afflicted by numerous sources of errors—including methods of questionnaire translation and respondent selection, survey mode, and interviewing style (e.g., Weisberg Reference Weisberg2005)—allowing for overdispersion in survey responses appears to be a prudent course of action.

I thus use the simpler binomial specification for three of the six models, and the binomial with beta prior, or beta-binomial, for the other three. The beta-binomial specification then also includes the following step:

(2) $$\begin{eqnarray}\unicode[STIX]{x1D70B}_{ikt}\sim \text{Beta}(\unicode[STIX]{x1D6FC}_{ikt},\unicode[STIX]{x1D6FD}_{ikt}).\end{eqnarray}$$

The two shape parameters of the beta distribution can be reparameterized to an expectation parameter, $\unicode[STIX]{x1D702}$ , and a dispersion parameter, $\unicode[STIX]{x1D719}$ :

(3) $$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D6FC}_{ikt} & = & \displaystyle \unicode[STIX]{x1D719}\unicode[STIX]{x1D702}_{ikt}\end{eqnarray}$$

(4) $$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D6FD}_{ikt} & = & \displaystyle \unicode[STIX]{x1D719}(1-\unicode[STIX]{x1D702}_{ikt}).\end{eqnarray}$$

3.2 Item and country parameters

In the case of the binomial specification, the probability parameters are modeled directly as a function of the latent country–year estimates and item parameters; in the case of the beta-binomial, the beta expectation parameter receives this measurement model. I utilize three variations of this measurement model. The first simply includes country–year latent effects and item intercepts (the beta-binomial version is shown here):

(5) $$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D702}_{ikt} & = & \displaystyle \text{logit}^{-1}(\unicode[STIX]{x1D706}_{k}+\unicode[STIX]{x1D703}_{it})\end{eqnarray}$$

(6) $$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D706}_{k} & {\sim} & \displaystyle \text{N}(\unicode[STIX]{x1D707}_{\unicode[STIX]{x1D706}},~\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D706}}^{2}).\end{eqnarray}$$

The item intercepts $\unicode[STIX]{x1D706}$ adjust the location of the latent opinions for the idiosyncrasies of each survey item. They can thus be thought of as item bias effects. These intercepts are modeled hierarchically, with an expectation $\unicode[STIX]{x1D707}_{\unicode[STIX]{x1D706}}$ and variance $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D706}}^{2}$ estimated from the data. This hierarchical specification shrinks the item intercepts toward the mean to the extent that data are scarce, which guards against small within-item samples producing extreme estimates.

Survey items are, moreover, likely to have differing effects in different countries, a problem known as lack of equivalence (Stegmueller Reference Stegmueller2011). For example, one method of measuring support for democracy is to ask respondents for their opinions about having the army govern the country. Respondents in countries with a history of military rule are likely to respond quite differently than respondents in countries without such a history.

Fortunately, each item is asked multiple times in a given country. When replicates of items across units (respondents or countries) are available, analysts may also include parameters capturing item by unit bias (Skrondal and Rabe-Hesketh Reference Skrondal and Rabe-Hesketh2004). The second version of the measurement model thus includes a set of item by country effects $\unicode[STIX]{x1D6FF}$ to capture the heterogeneity in item bias across countries:

(7) $$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D702}_{ikt} & = & \displaystyle \text{logit}^{-1}(\unicode[STIX]{x1D706}_{k}+\unicode[STIX]{x1D6FF}_{ik}+\unicode[STIX]{x1D703}_{it})\end{eqnarray}$$

(8) $$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D6FF}_{ik} & {\sim} & \displaystyle \text{N}(0,~\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D6FF}}^{2}).\end{eqnarray}$$

These item–country intercepts are also modeled hierarchically, which is helpful as the observed data are especially likely to be sparse when divided by country as well as item. By treating both the item and item–country effects as varying intercepts, or random effects, they can be interpreted as error terms (McGraw and Wong Reference McGraw and Wong1996). This lends an intuitive understanding to their role in the measurement equation: the $\unicode[STIX]{x1D706}$ effects can be seen as the item-level residuals, while the $\unicode[STIX]{x1D6FF}$ effects can be seen as the item–country-level residuals, leaving $\unicode[STIX]{x1D703}$ as the item and item–country adjusted estimates of latent support for democracy.

Finally, measurement models often also include item slopes, known as factor loadings in the factor analysis framework and discrimination parameters within the IRT approach. Whatever the name, item slopes allow the strength of the relationship between observed responses and latent traits to vary across the items. Where an item shows a weaker relationship with the latent variable, it “loads” to a lesser extent than items showing a stronger relationship. I extend the second model by incorporating such item slopes $\unicode[STIX]{x1D6FE}$ :

(9) $$\begin{eqnarray}\unicode[STIX]{x1D702}_{ikt}=\text{logit}^{-1}(\unicode[STIX]{x1D706}_{k}+\unicode[STIX]{x1D6FF}_{ik}+\unicode[STIX]{x1D6FE}_{k}\unicode[STIX]{x1D703}_{it}).\end{eqnarray}$$

With both varying intercepts and varying slopes, this is a type of hierarchical (generalized) linear model, with observed responses nested within both items and countries (ignore time for the moment). As such, it is desirable to model the item-level equations jointly using a bivariate normal (Skrondal and Rabe-Hesketh Reference Skrondal and Rabe-Hesketh2004; Gelman and Hill Reference Gelman and Hill2007). This allows item intercepts and slopes to be correlated, with the $\unicode[STIX]{x1D70C}$ parameter capturing the degree of covariation.

(10) $$\begin{eqnarray}\left(\begin{array}{@{}c@{}}\unicode[STIX]{x1D706}_{k}\\ \unicode[STIX]{x1D6FE}_{k}\end{array}\right)\sim \text{N}\left[\left(\begin{array}{@{}c@{}}\unicode[STIX]{x1D707}_{\unicode[STIX]{x1D706}}\\ \unicode[STIX]{x1D707}_{\unicode[STIX]{x1D6FE}}\end{array}\right),\left(\begin{array}{@{}cc@{}}\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D706}}^{2} & \unicode[STIX]{x1D70C}\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D706}}\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D6FE}}\\ \unicode[STIX]{x1D70C}\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D706}}\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D6FE}} & \unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D6FE}}^{2}\end{array}\right)\right].\end{eqnarray}$$

With three versions of the measurement model coupled with the two response distributions (binomial and beta-binomial), there are six models in total. These are outlined in Table 1.

3.3 Dynamic effects

Finally, for all six models, the latent opinion estimates are allowed to evolve over time. Doing so smooths over any gaps in each national time series. Following previous research on modeling dynamic latent traits (e.g., Jackman Reference Jackman2005; Caughey and Warshaw Reference Caughey and Warshaw2015), the temporal evolution of latent opinion is specified as a simple local-level dynamic linear model (Durbin and Koopman Reference Durbin and Koopman2012), where the current level of latent opinion is a function of the previous year’s level plus some random noise:

(11) $$\begin{eqnarray}\unicode[STIX]{x1D703}_{it}\sim \text{N}(\unicode[STIX]{x1D703}_{i,t-1},\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D703}}^{2}).\end{eqnarray}$$

The variance of the noise term, $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D703}}^{2}$ , is held constant across countries and estimated from the data.

3.4 Identification and priors

To identify latent trait models, analysts must impose restrictions on the location, scale, and perhaps also the direction (or sign) of the parameters (Bafumi et al. Reference Bafumi, Gelman, Park and Kaplan2005). Models without item loadings are simpler in this respect as they only require location restrictions. These models were identified by fixing the first item intercept $\unicode[STIX]{x1D706}_{1}$ at a value of 1. Models with item loadings (models 3 and 6) additionally require that the scale and direction of the parameters be identified. To do so, I fix the expectation of the item intercepts $\unicode[STIX]{x1D707}_{\unicode[STIX]{x1D706}}$ to 0.5, and the expectation of the item slopes $\unicode[STIX]{x1D707}_{\unicode[STIX]{x1D6FE}}$ to 1.Footnote ³ The direction of the item slopes $\unicode[STIX]{x1D6FE}$ is then identified by constraining these to be positive.

The estimated variances are given weakly informative half-Cauchy priors: $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D706}}\sim \text{C}^{+}(0,2)$ (and similarly for $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D6FF}}$ , $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D6FE}}$ , and $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D703}}$ ). For the models including item slopes as well as intercepts, the variance–covariance matrix of item intercepts and slopes is decomposed into the product of the variances for each vector of parameters and a $2\times 2$ correlation matrix, with $\unicode[STIX]{x1D70C}$ being the estimated correlation (Stan Development Team 2017). This correlation matrix is given an LKJ prior (Lewandowski, Kurowicka, and Joe Reference Lewandowski, Kurowicka and Joe2009).

The expectation of the item intercepts $\unicode[STIX]{x1D707}_{\unicode[STIX]{x1D706}}$ (for models 1, 2, 4, and 5) is given a $\text{N}(1,2)$ prior while the dispersion parameter $\unicode[STIX]{x1D719}$ (for the beta-binomial models), is given a $\unicode[STIX]{x1D6E4}(4,0.1)$ prior. Finally, for all models, the initial value of latent opinion for each country, $\unicode[STIX]{x1D703}_{i1}$ , receives a $\text{N}(0,1)$ prior.

4.1 The concept of support for democracy

Political theorists since Aristotle have long argued the presence or absence of a democratic political system is somehow related to the attitudes and orientations of the citizenry. Interest in this theory continued into the modern era, leading to the development of the concept of political culture. According to scholars such as Inglehart and Welzel (Reference Inglehart and Welzel2005) and Lipset (Reference Lipset1959) democracy is legitimate—and stable—when it is “congruent” with the political culture. Put another way, democracy requires a democratic political culture.

There are in fact two distinct conceptualizations of democratic political culture. According to the first, citizens provide explicit support for democracy when they prefer a democratic regime to some nondemocratic alternative (e.g., Rose, Mishler, and Haerpfer Reference Rose, Mishler and Haerpfer1998; Norris Reference Norris and Norris1999; Mattes and Bratton Reference Mattes and Bratton2007; Fuchs-Schündeln and Schündeln Reference Fuchs-Schündeln and Schündeln2015). Here, democracy is legitimate because it is believed to be preferable by the public. According to the second conceptualization, citizens provide implicit support when they subscribe to a broader set of values emphasizing trust, tolerance, and freedom (Inglehart Reference Inglehart2003; Inglehart and Welzel Reference Inglehart and Welzel2005). Here, democracy is legitimate because it is consistent with citizen’s deeper values and strivings. Although both kinds of democratic political culture have been advocated as providing support for democracy, or perhaps even spurring democratization, the focus of this paper is on the first kind, explicit support for democracy, often simply referred to as “support for democracy.”Footnote ⁴

4.2 Data on support for democracy

I collected all available nationally aggregated responses to questions on support for democracy that were gathered by cross-national survey projects utilizing representative national samples of citizens. Data are collected from eleven survey projects including the World Values Survey and all the Global Barometer projects (see online supplementary materials available online at https://doi.org/10.1017/pan.2018.32). Surveys with relevant items were fielded in 144 countries over a 24 year period between 1992 and 2015. There are 3,014 nationally aggregated responses, obtained from 1,165 separate national survey samples.Footnote ⁵

These data epitomize the challenges of measuring cross-national time-series opinion. First, they are sparse over time and space. If the focus is restricted to the 132 countries that were surveyed at least twice on explicit support over the years from 1992 to 2015, there is a potential dataset of 3,168 country–years. However, surveys were conducted in just over a third of these country–years. I present a visualization of the sparseness of this data in Figure 1. The top panel shows the fragmented supply of support for democracy measures across time for eight countries, selected for variance across regions and in availability of data.

To take the example of South Africa, questions on explicit support for democracy were asked in 11 national surveys over the period in question: by the World Values Survey in 1996, 2001, 2006, and 2013, the AfroBarometer project in 1999, 2003, 2005, 2008, and 2012, and Pew Global Attitudes in 2002 and 2013. Data on democratic support are thus only available for ten out of the 24 years in South Africa—and this is a case that has above average survey coverage.

Figure 1.

Sparseness of Aggregate Support for Democracy By Country, Year, and Survey Item. The first panel shows the availability of at least one survey item across a selection of eight countries and all 24 years. The other three panels indicate the availability of data for the three most common question themes. The wording of all survey items is included in the online supplementary materials.

Second, to compound the problem, researchers have not settled on standard survey questions for measuring democratic political culture. Indeed, there is an extraordinary diversity of approaches to measuring support for democracy: as many as 37 different survey questions clustered within nine broad approaches to question wording.Footnote ⁶ The lower panels of Figure 1 show the supply of support for democracy data within the three most prominent of these measurement approaches. Once disaggregated in this way, the data are clearly even more sparse across the country–year matrix.

To continue the South African example, although 11 surveys fielded questions on support for democracy in this country, each used several different items, resulting in 41 data points in total. These 41 data points are however fragmented across seven different questions. If one were to focus on a single survey question to obtain a meaningful time series, most of the data would have to be discarded. Even the most popular survey item, the question asking respondents the extent to which they support or oppose having a strong but undemocratic leader, is asked only in ten out of the 24 years.

As such, once available data on support for democracy are divided by survey item as well as country and year, they begin to look very sparse indeed. If survey questions from every one of the nine major approaches to question wording were asked annually in each country, there would be 28,512 country-level data points. Yet, only slightly more than 10% of these potential item–country–year cells actually feature data. Classical methods of measuring public opinion using multiple items, such as factor analysis of the complete cases, are simply not an option here. The data are too sparsely scattered across survey item, country, and year. Indeed, there are no complete cases unless one ignores the temporal dimension.

Given these limitations, analysts have tended to abandon any temporal variation in support for democracy (and other measures of cross-national opinion), focusing instead on cross-sectional variation. I aim to overcome these limitations by estimating support for democracy over 132 countries and over 24 years, creating a full time-series, cross-sectional dataset of 3,168 observations.

4.3 Estimation

The six models are estimated using Bayesian Markov-Chain Monte Carlo (MCMC) methods via Stan software, which implements Hamiltonian Monte Carlo sampling (Carpenter et al. Reference Carpenter, Gelman, Hoffman, Lee, Goodrich, Betancourt, Brubaker, Guo, Li and Riddell2017; Stan Development Team 2017). Four parallel chains were run for 1,000 samples each, with the first 500 samples in each chain used for warm up, and discarded, and the remaining 2,000 samples of the posterior density thinned by half and analyzed further. This number of iterations proved to be more than sufficient for convergence, with the $\hat{R}$ diagnostic reaching a value of between 0.95 and 1.05 for all parameters in all models.

4.4 Preliminary results

Before testing the accuracy of the six models and verifying the validity of the estimates, I include a table (Table 2) showing parameter estimates (from model 5) and observed response proportions for three countries—the US, South Africa, and China—over five years—2005 to 2009. This table includes labels for country, year, and survey item; parameter estimates for item bias, item–country bias, and the country–year latent opinion; and both observed and simulated response proportions.

By employing a “dataset” perspective, Table 2 helps explicate the modeling framework employed in this paper. In particular, in certain countries and years (e.g., the first row in the table, corresponding to the United States in 2005), no public opinion surveys asking support for democracy questions were fielded. The modeling framework, however, can estimate opinion even in years where no survey measures were available. The table also shows that other country–year combinations (such as the US the following year, in 2006) benefit from having several observed survey responses.

Table 2.

Estimated parameters and observed data for three countries and five years.

Parameter estimates are drawn from those obtained using Model 5 and are unstandardized.

Although the main focus of the paper will be on the latent country–year estimates $\unicode[STIX]{x1D703}$ , Table 2 also allows readers to compare observed response proportions ( $y_{ikt}/s_{ikt}$ ) with those simulated from the model, contingent on the other parameters ( ${\tilde{y}}_{ikt}/s_{ikt}$ ). Doing so helps demonstrate that the modeling framework can be compared to simple linear or generalized linear models, where an outcome (here, $y_{ikt}$ ) is modeled as a function of some parameters and/or data. Simulated response proportions are also used to test and compare the accuracy of the six models, which is the task we turn to next.

5.1 Internal and external validation

In this section, I compare the accuracy of the latent opinion estimates obtained from each model, both in comparison to an absolute standard, and in comparison to each other. The first test is a test of the predictive accuracy of the models when using the same data that were used to fit the model, which Hastie, Tibshirani, and Friedman (Reference Hastie, Tibshirani and Friedman2009) refer to as internal validation. In particular, the mean absolute error (MAE) is used to measure the average discrepancy between the observed proportions of respondents offering a prodemocratic attitude $y_{ikt}/s_{ikt}$ , and the simulated proportions ${\tilde{y}}_{ikt}/s_{ikt}$ :

(12) $$\begin{eqnarray}\text{MAE}=\frac{1}{J}\mathop{\sum }_{j\in ikt}^{J}\left|\frac{y_{ikt}}{s_{ikt}}-\frac{{\tilde{y}}_{ikt}}{s_{ikt}}\right|.\end{eqnarray}$$

Internal validation is simple to conduct, but favors more complex models. Reliance on metrics of internal validation could therefore lead to the selection of a model that overfits the dataset at hand. Analysts thus instead use information criteria, which attempt to estimate out-of-sample predictive error by penalizing models as their parameters increase in number. A good choice of information criterion for Bayesian MCMC methods is the “Leave-One-Out” information criterion (LOO-IC), which Vehtari, Gelman, and Gabry (Reference Vehtari, Gelman and Gabry2017) argue to be superior to alternatives such as the Deviance and Watanabe–Akaike information criteria. I follow their advice in including Stan code for estimating the LOO-IC for each of the six models, and report these results below.

Better still, however, is to select models using data that are not also used to fit the model, which is known as external validation (Hastie, Tibshirani, and Friedman Reference Hastie, Tibshirani and Friedman2009). To do so, the dataset of national opinions is randomly split into a 75% training set and a 25% test set. The six models are fit to the training set, and the resulting parameter estimates are used to predict the national proportions offering a supportive (i.e., prodemocratic) response for each of the 744 survey items comprising the test dataset. I again calculate the MAE, but now in predicting the error in the held-out test dataset.

Finally, I examine the accuracy of the estimates of uncertainty produced by each models by calculating their credible interval coverage (CIC). To do so, I find—for each model—the percentage of the $J=744$ observed survey proportions that are included in the 80% credible interval of the corresponding simulated survey proportions ( ${\tilde{y}}_{j}/s_{j}$ ):

(13) $$\begin{eqnarray}\text{CIC}=\frac{100}{J}\mathop{\sum }_{j=1}^{J}\left[\frac{y_{j}}{s_{j}}\in \text{CI}_{80}\left(\frac{{\tilde{y}}_{j}}{s_{j}}\right)\right].\end{eqnarray}$$

To provide a baseline comparison for the validation tests, I also fit Caughey and Warshaw’s (Reference Caughey and Warshaw2015) DGIRT model to the training dataset and used it to predict responses on the test set.Footnote ⁷ Three naïve methods of estimating the out-of-sample proportions are also included as additional baselines. In the first of these, I use the country-mean proportions from the training dataset to predict the response proportions in the test set. Second, I use the item mean proportions, and third, I use the grand mean response proportion across the entire training dataset.

The results of these internal and external validation tests are displayed in Table 3. Beginning with the tests of internal validation, three findings are apparent. First, all six models offer a substantially better fit to the observed proportions supporting democracy than the baseline estimates. Indeed, taking the country means as a baseline, the most accurate models offer up to 79% reduction in MAE in tests of internal validation. This is hardly surprising given the difficulty in estimating the proportion responding affirmatively to a particular support for democracy survey item in a particular country and year knowing nothing other than the average proportion responding affirmatively in that country across all years and items. Yet it does show already that these models are adding value.

Table 3.

Internal and External Validation Tests.

Internal validation uses the same data for model fitting and validation. External validation creates two separate datasets: models are fit to the 75% training set and validated using the 25% test (or hold-out) set. Percent improvement in MAE is a comparison between model MAE and country-mean MAE. The DGIRT model is proposed by Caughey and Warshaw (Reference Caughey and Warshaw2015) and implemented in the dgo R package.

Perhaps more interesting is the second finding, which is that models 1 and 4—which include item intercepts but not item slopes or item–country intercepts—offer substantially worse fit than the other, more complex models. The error rate is roughly halved when adding item–country intercepts (which are incorporated in models 2, 3, 5, and 6). This result is confirmed by the LOO-IC measures. Although the LOO-IC penalizes the log-likelihood for the number of estimated parameters, models 2 and 3 offer a better fit than model 1, as do models 5 and 6 when compared with model 4.Footnote ⁸ There is however little to distinguish between the models with item slopes (3 and 6) and the models without (2 and 5).

I now turn to the external validation tests. These offer a better means of gauging model fit because the models are estimated and tested using different datasets. Focusing first on the MAE results, one can see that the six models continue to offer improvements in accuracy when compared with the baseline, country-mean estimates. The difference, however, has diminished when compared with the internal validation MAE results. Part of the gap between the model estimates and the baseline estimates were thus due to the models overfitting the data. Nevertheless, any of the six models offers a gain in accuracy over the baseline fit, with up to 45% reduction in MAE.

In addition, any of the models developed in this paper perform at least as well in predicting out-of-sample survey responses, and usually better, than Caughey and Warshaw’s (Reference Caughey and Warshaw2015) DGIRT model. The DGIRT model is about as accurate as model 1, which uses a binomial specification and includes only item intercepts. As mentioned, although the DGIRT model can, in principle, be used to estimate cross-national opinion, it was developed instead for estimating subnational opinion. Analysts face differing challenges in these two contexts: when estimating subnational opinion (but not cross-national opinion), samples are small and unrepresentative; when estimating cross-national opinion (but not subnational opinion), items may not be equivalent across countries. These results demonstrate that analysts should use models designed for the idiosyncrasies of cross-national opinion when estimating cross-national opinion.Footnote ⁹

These external validation tests also confirm that the simpler, item-intercept only models (models 1 and 4) are the least accurate. The additional complexity added by including item–country intercepts (models 2 and 5) produces a meaningful reduction in MAE of around 1 percentage point when compared with the item-intercept only models. In contrast, adding item slopes or factor loadings does not improve predictive accuracy at all.

In addition, the external validation tests show that the beta-binomial (models 4–6) specifications are slightly more accurate than the corresponding binomial models (1–3), which contrasts with the results of the internal validation tests. The additional dispersion added by the beta-binomial enhances the accuracy of the estimates, perhaps by capturing some of the nonsampling error endemic in public opinion data.

Finally, on to tests of CIC. These tests measure the accuracy of the estimates of uncertainty produced by each model. A model with accurate uncertainty estimates should have similar empirical and nominal levels of CIC. Since I use 80% credible intervals, one should expect 80% empirical coverage. Coverage substantially below this nominal level shows overly optimistic standard errors; coverage substantially above this level indicates overly conservative standard errors.Footnote ¹⁰

The results indicate that the uncertainty estimates generated by the six models are all too optimistic. The standard errors, in other words, are too small. None of the rates of empirical CIC come appreciably close to the nominal level of 80%. There are of course, many sources of error in cross-national public opinion data (e.g., Weisberg Reference Weisberg2005), and I have explicitly modeled only a few. Moreover, some sources of error—such as the translation problems in the World Values Survey identified by Kurzman (Reference Kurzman2014)—are impossible to model.

However, the beta-binomial specification, which includes an overdispersion parameter, $\unicode[STIX]{x1D719}$ , proves to have substantially more accurate uncertainty estimates than the simpler binomial specification. While the binomial CICs are very poor, ranging from 17 to 39%, the beta-binomial CICs are much closer to nominal 80% level, ranging from 38 to 61%. Including item–country bias effects also produces substantially better CIC, in both the binomial and beta-binomial specifications. Models 5 and 6, with both item–country intercepts and beta-binomial distributions, have the most accurate estimates of uncertainty of all.

Weighing up all the evidence from these tests of external validation, it would appear that the models which use beta-binomial specifications and include item–country effects (models 5 and 6) are the most accurate. These models show similarly low levels of error in predicting survey responses in the test dataset and have similar CIC. Model 5 has the advantage of being simpler to code and run than model 6, as it does not require the covariance of the item effects to be estimated. Model 6, however, might be useful for situations where analysts are unable to benefit, as we have here, from prior research investigating the reliabilities (or factor loadings) of potentially relevant survey items.

5.2 Construct and convergent validation

The external validation tests have demonstrated that my modeling framework—and in particular, the beta-binomial specification with item–country intercepts—is fairly accurate in predicting observed survey responses in a hold-out sample. To further build confidence in this approach I examine, in this subsection, whether the latent estimates correspond to the theoretical construct of support for democracy. I examine, in other words, whether the country–year measures of support for democracy behave as previous scholars have suggested this variable should behave.

In particular, I consider the cross-national distribution of estimated support for democracy at certain points in time and the over-time evolution of support for democracy for certain countries. These analyses will permit some discussion of the construct validity of the estimates. I also examine the correlation, at certain points in time, between cross-national point estimates of support for democracy and cross-national experience with democracy. These correlations constitute a test of the convergent validity of the measures. The analyses that follow utilize estimates obtained from model 5.

Figure 2.

The Relationship Between Support for Democracy and Democratic Experience. Estimates of support for democracy, from Model 5, for all countries, plotted against cumulative democratic experience, in 2005 (top) and 2015 (bottom). Cumulative democratic experience is the sum of annual democracy scores discounted by 2% a year. A LOWESS line is added to each plot.

To assess the convergent validity of the model, I examine the covariation between support for democracy and cumulative experience with democracy at two points in time, 2015 and 2005. Cumulative experience with democracy is the sum of the democracy scores for a given country between the year in question and 1950, with each year’s score discounted by 2%.Footnote ¹¹ Scholars have previously demonstrated that supportive attitudes toward democracy are linked with the length of time a country has been democratic (Mattes and Bratton Reference Mattes and Bratton2007; Fuchs-Schündeln and Schündeln Reference Fuchs-Schündeln and Schündeln2015). Scholars have also argued—although perhaps not yet empirically demonstrated—that support for democracy helps democratic institutions to survive (Lipset Reference Lipset1959; Rose, Mishler, and Haerpfer Reference Rose, Mishler and Haerpfer1998; Norris Reference Norris and Norris1999). Since either of these processes would lead to a correlation between support for democracy and democratic experience, this can be interpreted as a test of the convergent validity of the estimates. To carry out this test, I plot point estimates of support for democracy against democratic experience in both 2015 and 2005 (Figure 2).

In both 2005 and 2015, a robust and positive relationship is evident between the two variables (in 2005 the Pearson’s correlation is 0.57; in 2015, 0.50). The more extensive a country’s experience with democracy, the higher the support that its citizens express for a democratic versus an autocratic system. These correlations show that the estimates of latent opinion do in fact behave as theories of democratic political culture have suggested.

Figure 3.

Estimated Support for Democracy for 8 Countries over 24 Years. Estimates of support for democracy, from Model 5, for eight selected countries over 24 years. Each plot shows 200 random draws from the posterior distribution of $\unicode[STIX]{x1D703}$ for a particular country. The posterior means are indicated using bold lines. Observed survey responses for each country are plotted using points; these are unit-normal standardized within survey item so as to display them on roughly the same scale as the latent estimates.

Moving on to construct validity, I examine the estimated levels of support for democracy for a selection of eight countries, which were selected as representing a range of levels of support as well as some interesting dynamics. These are displayed in Figure 3: each plot shows the latent estimates for a particular country over 24 years.Footnote ¹² The darker line indicates the mean value of $\unicode[STIX]{x1D703}$ in each year; the lighter lines indicate 200 random draws from the posterior density of $\unicode[STIX]{x1D703}$ , and collectively show uncertainty in the latent estimates. Finally, the observed data are also displayed on these plots using points.Footnote ¹³

First, when data are abundant in a particular country (e.g., Venezuela), the estimates are fairly precise (the y-axis is calibrated on the z-score scale). When data are scarce (e.g., Egypt and China before 2000), the estimates are noisier, and become increasingly so the larger the gap there is in the data. The beta-binomial model is fairly aggressive in smoothing across time compared with the binomial specification (not shown), which produces a more jagged, rapidly changing pattern of opinion. Such a pattern is not particularly plausible for a slow-moving orientation such as democratic political culture (Inglehart and Welzel Reference Inglehart and Welzel2005), which suggests again that the beta-binomial specification is preferable.

Second, established democracies such as Sweden show high levels of support for democracy. This is consistent with previous research focusing on particular subsets of the available survey data (e.g., Klingemann Reference Klingemann and Norris1999). An important exception to this pattern is the Anglophone democracies, such as the United States, where declining support for democracy is evident. This is consistent with recent research by Foa and Mounk (Reference Foa and Mounk2016).

Third, newer democracies show divergent trends. I examine a pair of cases from Southern Africa: Botswana has high and increasing support for democracy, which echoes previous case study research (Hjort Reference Hjort2009). In contrast, Botswana’s neighbor, South Africa, shows fairly low (and declining) support, which resonates with earlier survey research (Gibson Reference Gibson2003).

Finally, one can see that countries with a long history of autocratic rule, such as China and the Ukraine, have low levels of support, as existing research on authoritarian legacies would lead us to expect (Rose, Mishler, and Haerpfer Reference Rose, Mishler and Haerpfer1998; Fuchs-Schündeln and Schündeln Reference Fuchs-Schündeln and Schündeln2015). Moreover, in Egypt and Venezuela one can see public support for democracy reacting to political events, albeit in divergent ways. Venezuelan support for democracy steadily increased after Chavez began dismantling democratic checks and balances in 2000. In contrast, Egyptians reacted to the tumult of the Arab Spring in 2010 by turning away from democracy in the years that followed.

5.3 Item analysis

Finally, I examine the item parameters in more detail. Doing so will permit further discussion of the validity of the estimates. Moreover, the fact that item analysis is even possible illustrates another advantage of my modeling approach. I use estimates from model 6, which includes item slopes, for this section.

This analysis will focus on the item characteristic curves (ICCs), which are plotted in Figure 4. ICCs display the relationship between the proportion of a national sample responding supportively toward democracy ( $y$ -axis) and the latent estimates of support ( $x$ -axis). The vertical alignment of the curves is governed by the item intercepts $\unicode[STIX]{x1D706}$ , while the steepness of the curves is governed by the item slopes $\unicode[STIX]{x1D6FE}$ . To aid in interpretation, I group the items by their survey project, and use varying shades of gray and line types to identify the main question wording approach.

Turning to Figure 4, one can see that all 37 items display a positive relationship between the latent quantity and the observed responses. All items, in other words, have positive slopes. In addition, most items have slopes of similar magnitude. These are welcome findings, as they indicate that the included survey items do indeed measure the latent construct. It is not a particularly surprising finding, however, as items were selected based on the results of previous analyses of microlevel survey data (e.g., Rose, Mishler, and Haerpfer Reference Rose, Mishler and Haerpfer1998; Klingemann Reference Klingemann and Norris1999; Mattes and Bratton Reference Mattes and Bratton2007). Items that bore some superficial resemblance to support for democracy, but which did not display a deeper empirical relationship with this latent variable were not included in the analysis in the first place.Footnote ¹⁴

Figure 4.

Item Characteristic Curves for All Items, Grouped By Project. Lines show the item characteristic curves (ICCs) for each item, with parameter estimates drawn from Model 6. ICCs depict the relationship between the proportion of a national sample responding supportively toward democracy ( $y$ -axis) and the latent estimates of support for democracy ( $\unicode[STIX]{x1D703}$ , $x$ -axis). The vertical alignment of the curves is governed by the item intercepts $\unicode[STIX]{x1D706}$ ; the steepness of the curves is governed by the item slopes $\unicode[STIX]{x1D6FE}$ . The ICCs are grouped by survey project. The combination of line type and line shade indicates question theme. The legend is in the lower right corner. All parameters ( $\unicode[STIX]{x1D703}$ , $\unicode[STIX]{x1D706}$ , and $\unicode[STIX]{x1D6FE}$ ) are standardized such that $\unicode[STIX]{x1D703}$ has a unit-normal scale.

There are nonetheless a few items with weaker slopes and therefore more tenuous relationships with latent support for democracy. First is the “army rule” item from the New Democracies Barometer and second is the “evaluate democratic political system” item from the AsiaBarometer. Another three come from the World Values Surveys: the items asking respondents to rate the “importance of living in a democracy,” and to evaluate a “strong leader” and a “democratic political system.” This latter survey question has previously been criticized as offering only “lip service” to democracy, rather than deeply rooted support (Inglehart Reference Inglehart2003). In two out of the three survey projects in which it is employed, this type of question does indeed show a weaker relationship with latent support for democracy.

In contrast, two widely used approaches for measuring support for democracy—the “three statements” and “evaluate army rule” survey questions—perform well across regions and survey projects. Both have pronounced positive slopes, indicating that such questions allow researchers to discriminate between respondents who favor democracy and those who do not. Indeed, national samples show widely varying levels of agreement with these items as their underlying support for democracy increases: at low levels of support for democracy (two standard deviations below the mean), 25% to 40% of respondents tend to offer the democratic responses to the three statements questions; at high levels (two standard deviations above the mean), around 85% do so.

The main finding from this item analysis is that all included items show a marked, positive relationship between the latent variable and the observed responses. Indeed, most of the items show a strong relationship, indicating that they are sound measures of the latent construct of support for democracy. In addition, the ability to conduct such an item analysis illustrates another advantage of the proposed modeling framework.