Conflictual Eu News: An Empirical Illustration

In the article ‘Who’s Afraid of Conflict? The Mobilizing Effect of Conflict Framing in Campaign News’, SVdV theorize that exposure to EU- or EP-related news focusing on conflict will increase the likelihood of turning out in the EP elections, and that this effect is conditional on the general information environment – that is, the degree of positive vs. negative polity evaluations of the EU in the news.

SVdV use survey and media content data from twenty-one EU member states. The individual-level data contain turnout intention before the 2009 EP elections and self-reported turnout from a post-election survey. Media content data stems from the Providing an Infrastructure for Research on Electoral Democracy in the European Union project.Footnote ⁷ For both expectations put forward, SVdV report positive findings. We will show below that their approach is insufficient to derive such conclusions.

During our step-by-step reproduction and replication of SVdV’s presented empirical analysis, we discovered data and methods-related issues that required corrections.Footnote ⁸ On the one hand, these corrections help with the validity, transparency and interpretation of the results. On the other hand, they confirm that our broader concerns are not simply rooted in the particular data or analysis reported in the original article.

How ‘Specific’ is Exposure to the Specific Type of News?

While News Exposure is a simple summation of the weekly frequency with which an individual follows the specific media outlets (from 0–7 days, for five outlets, on average, per country), incorporating content-related information requires additional adjustment. The core operationalization of News Conflict combines both individual and content-related features, and SVdVFootnote ⁹ use the following formula to calculate this individual-level predictor:

$$\eqalignno{ News\,Conflict{\rm}=\,&{\rm [(exposure}\,{\rm medium}\,{\rm 1}{\times}{\rm conflict}\,{\rm frame}\,{\rm index}\,{\rm medium}\,{\rm 1)} \cr {\rm } & {\rm {\plus} (}...{\rm ) {\plus} (exposure}\,{\rm medium}\,n{\times}{\rm conflict}\,{\rm frame}\,{\rm index}\,{\rm medium}\,n{\rm )] } $$

From the media content data, each outlet has a conflict frame index. This is the proportion of news items with a conflict frame out of the total news items during the campaign that were about the EU/EP elections.

The first step carried out by SVdV includes a hierarchical model, with observations nested in countries, in which the probability to turn out in the EP elections is a function of both News Exposure and News Conflict. They also control for other variables. Next SVdV use models including only News Conflict: ‘[…] however, the model in Table 2 includes conflict news [but not mere news exposure] in light of previous findings’.Footnote ¹⁰ This may be because News Exposure had a substantively small effect, a negative effect and/or that adding News Conflict to the full model (in Table 1, SVdV) resulted in a significant improvement in model fit. We assess these decisions below, focusing on the substantive features.Footnote ¹¹ In the subsequent steps of the analysis, SVdV evaluate whether the effect of News Conflict on turnout varies across countries, and whether that variation can be explained by country-level features of the media content, modeled as cross-level interaction between polity evaluations and News Conflict.

Table 1

Multivariate Model Results: News Exposure vs. News Conflict

Note: *p<0.1; **p<0.05; ***p<0.01 (two-tailed).

The authors note that there is a substantial correlation between News Conflict and News Exposure, r= 0.89. Ideally, the inclusion of News Exposure would be required to tap into the specific effects of conflict. Using models with only one media predictor appears to be an empirical necessity, given the multicollinearity issue. Thus the decision of which predictor to use, and how it relates substantively to the excluded operationalization, becomes the most important step of the analysis – and basing this decision solely on model results with serious multicollinearity issues can be problematic.

SVdV do not discuss in detail the properties of the scenario with both predictors included in one model, but this model (Model 1 in SVdV) raises several interpretation difficulties. Within a single country, two individuals can have the same summed News Exposure score with different per-outlet consumption frequency, and this will translate into potential differences in their News Conflict scores. This difference is a function of both the variation in outlet conflict scores and how much their media consumption vector differs. The strict interpretation of coefficients would imply the classic formulation of: ‘everything else held constant’, the effect of News Exposure or News Conflict is the associated β coefficient. First, we know – given the correlation between the two predictors – that in most cases, News Conflict and News Exposure will change together. News Exposure has a small negative effect – we discuss below in detail why the magnitude is problematic – when News Conflict is held constant. This would entail at least some variation in News Exposure at each level of the News Conflict. In 65 per cent of the cases in the corrected dataset, one News Conflict value within a country is associated with one News Exposure value. More strikingly, in 97 per cent of the cases, the number of unique values of News Exposure for a fixed News Conflict value ranges between 1 and 3, which calls into question how meaningful the News Exposure coefficient (or that of News Conflict, for that matter) can be in the initial model reported by SVdV.

To address how serious the multicollinearity issue is, we estimated and report correlations for each country separately. As displayed in Panel a of Figure 1, these correlations are close to 1.0 in most countries, except Lithuania, which has the lowest degree of country-level conflict. The second-lowest correlation is for Belgium, but as the value is for country and not political system (Walloon vs. Flanders), this comes as no surprise. In short, this suggests that News Conflict is merely a rescaling of the News Exposure variable.

Fig. 1

News Exposure and News Conflict, correlations

Note: Panel (a): Country-level Pearson’s correlations between News Conflict and News Exposure, twenty-one countries, corrected measures. Panel (b): Country-level polyserial correlations between News Conflict and turnout (black) and News Exposure and turnout (gray), twenty-one countries, corrected measures. Error bars: 95% confidence intervals.

The ‘weighted’ index nature of News Conflict results in substantially different ranges for News Exposure and News Conflict, but SVdV present, discuss and base their decisions on unstandardized coefficients. We show that when standardized (and hence comparable) coefficients are used, the dismissal of News Exposure in favor of News Conflict becomes less straightforward. The first indication that this choice of reporting is problematic is displayed in Panel b of Figure 1. When looking at the zero-order correlations in each country between turnout and the two exposure variables, we find – unsurprisingly, as there is close-to-perfect collinearity in most countries – that the bivariate relationships are very similar.

For the multivariate setting, we recoded the two predictors to account for the difference in scales, which convinced us that the decision made by SVdV on the basis of the unstandardized coefficients is erroneous. We group-mean centered the two predictorsFootnote ¹² and divided them by their standard deviation. In Table 1 we report four models with the newly recoded variables.

All these models indicate that, when modeled separately, both News Exposure and News Conflict have very similar influences on the outcome variable. When we assess the standardized coefficients and their standard errors, we can see that they overlap. Furthermore, the aim of Figure 1, Panel B is to stipulate this at the bivariate level. Accordingly, the major difference in the magnitude of the effects was a simple artifact of the reporting chosen by the authors.Footnote ¹³ When the cross-level interaction is modeled, News Exposure is not statistically significant at the p<0.05 level, but absence of evidence is not evidence of absence, and using the thresholds proposed by SVdV, this effect is still statistically significant.

If the model comparisons or p-values larger than 0.05 leave room for skepticism, we can always look at substantive differences and implications. Figure 2 presents the slopes for Spain, Slovakia and Austria (as reported by SVdV in Figure 2) for News Conflict and News Exposure, estimated in separate models, using the full sample (twenty-one countries) and corrected News Conflict (see Online Appendix A). We fail to see support for their conclusion that exposure to conflict, instead of mere news exposure, increases the likelihood of participating in the EP elections.

Fig. 2

News Conflict vs. News Exposure, cross-level interaction

Note: predicted turnout as a function of News Conflict (solid lines, based on News Conflict only cross-level interaction model, Table 2) and News Exposure (dotted lines, based on News Conflict only cross-level interaction model, Table 2) for different polity evaluation levels (SP=Spain, SK=Slovakia, AT=Austria). All other variables fixed at values described in the text and by SVdV, predictions based on fixed effects only. Predictors were group-mean centered and standardized (one SD, within country).

The Mobilizing Effect of Non-Conflictual News: An Illustration

While the high correlation between specific and general news exposure shows the problem related to estimating the impact of specific content on political behavior, this does not necessarily mean that the proposed approach of linking media content features with individual exposure is necessarily wrong. However, we show through an extension of SVdV’s work that the methodological contribution is unlikely to benefit the study of media effects on individual political behavior. Even if we assume that News Conflict measures both exposure and exposure to content, our example below – which focuses on non-conflictual news – is intended to show that substantively, we could easily formulate the same conclusions about two opposite contextual situations. We use this section to show that with such a proposed operationalization, even if we disregard all the discussion about multicollinearity or results reporting, we are losing most information in substantive terms that is about context (or media content). Hence, evaluating what news conflict operationalized as such actually measures is very problematic. In our reading of the claims made in the article, we should perhaps expect no relationship (or a negative relationship) when it comes to news non-conflict.

We use the original media dataFootnote ¹⁴ to recreate the conflict scores at the outlet level and then merge it with the European Election Study 2009 individual-level data. Using SVdV’s operationalization and the 2009 PIREDEU Media Study,Footnote ¹⁵ we calculate the outlet-specific conflict score, using all four content variables indicated by the authors. Note that these are dichotomous indicators, 0 for no conflict, 1 for conflict. Each outlet can be described by an average level of conflict in its EU reporting throughout the coding period. Similarly, each outlet can be described by the average ratio of non-conflictual news. The degree of News Non-conflict will be 1 − News Conflict. This means there should be a perfect negative correlation between these two outlet features. That is indeed the case in the Media Study, at both the outlet and country levels.Footnote ¹⁶

Next, using the formula provided by SVdV and the EES data, we created the individual-level News Conflict variable, as well as a News Non-conflict variable, following the same logic, but multiplying outlet-specific exposure by the outlet-specific News Non-conflict score. The correlation between these two variables is 0.85 on the full sample, with country-specific correlation coefficients for these two variables and mere news exposure reported in the last three columns of Table 1, Online Appendix C. From the theoretically grounded perfect negative correlation between the two variables at the outlet level, we have ended up with an extremely strong positive correlation in the individual-level data. This is a joint effect of both the proposed method of linking the two levels together and the measurement of conflict framing at the media-item level.

While we do not have turnout intention measured in an earlier wave, we expect that the only implication is that the media coefficients will be larger than those reported in the previous section or the original study. However, how they compare to each other – both effects and variables – is independent of the turnout intention variable.Footnote ¹⁷ Otherwise, we used and coded all individual- and country-level variables employed in the original analysis, following the coding description from SVdV.Footnote ¹⁸

In Table 2 we present the results of interest from three turnout models on the EES data that mimic SVdV’s ‘fixed-effects’ model. Each model contains all individual- and country-level controls, with a varying intercept. Full model results are reported in Table 3, Online Appendix C. The row ‘Exposure’ contains the coefficients for the media variables, and each column head stipulates which exposure measure was used in the model. All three models indicate a positive effect of the exposure variables, independent of whether exposure is multiplied by conflict or non-conflict (see Online Appendix C for model formula and descriptive statistics). Again, this result is due to the fact that even after applying the proposed transformations, we are essentially looking at a simple news exposure measure. Furthermore, this also indicates that, given the coding at the news-item level, the proposed approach is unable to differentiate between qualitatively and quantitatively opposite media features; it supplies coefficients that are heavily conflated by mere exposure.

Table 2

The Impact of Mere Exposure, Conflict and No Conflict, EES 2009

Note: Best-fitting model based on model comparison: news non-conflict. *p<0.1; **p<0.05; ***p<0.01 (two-tailed).