Related Literature

The academic literature has made large strides towards understanding the triggers of civil conflict. A part of the literature has focused on establishing links to specific factors such as ethnic cleavages (Reynal-Querol and Montalvo Reference Reynal-Querol and Montalvo2005; Esteban, Mayoral, and Ray Reference Esteban, Mayoral and Ray2012; Caselli and Coleman Reference Caselli and Coleman2013), climate (Miguel, Satyanath, and Sergenti Reference Miguel, Satyanath and Sergenti2004; Dell, Jones, and Olken Reference Dell, Jones and Olken2012; Buhaug et al. Reference Buhaug, Nordkvelle, Bernauer, Böhmelt, Brzoska, Busby, Ciccone, Fjelde, Gartzke and Gleditsch2014), or natural resources (Brückner and Ciccone Reference Brückner and Ciccone2010; Bazzi and Blattman Reference Bazzi and Blattman2014). This literature is more concerned with causal identification and less with forecasting power. Another part of the literature has looked at using a mix of political and economic indicators to explain conflict (e.g., Fearon and Laitin Reference Fearon and Laitin2003; Collier and Hoeffler Reference Collier and Hoeffler2004; Collier, Hoeffler, and Rohner Reference Collier, Hoeffler and Rohner2009; Gleditsch and Ruggeri Reference Gleditsch and Ruggeri2010; Besley and Persson Reference Besley and Persson2011a). For a review of this literature, see Blattman and Miguel (Reference Blattman and Miguel2010).

Naturally, the forecasting literature started with relying on structural factors in forecasting.Footnote ² An exhaustive review of this growing literature is beyond the scope of this article. For an overview, see Schrodt, Yonamine, and Bagozzi (Reference Schrodt, Yonamine and Bagozzi2013), Ward et al. (Reference Ward, Metternich, Dorff, Gallop, Hollenbach, Schultz and Weschle2013), and Hegre et al. (Reference Hegre, Metternich, Nygård and Wucherpfennig2017). In what follows, we will therefore focus on the use of country fixed effects and newspaper text in forecasting.

Country fixed effects are typically absent in the forecasting literature, which explains why slow-moving, structural variables typically play such an important role in forecasting. Rost, Schneider, and Kleibl (Reference Rost, Schneider and Kleibl2009), for example, use cross-sectional logit regressions on economic and political variables as well as proxies for violations of human rights to predict conflict onset within a five-year window. They find substantial predictive power of their model within this time frame. Goldstone et al. (Reference Goldstone, Bates, Epstein, Gurr, Lustik, Marshall, Ulfelder and Woodward2010) provide predictions of political instability at the country level within a two-year horizon. Their statistical method compares country-years before instability to country-years in the same region that were not followed by onset. Their main finding is that the best predictors of instability are slow-moving variables such as political institutions or infant mortality. Hegre et al. (Reference Hegre, Karlsen, Nygård, Strand and Urdal2013) forecast conflict for the period 2010–2050 using a combination of variables such as population, infant mortality, and education.

News text has been used extensively to predict conflict. Brandt, Freeman, and Schrodt (Reference Brandt, Freeman and Schrodt2011), for example, use the Conflict and Mediation Event Observations (CAMEO) coding scheme in their analysis of news sources when defining conflict events in the Levant. CAMEO uses dictionaries of verbs and actors developed over a decade in several large research projects to identify events and the involved parties in these events.Footnote ³ Many modern applications such as the Global Data on Events, Location and Tone (GDELT) or Integrated Conflict Early Warning System (ICEWS) rely on such coding rules to automatically extract events from text in real time. Most closely related to our article is Ward et al. (Reference Ward, Metternich, Dorff, Gallop, Hollenbach, Schultz and Weschle2013), who use a combination of event data based on ICEWS. Their model has a striking degree of accuracy in predicting civil war incidence (occurrence) and performs well out of sample. A simpler way to forecast conflict with text is proposed by Chadefaux (Reference Chadefaux2014), who relies on keyword counts of a list of predetermined words to construct an index of tension on a weekly basis for the period 1902 to 2001. He uses the constructed tension data to predict onset of conflict weeks before it occurs, and shows that news data can contribute significantly to a standard model. Most recently, Chiba and Gleditsch (Reference Chiba and Gleditsch2017) combined structural and event data to forecast civil war on the monthly level in a logit framework without fixed effects. A lot of their forecasting power comes from constant slow-moving variables such as ethnic fractionalization, GDP per capita, or population. The within- and out-of-sample gains of adding conflict events is relatively modest. This is an interesting contradiction to the findings in Ward et al. (Reference Ward, Metternich, Dorff, Gallop, Hollenbach, Schultz and Weschle2013), which might be explained by use of country fixed effects in the latter.

We add to the forecasting literature in two ways. First, an important conceptual contribution of this project is that we explicitly separate the within-country from the between-country variation before forecasting. We do this by running linear fixed effects regressions and then relying on the within model to forecast. To the best of our knowledge, no paper in the existing literature has tried to separate within and between variation this way.Footnote ⁴ Second, we use a topic model to automatically summarize all news text in a few variables. Our forecasting model can therefore rely on the complexity of the entire newspaper text written on each country and we demonstrate that this has some benefits.

However, our approach comes at the cost of requiring the entire text of articles instead of relying on search queries only. In addition, we try to forecast rare events which implies that we need news sources that are consistently available for decades. We use three newspapers which leave us with a little more than 700,000 articles and prevents us from trying to predict conflict at the quarterly or even monthly level. We therefore see our approach as complementary to Ward et al. (Reference Ward, Metternich, Dorff, Gallop, Hollenbach, Schultz and Weschle2013) and Chadefaux (Reference Chadefaux2014). Ward et al. (Reference Ward, Metternich, Dorff, Gallop, Hollenbach, Schultz and Weschle2013) use event data constructed from more than 30 million news stories while Chadefaux (Reference Chadefaux2014) searches keywords in over 60 million pages of news text.

Quinn et al. (Reference Quinn, Monroe, Colaresi, Crespin and Radev2010) use a topic model, in which documents are assigned only a single topic, to categorize over 100,000 speeches in the U.S. congress. They estimate their topic model of 42 topics to show that topics can be used to analyze democratic agenda dynamics over a long time period. Topics generated by LDA have also been used by Hansen, McMahon, and Prat (Reference Hansen, McMahon and Prat2014) to quantify discussions in the central bank committee of the Bank of England. We contribute to this literature by applying the topic model to newspaper text, which spans large cross-country panels.

The Problem of Forecasting the Timing of Conflict

It is a well-known fact in the conflict literature that rich countries with strong political institutions are less likely to enter conflict than poor countries with weak institutions. A simple way to forecast conflict would therefore be to attribute a higher risk of conflict to poor countries with weak institutions. However, to make conclusions from such a model about how marginal changes in income or institutions affect stability requires a leap of faith. The required leap of faith is that the variation between countries, the between variation, is useful to forecast the variation within countries across time, the within variation. But this is not always true.

As a stylized example, take the standard democracy score from Polity IV (polity2), which is meant to capture the level of democratization in a country. If one runs a regression of civil war onset in year t + 1 on polity2 in year t, one gets a significant, negative relationship. Countries that are more democratic tend to be less at risk of experiencing a conflict the following year. However, if one uses this estimated relationship to look within countries, one gets a falling likelihood of conflict as conflict approaches. In other words, the overall model marks countries as relatively safe right before they experience an outbreak of armed conflict.Footnote ⁵

This example only serves as an illustration. The literature typically adds controls and uses a nonmonotonous relationship between civil war and democracy.Footnote ⁶ But the problem could be relevant in more sophisticated applications as well. For example, a commonly used fragility index, the Fragility Index of the Fund for Peace, relies on many more factors but was still falling in Syria, Libya, and Egypt right before these countries experienced dramatic outbursts of violence in 2011.

A textbook solution to this problem is the use of country fixed effects, but we will show in this section that when forecasting out of sample, the problem goes further. Even if a fixed-effect model is used, the within variation might not contribute to the forecast if the estimated fixed effects are included. Relative to the magnitude of the fixed effect, the within variation is relatively subtle and so the “signal” contained in it is not visible when forecasting out of sample. Accordingly, it might happen to be impossible to predict new developments, that is, onsets in previously peaceful countries or stability in violent countries.

Out-of-Sample Evaluation of Forecasts

We now present a method to evaluate the ability of a model to forecast the onset of conflict. Our starting point is the perspective a policymaker could have. On December 31 of year T, the policymaker is interested in where conflict might break out in the following year. This procedure has two steps; first, the policymaker trains a model using all information available up until year T; second, once the training is completed, the policymaker uses the trained model and data in T to produce forecasts for T + 1.

In the first step, we distinguish between the overall model, which contains the entire model including the estimated fixed effects, and the within model, which disregards the baseline risk contained in the fixed effects. For example, to test whether the democracy index polity2_it contains useful within variation for a forecast we would run a regression of the form

$$\begin{equation*} y_{it+1}=\alpha +\beta _{i}+polity2_{it}\cdot \beta ^{FE}+\varepsilon _{it}, \end{equation*}$$

where y _{it + 1} is conflict onset in t + 1 and β_i are a full set of country dummies.Footnote ⁷ The linear fixed-effects model allows us to separate the between variation contained in the β_i from the useful time variation contained in the within fitted values, $polity2_{it}\cdot \hat{\beta }^{FE}$.

In the second step, the two sets of fitted values in T are used to produce forecasts for T + 1. The fitted values are converted into a binary forecast, that is, negative for peace or positive for onset of conflict, depending on whether the fitted value is above a cutoff c or not. These negatives and positives can then be evaluated with the actual realizations of onset.

These two steps are repeated every year to get an impression of the ability of the model to forecast conflict. In our main application, we let T go from 1995 to 2013 and implement the above steps for all years in between. This means we first predict the onset of conflict in 1996 with information available in 1995. We then predict onset in 1997 with all information available in 1996 and so forth. To evaluate the forecasting power of our model, we collect all out-of-sample predictions and pool them for evaluation. This makes sure that the evaluation is conducted with a common cutoff, which could therefore also be used to forecast an unknown future, based on the performance in the past.

The key trade-off that a policymaker faces when deciding on the right cutoff c is between false negatives, outbreaks of conflict with a negative forecast, and false positives, positive forecasts without onset. If a policymaker chooses a high cutoff, there will be fewer positives and therefore also less false positives. However, the forecast will miss more actual outbreaks of conflict, that is, it will generate more false negatives. If the policymaker instead chooses to warn in all countries, then she will be able to have no false negatives but a lot of false positives.

This trade-off can be shown in receiver operating characteristic (ROC) curves, which visualize the performance of the model for all possible cutoffs c. In the figures containing ROC curves (e.g., Figure 1), we report the true positive rate (TPR) on the y axis. The TPR is given by the formula

$$\begin{equation*} TPR_{c}=\frac{TP_{c}}{FN_{c}+TP_{c}}, \end{equation*}$$

and is a measure of how many false negatives (FN _c) are generated for a given level of true positives (TP _c). We want this measure to be as close to 1 as possible, which would indicate that all conflict onsets have been spotted correctly without missing a single one, that is, with FN _c = 0.

FIGURE 1.

ROC Curves for Onset (X Axis: FPR, Y Axis: TPR)

Notes: Predictions result from a panel estimated as in Equation (2). The variables included for each model as xit are specified in Appendix Table C.1. The within model is the overall model net of country fixed effects as presented in Equation (3).

The false positive rate (FPR) is reported on the x axis of Figure 1. It is given by the formula

$$\begin{equation*} FPR_{c}=\frac{FP_{c}}{FP_{c}+TN_{c}}, \end{equation*}$$

and is a measure of how many false positives (FP _c) are generated for a given level of true negatives (TN _c). We want this to be as low as possible. Optimally, we would want no false warnings, that is, FP _c = 0. The 45-degree line in ROC curves is the benchmark that would be reached by random forecasts. For each ROC curve we also report the area under the curve (AUC).

There are some problems associated with the linear fixed-effects model and a broad academic debate has brought pro- and counterarguments for its adoption (Beck Reference Beck2015). We nonetheless choose the linear model for several reasons. First, because we are forecasting, we are not interested in the precision or even size of the estimated coefficients and, due to our focus on ranking forecasts, we do not mind that the fitted values from the model are not bound between zero and one. Instead, we can simply rely on showing that our model is able to produce useful rankings when forecasting out of sample. This deflates both of the main arguments against the use of a linear model.

The key in our application is, however, that within and between variation are additive in the linear model so that we can use the within fitted values alone in forecasting. This is a crucial difference to the fixed effect logit model, for example. The logit model estimates a set of fixed effects but drops all time series that have no variation in y _{it + 1}, that is, all countries which were always or never in conflict. In addition, it is unclear how one would separate the within variation contained in a logit model to use them separately in forecasting.Footnote ⁸

An Evaluation of Standard Models

We now illustrate that isolating the within variation from the between variation (which together form the overall model) can yield fundamental insights regarding how precisely a model predicts the timing of conflict out of sample. We do this by applying the method described in the previous section to five different models from recent publications in economics and political science shown to explain conflict onset and in some cases even forecast it. We discuss the details of the five models in the Online Appendix C.Footnote ⁹

First, we use a model of rainfall shocks in Africa. Second, we use foreign aid shocks and income shocks interacted with the country’s institutional environment. Third, we use a combination of standard economic and political variables. Fourth, we use a mixture of Integrated Crisis Early Warning System (ICEWS) event data and economic and political data. Fifth, we use a model of conflict word counts based on our articles together with measures of conflict history and political institutions. Finally, we add a model in which we regress conflict on country fixed effects only, that is, we do not even try to predict the timing of conflict but rely instead on the simple logic that onset will occur where it occurred before. This model should, by definition, have no useful within variation.

In all six models, we predict armed conflict and civil war onset as measured by battle-related deaths from the Uppsala Conflict Data Program (UCDP/PRIO).Footnote ¹⁰ This includes all battle-related deaths which took place in armed conflict. The UCDP defines an armed conflict as a contested incompatibility that concerns government and/or territory over which the use of armed force between two parties, of which at least one is the government of a state, has resulted in at least 25 battle-related deaths in one calendar year. It also gives four types of conflict—we include battle-related deaths that occurred during internal and internationalized internal armed conflict.Footnote ¹¹

As discussed in the previous section, we evaluate the forecasting performance of the different models using ROC curves. The blue ROC curves in Figure 1 show the performance of the respective overall model, which is what is typically reported in the forecasting literature. To illustrate, take the forecast of civil war of the “Economics and Politics” overall model in Figure 1(a). It reaches a TPR of around 70% for an FPR of 20% when predicting civil war. This means that the model correctly forecasts 70% of all civil war outbreaks while only marking 20% of peaceful years as under conflict risk. The AUC is almost 0.8, which is comparable to the findings in the literature. When predicting armed conflict in Figure 1(b), the model does a little worse. The blue ROC curve stretches out more in Figure 1(a) than in Figure 1(b), and the reported AUC is higher. Generally, forecasting civil wars in Figure 1(a) seems to be easier than forecasting armed conflict in Figure 1(b).

We then test how much of this forecasting power is maintained when looking at the within variation alone. The performance of this forecast is reported as a red dashed line in Figure 1. Almost all models show a dramatic decline in forecasting power from the overall model to the within model. This indicates that most overall models receive their forecasting power from the between variation. Notable exceptions are the model based on political and economic variables, which always retains at least some forecasting power, and the model based on events, which retains almost all of its forecasting power when predicting armed conflict. The latter finding suggests that using news text can provide a useful source of variation when predicting the timing of conflict.

Note also that the model with only country fixed effects performs as well as most other overall models. This confirms that it is possible to gain fairly good forecasts by assuming that conflict breaks out where it broke out before. However, in many applications, this is not satisfactory and leads to forecasting mistakes if peaceful countries destabilize or violent countries stabilize.

News Text

The first choice we face is the selection of our news sources. Due to their availability over a long time span and international coverage, we focus on three major newspapers published in English, namely the Economist (available from 1975), the NYT (available from 1980), and the WP (available from 1977). From the database LexisNexis, we downloaded all articles dating from January 1975 to December 2015 containing country names (or slight permutations thereof) or capital names in the title.Footnote ¹² In total, we downloaded more than 700,000 articles, of which 174,450 are from the Economist, 363,275 from the NYT, and 185,523 from the WP. The distribution of the number of articles over time is exhibited in Figure 2.

FIGURE 2.

Number of Articles by News Source Over Time

On average, about 120 articles are written on a country in a given year. However, the extent of coverage varies drastically so that we observe between one and more than 5,500 articles in a given year. As a general idea, more populous, richer and more democratic countries are covered more. In addition, coverage increases in and before conflict. On average, a conflict year is covered with about 100 articles more, while a preconflict year is covered with almost 70 articles more than the average year. However, our methodology accounts for changes in coverage by using topic shares, that is, we disregard how much is written on a country and focus instead on what is written on a country, as it facilitates forecasting across countries.

To improve the performance of our machine learning algorithm, we process the raw text of articles of all three newspapers according to standard text-mining procedures.Footnote ¹³ First, we remove a library of common words, which in text mining are referred to as stop words, such as “to” or “that.” Second, we lemmatize and then stem words using the Snowball algorithm, which is an updated version of the algorithm from Porter (Reference Porter1980).Footnote ¹⁴ Lemmatizing group’s distinct forms of the same word into one word, while stemming attempts to harmonize different usages of one word, such that, for example, “running,” “ran,” and “run” all become “run.” However, unlike the example, the outcome does not necessarily represent an English word. This leaves us with more than 5.5 million unique tokens, which are not only single words, but also tokens of sequences of two words and three words, referred to as bigrams and trigrams, respectively. Then, as a final step, we remove overly frequent and rare tokens, and are left with around 0.9 million tokens. This high dimensionality makes it impossible to use the token vectors in standard regressions. Here is where the literature has typically reduced dimensionality by focusing on particular words.

LDA Topic Models

To reduce the high dimensionality of our dataset, we use the LDA to model topics, a method introduced by Blei, Ng, and Jordan (Reference Blei, Ng and Jordan2003). Topics are probability distributions over words. The LDA model in text analysis assumes that each document is a mixture of a small number of topics and that each word’s creation is attributable to one of the document’s topics.

The exercise consists of splitting each article into topics k. One can imagine a journalist writing about a topic will use a combination of words related to that topic. For instance, an article about sports might be more likely to contain words such as “football,” “win,” “fans,” and “game,” whereas an article about a conflict might be more likely to use words such as “violence,” “casualties,” and “soldier.” Through Bayesian learning, the algorithm optimizes the weighted word lists, that is, the topics, to discriminate between articles. For instance, the word “game” might be more of a sports-topic word and, therefore, indicates that an article is on sports. Ultimately, the mixed-membership model represents each document as a set of shares of topics. One could imagine that an article is classified as 70% sports and 30% conflict if a particularly violent soccer match took place. While the number of topics K is prespecified, the content of the topics is not. The topics are identified by looking at which tokens co-occur in articles.

The LDA model requires just three parameter assumptions and can be implemented with a Gibbs sampling technique that we adapt from Phan and Nguyen (Reference Phan and Nguyen2007). In Appendix D, we provide a technical discussion of the LDA and the Gibbs sampler. The parameters to choose are α, β, and the number of topics K. High values of α imply that each article is likely to consist of a mix of many topics. Analogously, a high value of β favours a topic to contain a mixture of most words, whereas low values allow topics to consist of a limited number of prominent words. Concerning α and β, we follow the literature. Our preferred specifications, which we will be using for all of the baseline results, is composed of 15 topics and hyperparameters α = 3.33 and β = 0.01.Footnote ¹⁵