1 Introduction
The plenary sessions of the European Parliament (EP) are one of the most important arenas in which European representatives can air questions, express criticisms and take policy positions to influence European Union (EU) politics. The plenary thus represents the most visible venue where the content and evolution of the policy agenda of the EP can be examined. As a result, understanding how Members of the European Parliament (MEPs) express themselves in plenary, and investigating how the policy agenda of the EP evolves and responds to internal and external stimuli is a fundamentally important undertaking.
In recent years, there has been a concurrent explosion of online records capturing MEP speeches, and the development of data-mining techniques capable of extracting latent patterns in content across sets of these speeches. This allows us for the first time to investigate the plenary agenda of the EP in a holistic and rigorous manner. One approach to tracking the political attention of political figures has been to apply topic modeling algorithms to large corpora of political texts, such as speeches in the U.S. Senate (Quinn et al. Reference Quinn, Monroe, Colaresi, Crespin and Radev2010). These algorithms seek to distill the latent thematic patterns in a corpus of speeches (Blei et al. Reference Blei, Ng and Jordan2003), and can be used to improve the transparency of the policy agenda by providing a macro-level overview of the political debate in a time- and resource-efficient manner.
This study takes up the challenge of extracting latent thematic patterns in political speeches by developing a dynamic topic model to investigate how the plenary agenda of the EP has changed over three parliamentary terms (1999–2014).Footnote 1 The method applies two layers of Non-negative Matrix Factorization (NMF) topic modeling (Lee and Seung Reference Lee and Seung1999) to a corpus of 210,247 speeches from 1,735 MEPs across the 28 EU member states in the EU during that period.
Our proposed topic modeling methodology reveals the breadth of the policy agenda discussed by MEPs in the EP, and the results presented in Section 6 indicate that the agenda has evolved significantly over time. By examining a number of case studies, ranging from the Euro Crisis to EU treaty changes, we identify the relationship between the evolution of these dynamic topics and the exogenous events driving them. By using external data sources, we can also confirm the semantic and construct validity of these topics. In order to explain some of the patterns we observe in speech making, we conclude the study with an exploration of the determinants of MEP speech-making behavior on the detected topics.Footnote 2 Our results relate to the burgeoning literature on political attention, agenda formation, and agenda diversity (Baumgartner et al. Reference Baumgartner, Breunig, Green-Pedersen, Jones, Mortensen, Nuytemans and Walgrave2009; Downs Reference Downs and Peretz1996; Jones and Baumgartner Reference Jones and Baumgartner2005; Jennings et al. Reference Jennings, Bevan, Timmermans, Breeman, Brouard, Chaqués-Bonafont, Green-Pedersen, John, Mortensen and Palau2011).
2 Related Work
Major efforts to track and explain policy agendas have been developed in recent years. Beginning in the early 1990s, the Policy Agendas Project (PAP) and the Comparative Agendas Project (CAP) have tracked policy agendas across different political systems, including the EU. The major claim in both of these projects is that the variation in the attention that political figures pay to different issues across time can be described by a punctuated-equilibrium dynamic, whereby issue attention is stable for long periods of time, but these periods are punctuated by short bursts of increased attention (Baumgartner and Jones Reference Baumgartner and Jones1993). The sudden punctuations in political attention have been explained by factors including the bounded rationality of the political figures involved (Jones Reference Jones1994), (re-)framing of policy choices (Jones and Baumgartner Reference Jones and Baumgartner2005), and the influence of exogenous shocks on political priorities (Jones and Baumgartner Reference Jones and Baumgartner2012; John and Bevan Reference John and Bevan2012), all of which lead to abrupt spikes in issue attention. Despite some conceptual and measurement challenges (Dowding et al. Reference Dowding, Hindmoor and Martin2016), evidence for the existence of this type of agenda dynamic is found across a multitude of political systems (Baumgartner et al. Reference Baumgartner, Breunig, Green-Pedersen, Jones, Mortensen, Nuytemans and Walgrave2009).
In the EU context, and building upon the techniques developed by the PAP/CAP to capture the aforementioned punctuated-equilibrium dynamic, most academic work has focused on the evolving policy agenda of the European Council (Alexandrova, Carammia, and Timmermans Reference Alexandrova, Carammia and Timmermans2012). Similar to what has been found in other contexts, a punctuated-equilibrium dynamic appears to be in play in the European Council, with long periods of agenda stability interrupted with sharp spikes in issue attention. Institutional, contextual and issue-specific factors are found to explain these punctuations. To date, the policy agendas of other EU institutions have been neglected due to the challenges associated with capturing the diverse, diffuse, and multifaceted nature of the policy agendas found in institutions like the Commission, Council of Ministers, and EP.
Despite the fact that policy-agenda dynamics in the EP have to date been underexplored, MEP behavior within the Parliament has been well studied. The most prominent forms of MEP behavior to receive academic attention are plenary speeches and roll-call voting, both of which can be expected to affect the EP policy agenda. Political institutions have been found to shape these forms of MEP behavior. For instance, the formal committee structure of the EP has been shown to provide committee members with strategic advantages due to privileged access to information, and opportunity to shape the EP’s policy choices. This has led MEPs to self-select into committees dealing with salient issues with a view to influencing policy outcomes of interest to them (Bowler and Farrell Reference Bowler and Farrell1995). Within committees, holding roles such as the Chair and Rapporteur have also been shown to affect speech-making and voting behavior (Hix, Noury, and Roland Reference Hix, Noury and Roland2007).
Strict institutional rules also govern the allocation of MEP speaking time in the EP plenary (Proksch and Slapin Reference Proksch and Slapin2010). The total amount of speaking time for any particular issue is limited and divided between time reserved for actors with formal plenary duties such as rapporteurs, and time proportionally divided between party groups based upon their share of MEPs elected. Speaking time limits lead to competition between MEPs, and party-group leaders allocate scarce speaking time between MEPs for maximum impact (Slapin and Proksch Reference Slapin and Proksch2010).
MEP speech content has been shown to reflect latent ideological conflict between MEPs (Slapin and Proksch Reference Slapin and Proksch2010). Using text-analysis techniques based upon word-frequency distributions, these authors demonstrate the correspondence between the content of legislative speeches and other measures of ideological positions found in the literature based upon roll-call votes and expert surveys. To our knowledge, topic models have yet to be applied to the EP plenary.
Topic models aim to discover the latent semantic structure or topics within a text corpus, which can be derived from co-occurrences of words across documents. These models date back to the early work on latent semantic indexing by Deerwester et al. (Reference Deerwester, Dumais, Landauer, Furnas and Harshman1990), which proposed the decomposition of term–document matrixes for this purpose using Singular Value Decomposition. Considerable research on topic modeling has focused on the use of probabilistic methods, where a topic is viewed as a probability distribution over words, with documents being mixtures of topics, thus permitting a topic model to be considered a generative model for documents (Steyvers and Griffiths Reference Steyvers, Griffiths, Landauer, Mcnamara, Dennis and Kintsch2006). The most widely applied probabilistic topic modeling approach is Latent Dirichlet Allocation (LDA) proposed by Blei et al. (Reference Blei, Ng and Jordan2003). Following on from static LDA methods, authors have subsequently developed analogous probabilistic approaches for tracking the evolution of topics over time in a sequentially organized corpus of documents, such as the dynamic topic model (DTM) of Blei and Lafferty (Reference Blei and Lafferty2006).
Alternative algorithms, such as Non-negative Matrix Factorization (NMF) (Lee and Seung Reference Lee and Seung1999), have also been effective in discovering the underlying topics in text corpora (Wang et al. Reference Wang, Cao, Xu and Li2012). NMF is an unsupervised approach for reducing the dimensionality of non-negative matrices, which seeks to decompose the data into factors that are constrained so as to not contain negative values. By modeling each object as the additive combination of a set of non-negative basis vectors, a readily interpretable clustering of the data can be produced without requiring further post-processing. When working with text data, these clusters can be interpreted as topics, where each document is viewed as the additive combination of several overlapping topics. One of the advantages of NMF methods over existing LDA methods is that there are fewer parameter choices involved in the modeling process. Another advantage that is particularly useful for the application presented in this paper is that NMF is capable of identifying niche topics that tend to be under-reported in traditional LDA approaches (O’Callaghan et al. 2015). In the context of the EP plenary, this is an especially useful attribute of a topic model, as discussions are likely to include a mixture of broader general topics and more specific topics with specialized vocabularies, given the technocratic nature of some EU politics.
Topic modeling methods have been adopted in the political science literature to analyze political attention. In settings where politicians have limited time resources to express their views (e.g. plenary sessions in parliaments), they must decide which topics to address. Analyzing what they choose to speak about can thus provide insight into the political priorities of the politicians under consideration. Single-membership topic models, which assume each speech relates to one topic, have successfully been applied to plenary speeches made in the U.S. Senate in order to trace political attention of the Senators over time (Quinn et al. Reference Quinn, Monroe, Colaresi, Crespin and Radev2010). This study found that a rich political agenda emerged, where topics evolved over time in response to both internal and external stimuli.
Bayesian hierarchical topic models have also been used to capture the political priorities expressed in Congressional press releases (Grimmer Reference Grimmer2010), and structural topic models have been used to incorporate text “metadata” in the form of document-level covariates. Such covariates can include information about a document itself such as when and where it was created, alongside information about the creator of the document (Roberts et al. Reference Roberts, Stewart, Tingley, Lucas, Leder-Luis, Gadarian, Albertson and Rand2014).
In conclusion, the current literature provides some interesting insights into the factors that affect MEP speech-making and voting behavior, and the introduction of topic models to the study of political agendas has allowed researchers to consider larger and more complete data sets of political activity across longer time periods than has previously been possible.
3 Methods
In this section we describe a two-layer strategy for applying topic modeling in a non-negative matrix factorization framework to a time-stamped corpus of political speeches. We first describe the application of NMF topic modeling to a single set of speeches from a fixed time period, and then propose a new approach for combining the outputs of topic modeling from successive time periods to detect a set of dynamic topics that span part or all of the duration of the corpus.
3.1 Topic modeling speeches
While work on topic models often involves the use of LDA, NMF can also be applied to textual data to reveal topical structures (Wang et al. Reference Wang, Cao, Xu and Li2012). The ability of NMF to account for how important a word is to a document in a collection of texts, based on weighted term-frequency values, is particularly useful. Specifically, applying a log-based term frequency–inverse document frequency (TF–IDF) weighting factor to the data prior to topic modeling has shown to be advantageous in producing diverse but semantically coherent topics which are less likely to be represented by the same high-frequency terms. This makes NMF suitable when the task is to identify both broad, high-level groups of documents, and niche topics with specialized vocabularies (O’Callaghan et al. Reference O’Callaghan, Greene, Carthy and Cunningham2015). In the context of political speech in parliaments, this is a particularly desirable attribute of the model, as it can differentiate between broad procedural topics relating to the day-to-day running of plenary and more focused discussions on specific policy issues. This claim is demonstrated concretely in the analysis below.
3.1.1 Applying NMF
Given a corpus of
$n$
speeches, we first construct a document–term matrix
$\mathbf{A}\in \mathbb{R}^{n\times m}$
, where
$m$
is the number of unique terms present across all speeches (i.e. the corpus vocabulary). Applying NMF to
$\mathbf{A}$
results in a reduced rank-
$k$
approximation in the form of the product of two non-negative factors
$\mathbf{A}\approx \mathbf{W}\mathbf{H}$
, where the objective is to minimize the reconstruction error between
$\mathbf{A}$
and
$\mathbf{W}\mathbf{H}$
. The rows of the factor
$\mathbf{H}\in \mathbb{R}^{k\times m}$
can be interpreted as
$k$
topics, defined by non-negative weights for each of the
$m$
terms in the corpus vocabulary. Ordering each row provides a topic descriptor, in the form of a ranking of the terms relative to the corresponding topic. Essentially, the ordered row entries of the matrix
$\mathbf{H}$
allow us to identify the most common terms characterizing each topic, thus allowing for substantive interpretation. The columns in the matrix
$\mathbf{W}\in \mathbb{R}^{n\times k}$
provide membership weights for all
$n$
speeches with respect to each of the
$k$
topics. The columns in matrix
$\mathbf{W}$
can be used to associate individual speeches with the topic they are related to, and when we know from metadata what MEP makes a given speech, we can thus capture MEP contributions to a given topic.
NMF algorithms are often initialized with random factors, which can lead to unstable results where the algorithm converges to a variety of local minima of poor quality. To improve the quality of the resulting topics, we generate initial factors using the Non-negative Double Singular Value Decomposition (NNDSVD) initialization approach (Boutsidis and Gallopoulos Reference Boutsidis and Gallopoulos2008).
3.1.2 Parameter selection
A key parameter selection decision in topic modeling pertains to the number of topics
$k$
. Choosing too few topics will produce results that are overly broad, while choosing too many will lead to many small, highly similar topics. One general strategy proposed in the literature has been to compare the topic coherence of topic models generated for different values of
$k$
(Chang et al.
Reference Chang, Gerrish, Wang, Boyd-Graber and Blei2009). A range of such coherence measures exists in the literature, although many of these are specific to LDA. Recently, O’Callaghan et al. (Reference O’Callaghan, Greene, Carthy and Cunningham2015) proposed a general measure, Topic Coherence via Word2Vec (TC-W2V), which evaluates the relatedness of a set of top terms describing a topic. This approach uses the increasingly popular word2vec tool (Mikolov et al.
Reference Mikolov, Chen, Corrado and Dean2013) to compute a set of vector representations for all of the terms in a large corpus. We can assess the extent to which the two corresponding terms share a common meaning or context (e.g. are related to the same topic) by measuring the similarity between pairs of term vectors. Topics with descriptors consisting of highly similar terms, as defined by the similarity between their vectors, should be more semantically coherent.
For the purpose of assessing the coherence of topic models, TC-W2V operates as follows. The coherence of a single topic
$t_{h}$
represented by its
$t$
top-ranked terms is given by the mean pairwise cosine similarity between the
$t$
corresponding term vectors in the word2vec space:
$$\begin{eqnarray}\text{coh}(t_{h})=\frac{1}{\binom{t}{2}}\mathop{\sum }_{j=2}^{t}\mathop{\sum }_{i=1}^{j-1}\cos (wv_{i},wv_{j})\end{eqnarray}$$
where
$v_{i}$
and
$v_{j}$
represent each unique pair of term vectors. An overall score for the coherence of a topic model
$T$
consisting of
$k$
topics is given by the mean of the individual topic-coherence scores:
$$\begin{eqnarray}\text{coh}(T)=\frac{1}{k}\mathop{\sum }_{h=1}^{k}\text{coh}(t_{h}).\end{eqnarray}$$
An appropriate value for
$k$
can be identified by examining a plot of the mean TC-W2V coherence scores for a fixed range [
$k_{min},k_{max}$
] and selecting a value corresponding to the maximum coherence.
3.2 Dynamic topic modeling
3.2.1 Layer 1
When applying clustering to temporal data, authors have often proposed dividing the data into time windows of fixed duration (Sulo et al.
Reference Sulo, Berger-Wolf and Grossman2010). Therefore, following Sulo et al. (Reference Sulo, Berger-Wolf and Grossman2010), we divide the full time-stamped corpus of parliamentary speeches into
$\unicode[STIX]{x1D70F}$
disjoint time windows
$\{T_{1},\ldots ,T_{\unicode[STIX]{x1D70F}}\}$
of equal length. The rationale for the use of disjoint time windows as opposed to processing the full corpus in batch is twofold: (1) we are interested in identifying the agenda of the parliament at individual time points as well as over all time; (2) short-lived topics, appearing only in a small number of time windows, may be obscured by only analyzing the corpus in its entirety or using overlapping time windows. At each time window
$T_{i}$
, we apply NMF with parameter selection based on Equation (2) to all speeches delivered during that window, yielding a window topic model
$M_{i}$
containing
$k_{i}$
window topics. This process produces a set of successive window topic models
$\{M_{1},\ldots ,M_{\unicode[STIX]{x1D70F}}\}$
, which represent the output of the first layer in our proposed methodology.Footnote
3
3.2.2 Layer 2
From the window topic models we construct a new condensed representation of the original corpus, by viewing the rows of each factor
$\mathbf{H}_{i}$
coming from each window topic model as “topic documents”. Each topic document contains non-negative weights indicating the descriptive terms for that window topic. We expect that window topics that come from different windows, but share a common theme, will have similar topic documents. We then construct a topic–term matrix
$\mathbf{B}$
as follows:
(1) Start with an empty matrix
$\mathbf{B}$
.(2) For each window topic model
$M_{i}$
:(a) For each window topic within
$M_{i}$
, select the
$t$
top-ranked terms from the corresponding row vector of the associated NMF factor
$\mathbf{H}$
, set all weights for all other terms in that vector to 0. Add the vector as a new row in
$\mathbf{B}$
.
(3) Once vectors from all topic models have been stacked in this way, remove any columns with only zero values (i.e. terms from the original corpus which did not ever appear in the
$t$
top-ranked terms for any window topics).
The matrix
$\mathbf{B}$
has size
$n^{\prime }\times m^{\prime }$
, where
$n^{\prime }=\sum _{i=1}^{\unicode[STIX]{x1D70F}}k_{i}$
is the total number of “topic documents” and
$m^{\prime }\ll m$
is the subset of terms remaining after Step 3. The use of only the top
$t$
terms in each topic document allows us to implicitly incorporate feature selection into the process. The result is that we include those terms that were highly descriptive in each time window, while excluding those terms that never featured prominently in any window topic. This reduces the computational cost for the second factorization procedure described below.
Having constructed
$\mathbf{B}$
, we now apply a second layer of NMF topic modeling to this matrix to identify
$k^{\prime }$
dynamic topics that potentially span multiple time windows. The process is the same as that outlined previously, where
$\mathbf{B}$
is substituted for the matrix
$\mathbf{A}$
when applying NMF as described in Section 3.1.1. Here the TC-W2V coherence measure is used to detect number of dynamic topics
$k^{\prime }$
. The resulting factors
$\mathbf{B}\approx \mathbf{U}\mathbf{V}$
can be interpreted as follows: the top-ranked terms in each row of
$\mathbf{V}$
provide a description of the dynamic topics; the values in the columns of
$\mathbf{U}$
indicate to what extent each window topic is related to each dynamic topic.
We track the evolution of these topics over time in the following manner. Firstly, we assign each window topic to the dynamic topic for which it has the maximum weight, based on the values in each row in the factor
$\mathbf{U}$
. We define the temporal frequency of a dynamic topic as the number of distinct time windows in which that dynamic topic appears. The set of all speeches related to this dynamic topic across the entire corpus corresponds to the union of the speeches assigned to the individual time-window topics, which are in turn assigned to the dynamic topic.
The resulting outputs of the two-layer topic modeling process are (1) A set of
$\unicode[STIX]{x1D70F}$
window topic models, each containing
$k_{i}$
window topics. These are described using their top
$t$
terms and the set of all associated speeches; (2) A set of
$k^{\prime }$
dynamic topics, each with an associated set of window topics. These are described using their top-
$t$
terms and the set of all associated speeches; and (3) A ranking of every MEPs contributions relative to all window and dynamic topics in the corpus.
Example of 4 window topics, described by lists of top 10 terms, which have been grouped together in a single dynamic topic related to climate change.
Table 1 shows a partial example of a dynamic topic. We observe that, for the four window topics, there is a common theme pertaining to climate change. The evolution of the climate change topic can be seen in the emergence of the terms ‘Copenhagen’, ‘conference’ and ‘summit’ in 2009-Q4 and 2010-Q1, at exactly the time when the Copenhagen climate change summit was underway. Detecting the evolution of topics in this manner is one of the advantages of taking a dynamic approach to capture policy agendas. While the variation across the term lists reflects the evolution of this dynamic topic over the time period (2008-Q4 to 2010-Q1), the considerable number of terms shared between the lists underlines its semantic validity.
4 Data
In August 2014 we retrieved all plenary speeches available on Europarl, the official website of the European Parliament, corresponding to parliamentary activities of MEPs during the 5th–7th terms of the EP.Footnote 4 This resulted in 269,696 unique speeches in 24 languages. While we considered the use of either multi-lingual topic modeling or automated translation of documents, issues with the accuracy and reliability of both strategies lead us to focus on English language speeches in plenary—either from native speakers or translated—which make up the majority of the speeches available on Europarl. A corpus of 210,247 English language speeches was identified in total, representing 77.95% of the original collection. In terms of coverage of speeches from MEPs from the member states, this ranges from 100% for the United Kingdom, through 87% for Germany, down to 66.2% for Romania. However, the most recent state to accede to the EU, Croatia, represents an outlier in the sense that only 2.6% of speeches were available in English at the time of retrieval due to EP speech translation issues.Footnote 5
Following considerable previous work on time-stamped document collections (e.g. Blei and Lafferty (Reference Blei and Lafferty2006)), we subsequently divided the data into a set of sequential non-overlapping slices or time windows – specifically, 60 quarterly windows from 1999-Q3 to 2014-Q2. We select a quarter as the time-window duration to allow for the identification of granular topics, while also ensuring there exists a sufficient number of speeches in each time window to perform topic modeling. Initial experiments performed on shorter durations with small numbers of speeches per window often yielded results with a smaller number of coherent topics. In addition, a quarterly time window is appropriate in order to avoid empty time windows occurring due to the summer recess of the EP.
For each time window
$T_{i}$
we construct a document–term matrix
$\mathbf{A}_{i}$
as follows:
(1) Select all speech transcriptions from window
$T_{i}$
, and remove all header and footer lines.(2) Find all unigram tokens in each speech, through standard case conversion, tokenization, and lemmatization.
(3) Remove short tokens with
${<}3$
characters, and tokens corresponding to generic stop words (e.g. “are”, “the”), parliamentary-specific stop words (e.g. “adjourn”, “comment”), and names of politicians.(4) Remove tokens occurring in
${<}5$
speeches.(5) Construct matrix
$\mathbf{A}_{t}$
, based on the remaining tokens. Apply TF–IDF term weighting and document length normalization.
The resulting time-window data sets range in size from 679 speeches in 2004-Q3 to 9,151 speeches in 2011-Q4, with an average of 4,811 terms per data set.
5 Assessing the Coherence of LDA and NMF Topic Models: A Baseline Comparison
As noted previously, probabilistic methods such as LDA have been widely applied for topic modeling, although recent work has shown that factorization-based algorithms such as NMF are effective in identifying niche topics with more specific vocabularies (O’Callaghan et al. Reference O’Callaghan, Greene, Carthy and Cunningham2015). In the context of EU politics where technocratic issues are often discussed, being able to detect these niche topics should be an advantage. To illustrate this idea, Table 2 shows the top-5 terms associated with a topic relating to the Euro Crisis for a selection of consecutive time window as produced by competing NMF and LDA approaches. Terms in bold are unique to a topic produced by a given approach, while terms in italics are found in both sets of terms. As can be seen, the vocabulary produced by NMF to describe the Euro Crisis are much more rich and varied compared to those produced by LDA. If we were interested in the content and dynamics of the debate surrounding the Euro Crisis, then NMF appears to produce a more informative and time-variant picture of debate evolution.
NMF and LDA—Euro Crisis Topic Top-5 terms. Bold terms unique to one set of results, italic terms shared.
In order to more systematically compare NMF and LDA topic models, we apply topic-coherence methods (Stevens et al.
Reference Stevens, Kegelmeyer, Andrzejewski and Buttler2012) to assess model performance. Topic coherence refers to the level of semantic similarity between the top terms used to represent a topic (i.e. the topic descriptors). We apply these measures to each of our
$60$
time-window data sets described in Section 4 for different numbers of topics
$k\in [10,50]$
. We apply NMF as described in Section 3.1.1 and use LDA as implemented in the MALLET toolkit (McCallum Reference McCallum2002), with hyperparameter values
$\unicode[STIX]{x1D6FC}=0.01$
and
$\unicode[STIX]{x1D6FD}=50/k$
as recommended by Steyvers and Griffiths (Reference Steyvers, Griffiths, Landauer, Mcnamara, Dennis and Kintsch2006). In our experiments, we calculate topic coherence using two different measures to evaluate the top 10 terms for each topic in all 300 models produced by each algorithm (i.e. 60 data sets for 5 different values of
$k$
). Following Röder et al. (Reference Röder, Both and Hinneburg2015), we report the median value to provide a more robust score summarizing each model.
$C_{v}$
at
$k=10$
topics.
$C_{v}$
at
$k=50$
topics.
Firstly, we compare NMF and LDA using the
$C_{v}$
topic-coherence measure which was identified by Röder et al. (Reference Röder, Both and Hinneburg2015) as being particularly appropriate for evaluating topic quality, based on a large empirical comparison of different coherence measures. The
$C_{v}$
measure retrieves co-occurrence counts from a background corpus for a set of topic terms using a sliding window approach. These counts are then used to calculate the normalized point-wise mutual information (NPMI) for all pairs of terms within a topic descriptor. The intuition behind such an approach is that more coherent topics will have topic descriptors that co-occur more often together across the corpus. As our background corpus for each time-window data set, we use the full set of 210,247 English EP speeches. Results for
$k=10$
and
$k=50$
topics for NMF are shown in Figures 1 and 2. We observed that NMF achieves higher topic-coherence scores across all of the time-window data sets for all values of
$k$
. Inspecting the results suggests that this is largely due to the ability of NMF to uncover more niche and specific topics on the data, compared to the more broad and ultimately less semantically coherent topics extracted by LDA (i.e. the term descriptors are not distinctive).
TC-W2V at
$k=10$
topics.
TC-W2V at
$k=50$
topics.
To further investigate the differing topic coherence resulting from LDA and NMF models, we repeated the process using the TC-W2V coherence measure described previously in Section 3.1.2. As described in Section 3.1.2, this measure is based on a Word2Vec model, which provides a computationally efficient and effective method for quantifying the semantic relatedness of terms in a large corpus (Mikolov et al.
Reference Mikolov, Chen, Corrado and Dean2013). Again as our reference corpus we use the full corpus of English speeches. Representative results are shown in Figures 3 and 4. As with the
$C_{v}$
measure, we observe that NMF consistently achieves high coherence scores, with a higher score than LDA in the case of 94.7% of all 300 experiments. We also observe that the TC-W2V measure is generally more sensitive to changes in the top terms used to represent topics, highlighting time windows where topics have a greater or lesser level of coherence for both algorithms. These results provide our rationale for the use of NMF in the remainder of this paper.
6 Experimental Results
6.1 Experimental setup
After preprocessing the data, the first task was to identify
$k$
, the number of topics in each window. To do this, we applied NMF with parameter selection as described in Section 3.1. Given the relatively specialized vocabulary used in EP debates, when building the word2vec space for parameter selection, we used the complete set of English language speeches as our background corpus. We used the same word2vec settings and number of top terms per topic (
$t=10$
) as described in O’Callaghan et al. (Reference O’Callaghan, Greene, Carthy and Cunningham2015). At each time window, we generated window topic models containing
$k\in [10,25]$
topics, and then selected the value
$k$
that produced the highest mean TC-W2V coherence score (Equation (2)). The illustration of the number of topics per window in Figure 5 shows that there is considerable variation in the number of topics detected for each window, which does not correlate with the number of speeches per quarter (Pearson correlation 0.006). This suggests our results are not driven by the volume of speeches, but rather variation in topics being discussed across different windows.
Number of window topics identified from 1999-Q3 (#1) to 2014-Q2 (#60).
Plot of mean TC-W2V topic-coherence scores for different values for the number dynamic topics of
$k^{\prime }$
, across a candidate range
$[25,90]$
.
The process above yielded 1,017 window topics across the 60 time windows. We subsequently applied dynamic topic modeling as described in Section 3.2. For the number of terms
$t$
representing each window topic, we experimented with values from 10 to the entire number of terms present in a time window. However, values
$t>20$
did not result in significantly different dynamic topics. Therefore, to minimize the dimensionality of the data, we selected
$t=20$
. This yielded a matrix of 1,017 window topics represented by 2,710 distinct terms.
Our next task was to identify a value for the parameter
$k$
for the dynamic part of the model, i.e.- the number of dynamic topics in the corpus. To do this, we calculated TC-W2V coherence scores for a set of topic models with a range
$k^{\prime }\in [25,90]$
and then compared these coherence scores to identify the appropriate parameter value. The resulting plot (see Figure 6) indicated a maximal value at
$k^{\prime }=57$
, although a number of close peaks exist in the range [62,80]. When we manually inspected the results of the most coherent topic models for these values of
$k$
’, they were highly similar in terms of the topics detected, with minor variations corresponding to merges or splits of strongly related topics.
6.2 Case studies
In order to investigate the construct validity of our topics, we focus on three case studies to demonstrate how our topic modeling strategy captures variation in the EP policy agenda over time.Footnote 6
6.2.1 Financial/Euro Crisis
Our first case study, illustrated in Figure 7, relates to two topics covering the financial and Euro Crisis, respectively. This is an interesting case study, as the initial financial crisis peaked in 2008, and the Euro Crisis that followed has gone through a number of phases starting in 2009. These events can be thought of as exogenous shocks to the policy agenda, and their exogenous nature provides a way to externally validate the dynamic topic modeling approach in use here. Figure 7 demonstrates a number of distinct peaks in MEP attention to both the financial crisis topic and the Euro Crisis topic. Attention to the financial crisis starts to rise in 2008-Q3 and initially peaks in 2008-Q4 (Point A in Figure 7), corresponding to the collapse of the Lehman Brothers bank (15/9/2008). The other peaks in attention in Figure 7 correspond to important events in the Euro Crisis. Point B corresponds to the revelations about underreporting of Greek debt in October 2010, Point C to the Irish bailout (November 2010), and Point D to Mario Draghi’s statement that the ECB was “ready to do whatever it takes to preserve the euro” (July 2012). Draghi’s statement temporarily at least reassured markets, thus explaining why fewer speeches relating to this topic are observed after Point D. In effect, it appears that both the financial and Euro crises had the effect of punctuating a rather low-level equilibrium of attention to issues relating to the common currency and financial regulation that existed before 2008 and 2010, respectively.
“Financial & Euro crises” dynamic topics.
6.2.2 Treaty reform
Our second case study (Figure 8) relates to EU Treaty reforms. This topic is of interest, because one would expect that MEP attention to the topic varies over time, as Treaty revisions are not common. For example, the Nice Treaty was agreed upon in 2001 and put to a referendum in Ireland in June 2001. The ‘No’ vote that resulted from this referendum accounts for Point A in Figure 8. Similarly, Point B in Figure 8 corresponds to the October 2003 Intergovernmental Conference negotiating the Constitutional Treaty. Point C indicates the date the Enlargement Treaty was signed in May 2004. MEP attention relating to the Lisbon Treaty peaks when it was signed (Point D), and when the Irish rejected the Treaty in June 2008 (Point E). Point F corresponds to the second Irish referendum approving Lisbon in October 2009. If we view this variation in attention to Treaties in light of punctuated-equilibrium theory, it would appear that equilibrium levels of attention to Treaty changes in the EP is low, but this equilibrium is disturbed with spikes in attention when major exogenous events relating to Treaty change occur. The EP appears to be reactive rather than proactive in this regard (attention spikes after an event), which is not surprising given its limited formal role in Treaty negotiations.
“Treaty changes & referenda” dynamic topic.
“Fisheries” dynamic topic.
6.2.3 Fisheries policy
Our final case study relates to fisheries policy. Fisheries is an interesting policy-agenda item for the dynamic topic modeling approach to detect, because it relates to the day-to-day functioning of the EU as a fisheries industry regulator, rather than more headline-making policies and events already discussed. As a result one would expect a more constant level of attention to this agenda item with fewer punctuations. Figure 9 demonstrates the prevalence of the fisheries topic over time. As can be seen, MEPs pay a reasonably stable level of attention to fisheries between 2000 and 2010. This trend is interrupted in 2010, when MEP attention to fisheries increases. This corresponds to the Commission launching a public consultation on reforming EU fisheries policy in 2009, the results of which were presented to the EP in April 2010. Point A corresponds to the launch of this working document, while Point B corresponds with Commissioner Maria Damanaki introducing a set of legislative proposals designed to reform the common fisheries policy in a speech to the EP in July 2011. This is highly consistent with the patterns in agenda change described in punctuated-equilibrium theory.
In general, the fact that the variation over time that we observe in MEP attention to these case-study topics appears to be driven by exogenous events provides a form of construct validity for our topic modeling approach, and support for the idea that political agendas are relatively stable, but experience punctuations due to exogenous events (John and Bevan Reference John and Bevan2012; Jones and Baumgartner Reference Jones and Baumgartner2012).
6.3 Explaining MEP speech counts
We now focus our attention on the 7th EP term that sat between 2009 and 2014, as a set of interesting covariates are available at the MEP level that can help us explain MEP contributions to a given agenda item. We aim to explore the determinants of MEP topic contributions with reference to existing theories in the literature that show MEP ideology, party membership and institutional structure affect other forms of MEP behavior (including propensity to speak in plenary, rebelling against party principals, and report writing). Our first dependent variable is constructed directly from the dynamic topic model level-2 speech–topic weight matrix, in which each MEP speech can relate to multiple topics. The variable is simply the sum of all weights per MEP for each topic across the entire EP7 term, and thus captures the relative contribution of each MEP to each topic if one assumes that speeches can relate to multiple topics. The skewed and continuous nature of the variable being examined implies that a generalized linear model from the Gaussian family with a log-link function is appropriate for our analysis.
In constructing our second dependent variable, we assume that each speech belongs to one topic alone by allocating each speech to a topic based on the maximum topic weight observed for that speech.Footnote 7 Substantively it is reasonable to assume single-topic memberships for each speech, given that MEP speaking time is limited, thus focusing MEP attention on particular agenda items rather than allowing speeches to address multiple issues. Making such an assumption has the advantage of providing a more readily interpretable analysis, as the resulting variable captures each MEP’s contribution to each topic in terms of a speech count rather than a sum of NMF weights. We employ a negative binomial regression model suitable for analyzing count data with overdispersion on the resulting variable (Cameron and Trivedi Reference Cameron and Trivedi2013). In both Model 1 and Model 2 we cluster standard errors by MEP.Footnote 8
In order to explain the variation observed in our dependent variables, we include independent variables relating to MEP’s ideology, voting behavior, and the institutional structures in which they find themselves embedded within, as these variables have been found to be relevant to speech-making behavior in the EP (Proksch and Slapin Reference Proksch and Slapin2014). We account for the left–right ideological position of an MEP’s national party (as a proxy for MEP ideology) using data from Scully et al. (Reference Scully, Hix and Farrell2012). Following Proksch and Slapin (Reference Proksch and Slapin2014), we also include a measure of how often MEPs vote against their European party group in favor of their national party and vice versa. The idea behind including these variables is that MEPs rebelling against one party affiliation in favor of another will either try to explain such behavior in their speeches thus increasing the observed speech count, or hide their behavior by making no speeches, thus decreasing the observed count. These data were taken from an updated version of the Hix et al. (Reference Hix, Noury and Roland2006) data set provided by those authors. In order to capture an MEP’s committee positions we include dummies for committee membership, chairs, and Rapporteurs in committees that are directly related to a given topic. Committees were manually matched with topics to achieve this. We control for whether or not an MEP serves in the EP leadership. Controls are also included for the total number of speeches made by an MEP and the percentage of MEP speeches that are available in English as these are liable to affect the observed MEP speech count. Finally, we also include dummy variables to control for an MEP’s country of origin, EP party-group membership, and the topic on which they are speaking. All institutional and control variables were collected from the EP legislative observatory.
Plot of coefficients for regression models.
Plot of coefficients for regression models.
The regression results presented in Figures 10 and 11 provide further validation for our topic modeling approach. Model 1 provides strong evidence that voting behavior and institutional position are the main drivers of MEP topic attention. MEPs that tend to defect from the European party group to vote with their national delegation tend to contribute to topics much less than those that are loyal to their party group, suggesting incentives to hide defection voting by avoiding making speeches when such votes are cast. MEPs with EP leadership roles are found to speak on topics more often, while the committee system is also a significant driver of MEP attention, with committee chairs and Rapporteurs making significantly more speeches on topics relevant to their official committee roles.
Turning to Model 2, we see mostly similar substantive results, but this time we are presented with more directly interpretable odds ratios (exponentiated model coefficients) that capture the effects of our chosen independent variables on the odds of observing a speech relating to a given topic. The results suggest that the odds of MEPs who vote against their EP party groups in favor of their national party making a speech on a given topic are reduced by a factor of 0.86. The results also further reinforce the finding that MEP positions within the EP committee system impact upon how much attention they pay to a particular topic. When an MEP holds a committee chair, Rapporteurship, or committee membership relevant to a particular topic, the odds that said MEP will make a speech on that topic increase by a factor of 1.169, 2.283, and 2.126, respectively. These results reinforce the idea that the committee system fundamentally shapes speech-making activities and the policy agenda of the EP plenary. The variable accounting for EP leadership positions is no longer significant in the count model, probably due to the fact that speeches belong to a single topic in this model rather than having multiple memberships, thus reducing the impact of EP leaders.
7 Conclusions
In this study, we propose a new two-layer NMF methodology for identifying topics in large political speech corpora over time, designed to identify both niche topics with specific and specialized vocabularies, and broader topics with more general vocabularies. Firstly, we demonstrate that topic modeling via NMF can lead to the identification of topics that are semantically more coherent in a corpus of political speeches, when compared with a probabilistic method such as LDA. Subsequently, we apply this method to a new corpus of all
${\approx}210k$
English language plenary speeches from the EP between 1999 and 2014. In terms of providing substantive insight into EP politics, the topic modeling method allows us to unveil the political agenda of the EP, and the manner in which this agenda evolves over the time period considered. By considering three distinct case studies, we demonstrate the distinctions that can be drawn between the day-to-day political work of the EP in policy areas such as fisheries on the one hand, and the manner in which exogenous events such as economic crises and failed treaty referenda can give rise to new topics of discussion between MEPs on the other. With the EP agenda in hand, we explore the determinants of MEP attention to particular topics in the 7th sitting of the EP. We demonstrate how MEP voting behavior and institutional position affect whether or not they choose to contribute to an agenda topic.
The insights provided by the dynamic topic modeling approach presented here demonstrate how these methods can uncover latent dynamics in MEP speech-making activities and supply new insights into how the EU functions as a political system. Much remains to be explored in terms of the patterns in political attention that emerge from our topic modeling approach. For instance, one would expect that political attention might well translate into influence over policy outcomes decided upon in the EP. Tracing influence to date has been difficult, as a macro-level picture of where and on what topics MEP attention lays has been unavailable. Linking political attention to political outcomes would help to unveil who gets what and when in EU politics, which is a central concern for a political system often criticized for lacking democratic legitimacy.
Outside the European context, our method can be applied to any political situation in which policy agendas are captured in text form. Plenary debates in other political systems are a prime candidate for analysis, but legislative agendas, media agendas and other contexts where large corpora of text exist and are available digitally also lend themselves to analysis using our method.
Supplementary material
For supplementary material accompanying this paper, please visit https://doi.org/10.1017/pan.2016.7.
