Responding to a recent review article in this journal,Footnote ¹ Selck, Yardımcı and Kathan argue that in order to improve our understanding of decision-making processes in the EU Council, we need to improve the quality of the data we collect and analyse.Footnote ² The authors also suggest that instead of further analysis of existing data sets, we should expand our horizons to search out new forms of information on the Council. On these two points we are in complete agreement; indeed this argument forms the crux of our original review article, where we discuss the potential exploitation of increasingly abundant primary data made available by the Council. While we acknowledge Selck et al.'s argument that the information contained within these primary sources is sometimes limited, we question their suggestion that newspaper analysis offers a potential improvement. In this short note we briefly develop three points. First, the data that it is possible to derive from newspapers simply does not meet the exacting requirements of the bargaining models that Selck et al. argue it can improve, even in combination with alternative data sources. Second, given the nature of Council decision-making and issues of bias that affect all research using newspaper coverage, we question the assertion that ‘newspapers [are a] proxy of what really happened during the bargaining process.’Footnote ³ Third, Selck et al. appear unaware of the methodological challenges involved in generating requisite data from newspapers.

Although sympathetic to Selck et al.'s proposition that ‘the data is out there; all we have to do is use it’,Footnote ⁴ we are not convinced that newspapers are appropriate for the task. Selck et al. argue that ‘properly applied’ newspaper analysis can provide ‘the type of valid and reliable data that is needed to improve model-testing’.Footnote ⁵ We question whether this is the case, given that the various model types discussed in our review article require specific numerical estimates of actor positions at various stages of the decision-making process, estimates of the salience that actors attach to each issue, actors' relative capabilities to influence negotiations etc.Footnote ⁶ Furthermore, evaluating the predictive power of decision-making models also requires numerical estimates of the location of the status quo and the bargaining outcome.Footnote ⁷ Although prior research demonstrates that it is possible to derive numerical estimates of policy positions from political texts,Footnote ⁸ we are unaware of any research that uses newspaper sources to generate detailed numerical estimates on actor positions and the other requisite variables. Although Selck et al. imply that newspaper data represents a less costly and more sustainable source of information than the current reliance on experts, it is unclear by which method they propose to ‘improve data quality’ using newspapers.Footnote ⁹ Our original argument about the potential utility of Council documentation was based on the observation that preliminary empirical applications demonstrate how it can deliver appropriate and reliable data comparable to expert estimates.Footnote ¹⁰ The same cannot be said for newspaper analysis.

Our second concern relates to the extent to which newspaper coverage reveals what ‘really happened’ in the bargaining process, particularly as Council negotiations are notoriously secretive and specialized. The claim raises the issue of selection bias and reliability that affects all research using media coverage as a data source. Selck et al. advocate collecting newspaper articles from multiple sources, claiming that any biases in reporting would be offset, but one factor that would not be offset is the so-called newsworthiness of an event.Footnote ¹¹ Several variables influence the probability that an event is covered, the extent to which it is covered over time and the degree of detail that is reported. Legislation with far-reaching consequences, particularly when it incurs high levels of contestation among the more influential states, may indeed receive comparatively extensive coverage.Footnote ¹² However, the majority of EU legislation does not fit this description, which makes us wonder how much newspapers reveal about the ‘day-to-day decision-making process' of the EU,Footnote ¹³ particularly with the Council's well-known predilection for masking conflict between its members. Notwithstanding Selck et al.'s observation that the Financial Times published over 300 articles on the crucial and strongly contested EU Services Directive (over a period that we calculate to be at least five years), we suggest that media coverage of ‘day-to-day decision-making’ in the EU that Selck et al. want to illuminate is too limited to make anything but the most tentative and rudimentary inferences.Footnote ¹⁴

The final issue to mention here concerns methods. Although Selck et al. mention ‘electronic content analysis’Footnote ¹⁵ in passing, there is no indication of how the authors propose to generate the required quantitative data from the newspaper sources they promote. And though they provide an example of analysing a text manually, which they claim ‘reveals substantial pieces of information on the bargaining situation’,Footnote ¹⁶ this leaves us none the wiser as to how to apply extant computer-assisted content analysis (CCA) techniques, or whether indeed the authors have developed an alternative method. Surveying the techniques currently available, we find several that may be applicable. Wordscores and Wordfish are well-known CCA methods that are able to generate positional estimates of the author of the text.Footnote ¹⁷ These methods are thus usually employed to analyse political speeches or party manifestos, where the de facto author – a politician or party – is clearly identified. A newspaper article, on the other hand, does not generally explicate the journalist's policy preference, but instead reports the interactions of political actors. For this reason, event data research, which typically focuses on conflict and provides measures of the type and intensity of actor interactions, is one of the few techniques for which newspaper articles are well suited, but is of limited use in the context of bargaining and decision-making models. Although promising advances in the field of semantic network analysis has successfully yielded estimates of party policy positions and salience during recent Dutch elections,Footnote ¹⁸ estimating actor positions from non-Dutch-language texts is currently out of reach for existing software.

In areas where textual data appropriate to the research purpose is available in sufficient quantities, CCA can deliver reliable quantitative data. But in the context of improving models of decision-making in the EU Council, newspapers contain insufficient information and current methods are not developed enough to generate the specific data we need. Nonetheless, we concluded our original article by noting the timeliness of exploring new sources of information on the decision-making processes of the EU Council. To this end, Selck et al.'s contribution is to be encouraged. And while we have voiced scepticism about their projected use of newspaper sources, we look forward to future empirical applications where Selck and colleagues show our scepticism to be misplaced.