MEASURING AUTHORITY AND CONSTRAINT

Measures of authority and constraint in Franchino (Reference Franchino2004, Reference Franchino2007) are calculated using delegation and constraint ratios as defined by Epstein and O’Halloran (Reference Epstein and O’Halloran1999). In both the EU and US cases, these ratios are combined to create an index of discretion. To capture the identity of agents (authority), the delegation ratio, Δ_i, for law i represents the number of provisions delegating authority, D _i, to national administrations or to the EC divided by the total number of provisions in the law P _i:

(1)$${\Delta _i} = {{{D_i}} \over {{P_i}}}.$$

A constraint ratio captures restraint on the authority of national administrations or the EC to implementing policy. This ratio is constructed by dividing the total number of restraint categories used in law i, C _i, by the possible number of categories of restraint, R _i.^⊘ (NB: A supplementary materials document covers a variety of technical aspects of our method and ⊘ indicates specific points on which it provides helpful details.)

(2)$${{\cal C}_i} = {{{C_i}} \over {{R_i}}}.$$

A discretion index δ _i, is created by subtracting the product of (1) and (2) from (1) (Franchino Reference Franchino2004, 283):

(3)$${\delta _i} = {\Delta _i} - \left[ {{{\cal C}_i}\, \times\, {\Delta _i}} \right].$$

Essentially, the authority granted to national administrations or the EC is down-weighted by the constraints placed on that authority in an enabling law. Beyond the EU, the foregoing ratios and index are used in a variety of contexts (e.g., Lavertu Reference Lavertu2012; Clouser McCann Reference Clouser McCann2016).

DATA

Our data include the full text of all 158 pieces of “major legislation” described in Franchino (Reference Franchino2004, Reference Franchino2007). Each law was divided into its constituent provisions, effectively articles and sub-articles, which yielded a total of 8,417 provisions for all 158 pieces of legislation. Using Franchino’s original codebook containing coding instructions for each law (Figure 1 provides a sample), each of the provisions was prepared for training with the ML algorithm by creating binary variables indicating whether the provision delegated authority to or imposed constraints on national administrations or the EC.

FIGURE 1.

Sample of Franchino’s Coding Guide for One Piece of Legislation

Notes: Source: Codebook (“Textual Analysis of Main Acts”) Available at http://users2.unimi.it/fabiofranchino/

Of these 158 pieces of legislation, five were available only in the French language and were translated into English using the Google Translation API^{Footnote 4} and included in the training data.^{Footnote 5⊘} An additional 59,423 laws enacted between 1958–2017 were collected from the EUR-Lex database (https://eur-lex.europa.eu/homepage.html).

PHASE I RESULTS

Prior to analysis, the text of each provision for training was pre-processed using the quanteda package in R (Benoit et al. Reference Benoit, Watanabe, Wang, Nulty, Obeng, Mller and Matsuo2018). Provisions were tokenized into unigrams and bigrams; special characters, punctuation, and stopwords were removed; and each term was stemmed. The cleaned, tokenized texts were then transformed into a document-feature matrix (DFM) resulting in a very sparse 8,417 × 79,088 DFM. Because tree-based methods are sensitive to poorly predictive terms, automatic sparsity reduction using a 99% threshold was imposed on the DFM, yielding an 8,417 × 1,278 DFM used for algorithm training and performance assessment. Final models were selected using a process of hyperparameter tuning with five-fold cross-validation. As Table 2 demonstrates there were significant class imbalances in the training data. These were handled via a process of re-weighting positive labels using a positive predictive-weight tuning parameter, namely, the fraction of the negative over positive examples in the training data^⊘.

TABLE 2.

Performance Metrics for Authority and Constraint Classifiers Ordered by F ₁ Score With Training Fraction of Positive Class

Table 2 contains performance metrics for each of the classifiers. Figures 2a and 2b contain information about term importance for the EC and national administration authority classifiers. The overall quality of each classifier is captured by the F ₁ statistic, which combines precision and recall (sensitivity) in a single metric ranging from 0–1. Recall refers to classifier performance in detecting authority and constraint, while precision refers to its ability to distinguish between correctly and incorrectly labeled provisions. The precision and recall statistics in Table 2 suggest that our trained classifiers perform well for detecting authority and some categories of constraint (high recall), but that they may tend to overestimate constraint (low precision for some constraint categories). Figures 3a, 3b and 4a, 4b plot predicted versus hand-coded delegation ratios averaged over each year in the training sample, 1958–93 using 50% out of sample predictions. Both show that our classifiers accurately track aggregate trends in delegation and constraint over time.

FIGURE 2.

Terms Distinguishing Provisions Which Contain (a) Delegation of Authority to the EC and (b) Delegation Of Authority to National Administrations

FIGURE 3.

GBT Predicted versus Hand-Coded (a) Delegation and (b) Constraint Ratios for National Administrations Aggregated Annually, 1958–92

FIGURE 4.

GBT Predicted versus Hand-Coded (a) Delegation and (b) Constraint Ratios for the EC Aggregated Annually, 1958–92

Figures 2a and 2b contain examples of information about the importance of terms in identifying provisions delegating authority as an example.^⊘ The most informative terms for identifying EC delegation (including “mission,” “procedure,” “treaty”), and those for national administration delegation (including “state,” “may,” “member”) make substantive sense. The former relate to formal mechanisms for authority codified in EU law, while the latter suggest permission.^⊘

PHASE II RESULTS

Because provisions are difficult to parse for automated text analysis, the unit of analysis for our population of 59,423 laws is the article, which is a clearly demarcated section of EU directives and regulations addressing related topics.^⊘ Although multiple provisions can be nested within an article, the cleaner identification of the unit and the additional information for the classifier to process are important justifications for our choice.

We aggregate classifier predictions from articles to compute a delegation ratio $( {{{\hat{\Delta }}_i}} )$, a constraint ratio $( {{{\hat{{\cal C}}}_i}} )$ and a discretion index $( {{{\hat{\delta }}_i}} )$ for each of the EU laws collected. This involved three steps. First, each of the EU laws collected were parsed using NLP methods, resulting in 404,852 articles. Second, articles were cleaned and transformed into a DFM in a manner identical to that discussed in Phase I. Finally, each of the trained classifiers were applied to the articles, and each received a binary label indicating whether it contained content delegating authority to or imposing any category of constraint on national administrations or the EC. Estimates for each of the aforementioned quantities were then aggregated for each law.

Descriptives

Our trained classifiers labeled provisions from all directives and regulations available in the EUR-Lex database between 1958–2017. In the descriptives below, we focus on the years 1970–2017 given both the greater production of legislation and the available selection of laws in EUR-Lex. Figure 5 shows the growth of law availability after 1970. By 1980, the annual number of laws in our sample is above 1,000, on average.

FIGURE 5.

Annual Number of EU Regulations and Directives Analyzed Between 1958–2017

Discretion indices, delegation and constraint ratios for each law were averaged over each year to assess patterns over time. Figure 6a shows clear patterns of declines in member state discretion as the EU expanded. Of note are two periods in which large drops in discretion to national administrations occur around important events. The first was during enlargement in 1973, which added the UK, Ireland, and Denmark to the bloc. The second followed the introduction of fixed exchange rates in 1999 as the first stage in introducing the Euro.

FIGURE 6.

Average Yearly Discretion Indices $\left( {\hat{\delta }} \right)$ for (a) National Administrations and (b) the EC Within all Collected EU Legislation Between 1970–2017

Discontinuity Tests for Out-Of-Sample Consistency

A challenge common to many supervised ML problems is the issue of the validity of predictions outside the training sample. As noted, the hand-coded data available for training come from a sample of 158 “major” laws over 25 fewer years than our dataset covers. The principal concern in our case is that our measures of authority, constraint and discretion may be influenced by structural shifts in the language of EU lawmaking. To address this issue, we introduce a discontinuity test using conditional probabilities estimated by each of the GBT classifiers during the training process. Classification decisions at the provision level were ultimately made on the basis of whether the estimated probability of identifying authority (ℙ[D _j|X _j]) or constraint (ℙ[C _kj|X _j]) was above a 50% threshold based upon the words or terms in each provision X _j. All provisions above this threshold were classified as either delegating authority or imposing one or more of the constraint types. For these provisions, then, the predicted probabilities ℙ[D _j = 1|X _j] and ℙ[C _kj = 1|X _j] provide information regarding about how certain or confident the classifier was for each of the provisions classified.^⊘

Major changes in the structure of the language of delegation would be reflected by shifts in classifier certainty over time as measured by these conditional probabilities. We are particularly concerned about shifts in language in the out-of-sample years (1994–2017), which can threaten the validity of our estimates. Our discontinuity test leverages average differences in these certainty estimates with the last in-sample year, 1993, as a cutpoint. The classifier is said to “pass” our test if the average delegation and constraint certainty estimates are approximately equal around a window of the cutpoint, c. For example, the test in regard to identifying authority is as follows:

$${\mathbb{E}}{\left[ {{\mathbb{P}}\left( {{D_j} = 1|{X_j}} \right)} \right]_{c - \epsilon }} \,\approx \,{\mathbb{E}}{\left[ {{\mathbb{P}}\left( {{D_j} = 1|{X_j}} \right)} \right]_{c + \epsilon }}.$$

Passing this test implies that no significant linguistic changes occurred in the out of sample years. Because we estimate a total of nine classifiers (one regarding authority and eight regarding constraints), we conduct a total of nine discontinuity tests. Our classifiers pass all nine tests.^⊘ To visualize the results, Figure 7 plots the discontinuity test for the national administration authority classifier.

FIGURE 7.

Discontinuity Test for National Administration Authority Classifier

Notes: Bins are mean values of ℙ(D _j = 1|X _j). Local linear regressions estimated on each side of the cutpoint using the Imbens–Kalyanaraman bandwidth to identify discontinuity in ${\mathbb{E}}{\left\hbox[} {{\mathbb{P}}{\hbox\left(} {\bi {{D_j}}\ {\hbox=}\ {\rm1}|{X_j}} {\right\hbox)} {\right\hbox]}$

We next use our estimated discretion index $\left( {\hat{\delta }} \right)$ in a replication exercise to check its face validity.

REPLICATING SUBSTANTIVE RESULTS OUT-OF-SAMPLE

Franchino (Reference Franchino2007) tests a hypothesis regarding the relationship between the decision rule for a law—specifically, qualified majority (QMV) versus unanimity rules—and discretion to national administrations. Proposition 2.2 from his formal theory states that the discretion of national administrations under unanimity voting is greater than or equal to that prevailing under QMV (54–55). He tests this claim in several models, but our focus is on the “summary” tobit specifications for national administrations in the right-hand column of Table 5.6 that control for other hypothesized variables (172). The dependent variable is the discretion index $\left( {\hat{\delta }} \right)$, while the independent variable of interest is Decision rule, which takes a value of 1 for QMV and zero for unanimity rules (122), and is expected to have a negative effect in Table 5.6.^{Footnote 6} We focus on this variable because of its measurement quality: QMV versus unanimity rule is the precise distinction that Franchino’s formal theory incorporates.

As a final validity check, we seek to replicate the foregoing regression results while treating 90% of Franchino’s original observations as out-of-sample. This allows us to show that even with a very limited random sample of hand-coded provisions, we can achieve substantive effects that are similar in sign, significance, and magnitude to Franchino’s original inferences.

To do this exercise, we, first, trained a model on a random sample of only 10% of the hand-coded provisions. Second, we made predictions on the remaining 90% of those provisions. Third, we used those predictions to construct law-level predicted delegation and constraint ratios, which were combined into a predicted discretion index in the manner described above. Fourth, we used the law-level predicted discretion index (N = 158) in the tobit regression specification that Franchino (Reference Franchino2007) presents in Table 5.6.

Table 3 displays Franchino’s original results on the left and those $\hat{\delta }$ estimated from a 10% training sample. The reduction in the training sample by 90% introduces a measurement error into the predicted discretion index. Strikingly, we recover estimates similar in sign and magnitude to Franchino’s original inferences. In Table 3, the coefficient capturing the effect of a QMV rule on discretion to national administrations remains negative. The effect is only 8% smaller with a t-statistic that is just 2.8% larger. Because of the relatively small sample size and measurement error inherent in the policy complexity and policy position covariates, we note that the statistical significance of two CM policy conflict measures, Integration range and Left–right range, change. This may well be due to chance, but, nonetheless, the coefficient sizes and signs remain comparable.

TABLE 3.

Replication of Franchino (2007, 172) Table 5.6, Rightmost Column

*p < 0.10, **p < 0.05, ***p < 0.01.

Notes: Variable labels follow Franchino. Dependent variable is the national administration discretion index $( {\hat{\delta }} )$ and Decision rule indicates QMV (=1). Additional variables described in Franchino (Reference Franchino2007) at the specified pages, capturing conflict over EU integration, Integration range on the CM (130–31); policy positions, Standardized policy range on the CM (130–31); and left-right ideology, Left-right range(130–31) on the CM; and policy complexity, the existence of action programs and committee requirements Programme–committee (145–46, 166).

SUPPLEMENTARY MATERIAL

To view supplementary material for this article, please visit https://doi.org/10.1017/S0003055419000522.

Replication materials can be found on Dataverse at: https://doi.org/10.7910/DVN/FF3DQM.