Convergent validation

This section first reviews the main literature on measuring the robustness of lobbying laws. We investigate the indices by Opheim (Reference Opheim1991), Newmark (Reference Newmark2005), Chari et al. (Reference Chari, Hogan and Murphy2010) and Holman and Luneburg (Reference Holman and Luneburg2012), retracing the authors’ contribution to the interest group literature. The second part presents the result of a quantitative test performed on the coding applied to the regulations of Austria (which established its law in 2012), Australia (2008), Canada (2008), European Union (EU) (2011), France (2013), Germany (1951), Lithuania (2001), Mexico (2010), the Netherlands (2012), Poland (2006), Slovenia (2010), the United Kingdom (UK) (2014) and the US (2007). In order to evaluate the presence of dissimilarity between the four measurements under investigation, we present the results of the analysis of the standard deviations for the robustness scores of each case. This allows one to consider whether or not measures perform differently depending on the case of analysis. Next, we present the results of a correlation test.

Opheim’s (Reference Opheim1991, 405) pioneering work on measuring the rigour of lobby laws at the US state level represents the first contribution to the quantitative analysis of interest group regulation. She looks at 47 state regulations, identifying variations in terms of the degree of transparency and accountability that such laws guarantee. With the aim of investigating this variation, the author develops a measurement of the rigour of the regulation, which indicates the legislative independence and accountability from interest group pressure (Opheim Reference Opheim1991, 405). The measurement is based on a dichotomous coding procedure of the lobbying law according to 22 separately scored items drawn from three key dimensions of the lobbying regulation (Opheim Reference Opheim1991, 407). As the author argues, such key dimensions are “critical to the state’s effort to regulate special interest activity” (Opheim Reference Opheim1991, 407). The three dimensions are as follows: the definition of a lobbyist (seven items), the frequency and quality of disclosure of personal and financial information (eight items) and the enforcement of the regulation (seven items). Each item is coded 1 if the regulation includes the item and coded 0 otherwise. The result is an additive index that scores from 0 to 22, where higher scores indicate more robustness. The impact of her work is reflected in the main contributions to the US interest group literature – namely, Brining et al. (Reference Brining, Holcombe and Schwartzstein1993), Hamm et al. (Reference Hamm, Weber and Anderson1994) and Lowery and Gray (Reference Lowery and Grey1997) – either discussing or drawing upon this study. With the aim of producing a measurement, which is replicable over time, Newmark (Reference Newmark2005, 185) revised Opheim’s measure 15 years later. Newmark’s robustness measure is based on 18 items, which include elements of how lobbying is defined in the regulation, what information lobbyists have to disclose and what activities pursued by lobbyists are prohibited by the law. The coding procedure is dichotomous and the additive index results in a measure that varies from 0 to 18. Similar to Opheim’s index, high scores indicate high robustness. Unlike Opheim, Newmark’s index does not include items on the enforcement of lobbying laws.

Cognisant of the upsurge of jurisdictions beyond the US that were pursuing lobbying laws in the 2000s, as noted by several scholars examining the worldwide experience,Footnote ⁴ Chari et al. performed a global comparative analysis of the robustness of lobbying laws. In order to do so, they applied an index developed by the American think tank Centre for Public Integrity to existing regulations.Footnote ⁵ This index, named the Centre for Public Integrity (CPI) index, results from a coding procedure based on 48 items and eight key elements of the regulations. These key elements are as follows: (1) the definition of lobbyists, (2) individual registration requirements, (3) individual disclosure of financial information, (4) employer spending disclosure, (5) electronic filing, (6) public access to a registry of lobbyists, (7) enforcement and (8) revolving door provisions. In the case of the CPI index, the coding procedure is not dichotomous. The procedure weighs some items more than others, depending on whether the item is to be considered a critical feature of the key element.Footnote ⁶ The additive index results in a measure ranging from 0 (meaning low robustness) to 100 (meaning highest robustness). According to the levels scored by each regulation on the key elements of the CPI index, the authors distinguish between lowly regulated, medium-regulated and highly regulated systems (Chari et al. Reference Chari, Hogan and Murphy2010, Chapter 4). At first glance, this index presented at least one advantage compared with the two previous measurements: its increased number of key elements (eight compared with three of Ophiem’s measure) and items (48 compared with 22 and 18 of Opheim’s and Newmark’s indices) allow one to consider robustness more precisely across and within the elements of the regulation.

Finally, Holman and Luneburg (Reference Holman and Luneburg2012) were the first to consider the relative strictness of lobbying rules within European political systems. In their study, they provide a theoretical classification of regulated systems in line with Chari et al.’s contribution. The authors design their classification on 21 items that characterise lobbying regulation, which allows for an additive index to be made, as has been recently performed by Crepaz (Reference Crepaz2016a). Similar to previous studies, these items include the definition of lobbying, the disclosure requirements and enforcement of the rules. In addition, some items include whether the regulation is mandatory or voluntary,Footnote ⁷ whether or not the rules include the presence of codes of conduct for lobbyists and whether or not some interest groups are exempt from the rules. The regulated systems in Europe can be classified into strong or weak regulations depending on how they perform on the 21 items (Holman and Luneburg Reference Holman and Luneburg2012, 21). Similar to Crepaz (Reference Crepaz2016a), we summed up the scores into a dichotomous index ranging from 0 (meaning low robustness) to 21 (meaning maximum robustness), with the aim of transforming the indicators provided by the authors into a quantitative measurement of the robustness of lobbying regulations. Compared with the previous measurements, this index presents the advantage of including features that are based on European lobbying regulations. However, its limited number of items does not allow one to consider all aspects of robustness in detail.

With the above in mind, the robustness levels for 13 lobbying laws have been calculated applying the four methods of measurement. The results of such coding are shown in Table 1, which shows nominal scores. Our coding matches with the results presented by Chari et al. on Australia, Canada, Germany, Lithuania, Poland and the US. Similarly, our results are consistent with Holman and Luneburg (Reference Holman and Luneburg2012, 21) on Germany, Lithuania, Poland and Slovenia.

Table 1

Scores of 13 lobbying laws applying four different measurements of robustness

Note: EU=European Union; UK=United Kingdom; US=United States.

Source: Own calculations.

With the aim of allowing comparison of relative scores, the robustness scores have been normalised, ranging from 0 (meaning lowest robustness) to 1 (meaning highest robustness), as seen in Table 2.Footnote ⁸

Table 2

Normalised scores of 13 lobbying laws applying four measurements of robustness

Note: All scores vary from 0 to 1.

EU=European Union; UK=United Kingdom; US=United States.

Source: Own calculations.

The results shown in Table 2 suggest that the robustness levels are more dissimilar for some cases than for others. Generally, the measurement by Holman and Luneburg scores higher values of robustness levels for all cases of analysis. On the contrary, the robustness scores using the measurements by Opheim, Newmark and Chari et al. tend to be more similar with the exception of some cases such as Austria and the EU. In the rest of the cases, at least two measures have equal or similar values. Interestingly, the calculated robustness score for the US shows a high similarity between the measures by Opheim, Newmark and Chari et al., suggesting that these measurements, developed on the American tradition of regulating lobbying, best apply to this case of analysis. The opposite is true for European lobbying laws, especially the Austrian law.

The last row of Table 2 shows the standard deviation on each measurement. Quantitative studies in political science typically have an approach that seeks to explain variation in the outcome (Mahoney and Goertz Reference Mahoney and Goertz2006). Hence, the more variation scholars have in the outcome, the better it is for their empirical investigation. In the case of this analysis, the measurement by Holman and Luneburg shows a higher variance compared with other measures. Figure 1 shows why this is the case.

Figure 1

Distribution of the scores applying four measurements of robustness. Source: Own calculations.

The last box plot shows the maximum (0.86 for the US) and the minimum value (0.24 for Germany). The minimum is treated as an outlier in the distribution. This is because 92% of the scores vary between 0.38 and 0.86. The distribution of the measurements by Opheim and Chari et al. are symmetric and similar in terms of minimum, maximum, median and first and third percentiles. Newmark is dissimilar in its maxima (France, Austria and Slovenia with a score of 0.61). However, none of the scores takes higher values than 0.64. This confirms a main difference between the first three measurements and the last measure by Holman and Luneburg. The latter tends to score higher levels of robustness compared with the other measurements. The following considers this idea in more detail.

Low standard deviation scores indicate similarity in the robustness level using the four measures, whereas high scores indicate dissimilarity. For example, the high SD of Austria (0.22) suggests that the calculation of the robustness level using one measure or the other leads to dissimilar results. On the contrary, the opposite result applies to Lithuania, Slovenia, the UK (all SD<0.10) and in particular Germany (SD<0.05). In the next section, we will examine in more detail how one can account for the differences between the countries.

We will now evaluate whether the measurements under investigation perform differently on average by providing rank-order correlations between all measurements. This provides insights into the validity of each measurement using convergent validation (Adcock and Collier Reference Adcock and Collier2001; Rogers and Weller Reference Rogers and Weller2014). Correlations indicate the covariance of how the robustness scores are ranked, expressed as deviations from their mean. In other words, they indicate average similarity. The results are shown in Table 3.

Table 3

Rank-order correlations between the scores of 13 lobbying laws using four measures of robustness

Source: Own calculations.

The results of the correlation test suggest that there is a fairly close similarity on average between the four measures of robustness, although some findings are worth highlighting. The CPI index by Chari et al. correlates closely with Opheim’s and Holman and Luneburg’s measurements (0.74 and 0.73, respectively). Its correlation with Newmark’s index is, however, low (0.51). The second lowest correlation is between Newmark’s index and Opheim’s index (0.68). Holman and Luneburg’s index fairly correlates with all measurements under investigation (0.77, 0.81 and 0.73). This suggests that the American and European traditions of coding lobbying laws are not a dividing principle. From this test of convergent validation, we conclude that, on average, Holman and Luneburg’s measurement is most similar to other measurements, whereas Newman’s is most dissimilar.

In conclusion, the four measurements of robustness perform differently depending on the case of analysis: evidence has shown that some indices disagree on the robustness levels of some regulations. This is particularly the case of the measurement by Holman and Luneburg. Compared with the other measures, this index tends to score higher levels of robustness. The other indices by Opheim, Newmark and Chari et al. show a higher degree of similarity, especially when it comes to coding the US legislation. However, on average, Newmark’s index shows a higher degree of dissimilarity compared with the other measurements, whereas Holman and Luneburg’s index shows the highest similarity.

A limitation to convergent validation, however, is that high correlations among indicators “may reflect factors other than valid measurement. For example, two indicators may be strongly correlated because they measure some other concept; or they measure different concepts, one which causes the other” (Adcock and Collier Reference Adcock and Collier2001, 541). To dispel any possibility of misspecification between concept and measurement, it is necessary to consider another method of validation, which seeks to examine whether an indicator “capture(s) the full content of the systematised concept” (Adcock and Collier Reference Adcock and Collier2001, 541), or, namely, content validation.

Content validation

This second section investigates and evaluates the indices’ construction, probing whether the items of the different measurements consider all relevant conceptual aspects of robustness of the lobbying regulation. Adcock and Collier (Reference Adcock and Collier2001, 539) examine various examples of content validation by focussing on developments in the literature on democracy, highlighting how authors such as Paxton (Reference Paxton2000) consider the problems of “omission of key elements from the indicator and inclusion of inappropriate elements”.

Although not developed in the extant literature, a novel way to evaluate what may be missing in an indicator, which is particularly useful when examining indices that attempt to evaluate a public policy, is to consider a relatively objective “gold standard” from a reputable international organisation that outlines what should be entailed in a law in terms of best practice, and then see how well indices capture this. Such benchmarks are established by various international organisations such as the OECD and the Council of Europe for various public policies pursued at the domestic level, such as lobbying regulation, whistleblowers legislation, ethics reform laws and privacy protection laws to name a few.Footnote ⁹

One reason for taking the OECD standard as a point of comparison is that national policymakers often seek inspiration in norms established by international organisations. Previous studies have shown that national policymakers gain such inspiration from standards that are established by international organisations (True and Mintrom Reference True and Mintrom2001; Stone Reference Stone2004). Yet, we should be careful to not consider the policy recommendations of international organisations as being scientific standards for validation, as these policy positions may be the result of subjective deliberations, or simply ill-informed discussions, which may have left out important indicators of what constitute robust lobbying laws. However, the relevance of the policy recommendations of international organisations about the adoption of lobbying laws has been increasingly acknowledged in the academic literature and in the real world of politics. For instance, Crepaz (Reference Crepaz2016b) found that the OECD and the EU recommendations about lobbying regulations have been successful in encouraging the adoption of lobbying laws among member states. This finding is supported by qualitative evidence: the Austrian and Irish governments introduced their lobbying law following the principles established by the OECD.Footnote ¹⁰ With this justification in mind, we decide to conduct content validation considering the elements that characterise robust regulations according to the OECD’s (2008) report on lobbying regulations, which is worth considering in some detail.

The OECD has focussed on the promotion of lobbying laws over the last decade through its Department of Government Integrity, dedicating particular attention to the topic of regulating lobbying as part of fighting corruption in the public sector. In 2008, the OECD published a report containing guidelines for the introduction of robust lobbying rules in its 34 member states. Such guidelines are based on the conceptual elements of defining lobbying, disclosure requirements, reporting processes and technology, timeliness (namely update of information), enforcement and compliance (OECD 2008, Chapter 2). According to the report, regulations that extensively consider these five key elements are to be considered robust (OECD 2008, 16). Hence, the more the construction of the indices under investigation captures these key items, the more such measurements catch the concept of robustness as defined by the OECD, which thus represents what can be deemed to be the “gold standard”.

The key items, and how the different indices capture them, are as follows:

1. 1. Defining lobbying: in order to be robust, lobbying laws need to clearly define lobbying, critical for understanding “who is to be regulated” (OECD 2008, 42). There are three main aspects to defining lobbying. First, the regulation needs to distinguish between government officials and lobbyists (OECD 2008, 42). The measurements by Opheim and Newmark include this in items 3 and 4, asking whether the definition of lobbyists considers public officeholders as well. Items 5, 6 and 7 of the same measurements consider whether a minimum compensation, expenditure or time standard apply to the definition of lobbyist. Similarly, item 2 of the CPI index by Chari et al. include the same aspects to the definition of lobbying activity. Second, a robust definition of lobbying should also include all interest group categories, meaning that it should aim at regulating consultant lobbyists acting on behalf of third parties, in-house corporate lobbyists and NGOs (OECD 2008, 44). Only the index by Holman and Luneburg considers whether the regulation affects lobbying consultancies, for-profit interest groups and nonprofit interest groups equally (items 3, 4 and 5). Third, the definition of lobbying needs to clearly identify state actors targeted (OECD 2008, 46). The measures by Opheim and Newmark consider whether the regulation affects lobbyists seeking contacts with members of Parliament and civil servants (items 1 and 2). Ministers are therefore excluded from these items. On the contrary, Holman and Luneburg include both legislative and executive lobbying in items 6 and 7. Finally, the CPI index assumes that legislative lobbying is covered and codes executive lobbying only (item 1).

2. 2. Disclosure requirements: a further critical aspect of regulating lobbying is related to the disclosure requirements. Typically, lobbyists need to enter a certain amount of information in a (public) register before establishing any contact with public officeholders, where the amount and accuracy of the information is directly related to the law’s robustness. This includes personal details, objectives of lobbying and financial disclosure (OECD 2008, 50, 51, 57, 58). Regarding the construction of the measurements, only the indices by Chari et al. and Holman and Luneburg consider the disclosure of personal information (items 8, 9 and 10 for both indices). These include personal details such as name, address, contacts and photograph. When it comes to items concerning the disclosure of objectives of lobbying, again Chari et al. and Holman and Luneburg provide the most complete index construction. Item 5 of the former index and items 11 and 12 of the latter consider whether lobbyists need to disclose the subject matter of the lobbying activity or even who is targeted. Another item in Opheim’s index (item 12) asks whether interest groups approve or oppose the legislation they are seeking to lobby on at the moment of registration.Footnote ¹¹ On the disclosure of financial information, all four measurements have some items that consider total spending, categorised spending, total income of lobbyists and sources of income. In addition, only the CPI index includes separate items for campaign contributions and gifts (items 23 and 24). The same measurement also considers whether employers of lobbyists need to submit financial information (items 26 and 27). On the contrary, Holman and Luneburg do not consider gifts and other forms of donations benefitting public officeholders. Newmark focusses on whether campaign contributions and donations are considered as prohibited activities rather than to be disclosed (items 15–18). The last section of this key element focusses on other disclosure requirements (OECD 2008, 60), where additional elements are found as follows: Opheim’s index provides one item on the disclosure of potential conflicts of interest or influence peddling (item 15); Holman and Luneburg consider whether lobbyists need to disclose every contact with public officeholders (item 17); and Chari et al. collect any additional form of disclosure (item 10.)

3. 3. Timeliness and ethics: the OECD report (2008, 66) evaluates the quality of disclosure, such as the frequency of registration and updates of the information in the register. Opheim and Newmark provide items on the frequency of registration (items 8). Chari et al., more in detail, provide items on whether registration needs to be completed before lobbying (items 4 and 6), whether the regulation sets rules on the notification of changes in the registered information (item 7), whether lobbyists need to submit a no activity report (item 25). Further, the OECD (2008, 67) outlines that lobbyists should sign a code of conduct, where Holman and Luneburg uniquely consider such codes of conduct in item 21.

4. 4. Reporting processes and technology: this element focusses on the electronic filing of information and the public access to such information (OECD 2008, 65–66). Although Opheim and Newmark do not dedicate items to this, Holman and Luneburg consider whether or not the information on the lobbying register is accessible by the public (item 20).Footnote ¹² Chari et al. dedicate more attention to this aspect by considering whether or not the access to the register is public (items 31–34), the register provides users with summary reports (items 35–37), lists are updated frequently (item 38) and lobbyists can file their reports electronically (items 28, 29).

5. 5. Enforcement and compliance: the first of four dimensions of this element (OECD 2008, 71–73) considers education and training on lobbying regulation compliance. Chari et al.’s index uniquely considers this aspect (item 30). The second aspect focusses on whether the regulation is voluntary or mandatory in nature. Here, only Holman and Luneburg include this in their robustness measure (item 1). The third aspect considers the enforcement of the rules (and the efficiency thereof) by the public body entrusted to monitor compliance. On the power of the monitoring agency, Opheim’s includes its ability to review information and reports (item 16), demand subpoena witness or record (items 17, 18), conduct administrative hearings (item 19), apply fines or penalties in cases of noncompliance (items 20, 21) and file an independent court action (item 22). Although Newmark does not include any items on this key element, Chari et al.’s index has a high number dedicated to this, including the power to review reports (item 40), publish the names of lobbyists infringing the rules (item 47), apply fines or penalties for noncompliance (items 41, 42, 44, 45) and ask about the last levied case of noncompliance (items 43, 46). Less detailed, Holman and Luneburg include whether the monitoring agency can apply fines or penalties for cases of noncompliance in one single item (19). The fourth aspect of this key element concerns cooling-off periods, which delay politicians and civil servants from entering the lobbying industry. Of all measurements, only the CPI index used by Chari et al. considers this (item 48).

In Figure 2, we illustrate the number of items (per measurement) that fall within the conceptual dimensions of the robustness of lobbying laws as identified in the OECD policy recommendations. The figure shows, for example, that, on the first key element Defining Lobbying, the measurements by Opheim and Newmark count seven items each, whereas Chari et al.’s and Holman and Luneburg’s indices dedicate two and five items, respectively, to this dimension. This suggests that, compared with others, the first two measurements demonstrate more content validity on this element, meaning that the index is better able to catch the concept of robustness using the OECD’s report as gold standard. Considering all five OECD’s key elements of the robustness of lobbying laws, Chari et al.’s index appears as the most valid (in terms of content) as the highest number of items falls in conceptual dimensions identified by the OECD, followed by Opheim’s index, Holman and Luneburg’s and then Newmark’s.

Figure 2

Number of items (per measurement) that fall in the five Organisation for Economic Co-Operation and Development key elements of the robustness of lobbying laws. Source: Own calculations.

It is interesting to note that using the OECD’s conceptual dimensions also allows us to better understand the differences in the robustness levels across the countries, which was seen in the previous section on convergent validation. In terms of method, we analyse the levels of robustness by disaggregating the total scores for each of the indices for the 13 pieces of legislation into the five conceptual dimensions of the OECD. We assign each item of the indices to one of these conceptual dimensions (as seen in Figure 2) and calculate the mean robustness level of the dimension for each country. This results in 20 observations for each country (260 observations in total). Next, we calculate the average standard deviation for each dimension considering all 13 pieces of legislation. This process results in five means (one for each dimension of the OECD) that indicate levels of dissimilarity between measurements. High averages indicate dissimilarity between the measurements concerning a given conceptual dimension, whereas low scores signify similarity. Figure 3 displays these levels of similarity for each conceptual dimension in percentages.

Figure 3

Percentage of dissimilarity between measurements by conceptual dimensions of the robustness of lobbying laws (averages calculated on all 13 pieces of legislation). Source: Own calculations.

The results suggest that the four measurements are the most dissimilar in the conceptual dimensions of definition of lobbyists (36%) and (to a lesser extent) timeliness and ethics (24%). From this perspective, a dividing principle between the measurements is the way in which they score “how lobbying is defined by the regulation” and “how frequently information is disclosed and updated”. Together, these two dimensions account for 60% of the total dissimilarity between measurements. The levels of similarity between the scores of the items assigned to the dimensions concerning disclosure requirements, the reporting processes and the enforcement of regulation, on the contrary, reveal, on average, a higher degree of similarity. From these observations, we can conclude that the way indices capture the dimensions of the definition of lobbyists, in particular, influences the calculation of the robustness level and that the indices under consideration measure the latent concept on, at least, four out of five of the OECD’s conceptual dimensions.

Summarising the main findings for this section, the results in favour of the CPI index suggest that, if the right key elements are addressed in the construction of the index, having more items makes an index more valid. In other words, if you ask more questions, you are more likely to come up with a more complete and objective answer. In fact, Opheim, which is constructed by 22 items, scores second, Holman and Luneburg, which is constructed by 21 items, scores third and Newmark with its 18 items scores last.

Nevertheless, content validation is limited by a trade-off between parsimony and completeness that arises because indicators routinely fail to capture the full content of a systemised concept. Capturing this content may require a complex indicator that is hard to use and adds greatly to the time and cost of completing the research. It is a matter of judgement for scholars to decide when efforts to further improve the adequacy of content may become counterproductive (Adcock and Collier Reference Adcock and Collier2001, 539).

The next section thus elaborates on this argument in more detail, first considering how the trade-off referred to impacts on reliability and then focussing on the indices.

Reproducibility

In the previous sections on methods of validation, we concluded that Holman and Luneburg’s and Chari et al.’s indices appear to be the most valid (the former in terms of convergence and the latter in terms of content). From these observations, we inferred that high-dimensional indices tend to be more valid than lower-dimensional ones. However, we argue that validation is not the only criteria upon which to judge a measure as it might favour the construction of complex and high-dimensional indices. In this section, we introduce a third criterion of evaluation – namely, reliability – which indicates whether a measurement technique secures consistent results upon repeated application. We argue that reliability is a function of “repeatability” and “reproducibility”. Both repeatability and reproducibility refer to a variation in the level of agreement adopting the same measurements on the same subject. However, repeatability is attained when the same observer or rater attains consistent measurements, whereas reproducibility is attained when different observers or raters attain similar results to the initial observer (Bartlett and Frost Reference Bartlett and Frost2008, 467–468).

The importance of the “lack of reproducibility” is based on an emerging concern within the natural science community related to challenges in reproducible research as recently examined in a series of special reports by Nature.Footnote ¹³ As an example in natural science, let us say Medicinal Chemist “A” has developed procedures to obtain a significant yield for a new compound, which may be of value not only to fellow academics, but also to the pharmaceutical industry that may use the compound to help develop a new drug. However, the pharmaceutical company may think twice about using A’s procedures if they are long, complex and have a low reproducibility rate. As such, for practical reasons, the company may consider a simpler, more parsimonious, procedure developed by Medicinal Chemist B to develop the same or similar structure, even if it gives a lower yield.

Analogously, one may argue that the rise of lobbying laws over the last 15 years means that many governments throughout the world are relying on scholars’ work on lobbying regulation when developing their laws, as evidenced in various submissions, committee reports and proceedings from the UK, Scotland and Ireland.Footnote ¹⁴

Governments may also consume this research when developing their own laws as well, not dissimilar to how a pharmaceutical company may consume research from a medicinal chemist. Let us say that a team of civil servants and their relevant Minister are tasked to develop a draft lobbying bill. After they have developed this draft, the team may seek to measure what “level of robustness” the potential law will have before it is presented to Cabinet. This would be done in order to better gauge how much transparency and accountability the new potential law provides for. The importance of doing this can be illustrated with two hypothetical scenarios. If it is found that the draft bill only provides for a lowly regulated environment, then this may be unacceptable for a newly elected government that wants to “clean up” politics in a country historically riddled with corruption. Conversely, a government of a country with low levels of corruption may prefer to develop less robust legislation given higher costs associated with developing tighter regulation. In both scenarios, the policymaking team that has measured the robustness of the draft bill may potentially make changes before final presentation to Cabinet. For example, relying on Chari et al., if the policymakers’ analysis of the draft bill reveals a point score of more than 60 points, then this would be representative of law in a highly regulated system, between 30 and 59 points, medium regulation, and between 0 and 29, low regulation.

With this in mind, the team of policymakers would want to know which index can be applied simply and reliably, wherein the attained results are the same as researchers in the field who apply the index for their own academic work: “which of the four indexes gives me a reliable score?” That is, the team would want an index that does not suffer from a lack of reproducibility because it is so complex to apply. Extending on this logic, one may even argue that interest groups that feed into the consultation process when lobbying laws are developed may equally want to know which index is the most reliable when seeking to measure the robustness of lobbying bills that are eventually tabled by Cabinet. Equally, scholars who are not necessarily experts of lobbying regulation may want to know which is the most reliable index to measure the robustness of lobbying laws of a country – and then compare this with, for example, the strength of FoI legislation – in order to better evaluate the “sunshine laws” of said state.

In this regard, having a more complex indicator when evaluating public policy is particularly important not only in terms of costs to researchers but also to practitioners who seek to reproduce results. It is reasonable to hypothesise that more items in an index could lead to a higher complexity of the coding procedure, and therefore increase the error when trying to measure the robustness of the law. With this in mind, this section considers whether or not the most valid indicators as seen in the previous sections are really the ones that scholars and practitioners would want to use in the context of being reliable, which refers to minimising measurement error.

This reliability test is based on the comparison between our results presented in Table 2 in the first section and the results produced by trained coders. It is important to note that coding is a common procedure when research in social science involves the analysis of texts, where such procedures have been developed in measuring the position of political parties on a left–right dimension (Mikhaylov et al. Reference Mikhaylov, Laver and Benoit2012), the degrees of populism (Rooduijn and Pauwels Reference Rooduijn and Pauwels2011) and research on interest groups (Boräng et al. Reference Boräng, Eising, Klüver, Mahoney, Naurin, Rasch and Rozbicka2014). However, when human coding is involved, vague, misleading or incomplete coding procedures may lead to arbitrariness in the coding (Krippendorff Reference Krippendorff2004). Reliable coding procedures must not leave space for interpretation to coders (Mikhaylov et al. Reference Mikhaylov, Laver and Benoit2012). In order to be perfectly reliable, a coding procedure needs to lead to the same result regardless of the coder. This is very unlikely and developers of coding procedures normally aim at minimising intercoder error.

Our coding results were produced in two stages. First, a pilot-coding test was performed on 10 coders. This allowed us to collect the robustness scores on 10 out of the 13 pieces of legislation studied in the article: Austria, Australia, the EU, France, Germany, Lithuania, the Netherlands, Poland, UK and the US (2007).Footnote ¹⁵ This preliminary test resulted in 40 robustness scores, which were compared with our coding results (seen in the first section). This pilot stage also saw that the EU legislation registered the lowest level of reliability. This resulted in our decision to focus on the EU with a larger sample of coders in the second stage in order to guarantee a higher degree of accuracy.

The second coding test thus involved 15 trained coders that focussed exclusively on EU lobbying regulations and used all four indices. This resulted in 60 robustness scores on one piece of legislation, reducing the confounding factor of coding different pieces of legislation. According to Benoit et al. (Reference Benoit, Conway, Lauderdale, Laver and Mikhaylov2016), the number of coders is to be considered adequate, as standard deviations tend to not vary when such tests involve 15 coders or more. The coders were familiar with the EU lobbying law of 2011 and were familiar with the literature on lobbying regulation as well as the coding procedures found therein. The coders were third-year undergraduate students enrolled in the module on EU politics, which also covers the issue of lobbying in the EU and policies aimed at increasing transparency in the European Parliament and in the Commission. One traditional career path of these students is to work in the civil service, meaning that they are representative of future policymakers who may one day be involved in developing transparency laws. The students chose to partake in the test on a voluntary nature given their interest in the topic. The Ethics Research Committee of the School of Social Sciences and Philosophy, Trinity College Dublin, approved the methodology and the procedures of the test.

In terms of the findings, we first present the distribution of the coding error, where the error is expressed in proportion of agreement between coders. For each code assigned to each item of the measurements’ construction, we calculated the level of agreement (range 0–1), as displayed in Figure 4. The closer the level of agreement for each item is to 1, the better the index performs in terms of reliability on that particular item. The y-axis shows the proportion of levels of agreements. For instance, a value of 0.8 on item 10 of Opheim’s index means that 80% of the coders answered in the same way to item 10 of Opheim’s coding methodology.

Figure 4

Distribution of the coding error expressed in proportion of agreement for each item. Source: Own calculations.

In the distribution of the levels of agreement between coders for Chari et al.’s index, we also included the number of possible answers to the items (according to the scoring methodology). For example, the scoring methodology for item 1 is dichotomous, whereas for item 2 coders can assign a score choosing from five different options. We account for the number of possible answers because we believe that a more complex answer scheme could impact the overall level of reliability of the index.Footnote ¹⁶ For example, the proportion of agreement of Chari et al.’s index for item 32 is below 0.50. This might be due to the fact that the scoring methodology allows coders to choose between four different answers to item 32 with the risk of reducing the overall reliability of the index on this particular item.

The results shown in Figure 4 suggest that the index by Chari et al. appears to have the lowest intercoder agreement expressed in proportion of agreement for each item. The measurements by Opheim, Newmark and Holman and Luneburg show higher levels of agreement. In particular, the index by Holman and Luneburg reaches levels of 100% agreement in 24% of the items and agreement levels above 90% in 43% of the items. This is the highest result compared with the other measurements. On average (for all items and for each measure), the proportion of agreement equals 70% for the index of Chari et al., 78% for Opheim, 82% for Newmark and 84% for Holman and Luneburg. This is better shown in Figure 5, which illustrates the distribution of the levels of agreement for each measurement.

Figure 5

Distribution of the levels of agreement for each measurement. Source: Own calculations.

The results shown in Figure 5 suggest that levels of agreement, on average, seem to be highest for the index by Holman and Luneburg and lowest for the CPI index of Chari et al. To test whether this agreement is statistically consistent, we provide the results of an intercoder agreement test using Krippendorff’s α statistic (as shown in Table 4), which is commonly used by researchers to evaluate levels of agreement (Mikhaylov et al. Reference Mikhaylov, Laver and Benoit2012).

Table 4

Intercoder agreement using Krippendorff’s α statistic

Source: Own calculations. Level of Agreement is defined by the categories of no agreement (0), slight agreement (0.01–0.20), fair agreement (0.21–0.40), moderate agreement (0.41–0.60), substantial agreement (0.61–0.80) and almost perfect agreement (0.81–1) (Landis and Koch Reference Landis and Koch1977).

All calculated αs are statistically significant, meaning that levels of agreement are not random (Lombard et al., 2002). The level of agreement varies from fair (for the indices of Opheim and Chari et al.) to moderate (for the indices of Newmark and Holman and Luneburg). Above all, the index by Holman and Luneburg has the highest α statistic, confirming the argument that levels of agreement are more consistent throughout all elements that compose the measurement. One may reasonably argue, however, that there is no measure that is more reliable than “moderate”, which is still not very high. This lack of reliability for scholars interested in lobbying regulations suggests that, although some perform reasonably well, there is room for improvement to develop new ones in future research. A unique way of evaluating whether or not a new index is reliable would be to fruitfully test its reliability throughout the development stage of the index, asking coders to score legislation, as was performed in this section.