Biases in Global Performance Indicators

Over the past few decades the proliferation of Global Performance Indicators (GPIs) has risen sharply. The stakes are high for states assigned GPI outcomes, particularly with regard to human rights measurements in the age of successful ‘naming and shaming’ (Bandura Reference Bandura2008; Davis et al. Reference Davis2012; Kelley and Simmons Reference Kelley and Simmons2019; Murdie and Davis Reference Murdie and Davis2012). Meaningful consequences for GPI outcomes incentivize bias in TIP Reports.

The first possible source of bias in GPIs is changing standards of accountability. Clark and Sikkink (Reference Clark and Sikkink2013) and Fariss (Reference Fariss2014) argue that, rather than explicit political bias, the information environment that feeds the reporting agencies has markedly changed over the last few decades. Recent work by Greene, Park and Colaresi (Reference Greene, Park and Colaresi2018) tests whether the assumption of changing standards in the State Department's Human Rights Reports is warranted, and concludes that standards of accountability are changing. We build on this work to test this argument with the TIP Reports. However, due to the unique nature of the TIP Report, we argue that political bias may be present as well.

We propose two mechanisms for how political biases may affect how report narratives are translated into rankings. The first stems from the way in which the report narratives and rankings are constructed – the data-generating process. The second is determined by the expected consequences of the rankings, which lead to pre-emptive political changes. Variation in the data-generating process and the rankings' high-stakes consequences drive the types of bias that are likely to manifest.

Unlike other human rights measurements (for example, the Political Terror Scale (Gibney et al. Reference Gibney2015) and CIRI (Cingranelli, Richards and Clay Reference Cingranelli, Richards and Clay2014)), the TIP Reports do not receive independent third-party input. Mandated by Congress starting in 2000, the State Department produces the report narratives and rankings annually in a multi-stage process. First, the Office to Monitor and Combat Trafficking in Persons within the State Department (J/TIP) makes recommendations for states' rankings and drafts the narratives based on input from trafficking experts who consult extensively with local non-governmental organizations (NGOs), stakeholders and leaders. These rank recommendations are then presented to more senior diplomats, including political appointees who can amend the rankings in a process that is not visible to the public. The rankings and narratives are then released to the public in their final format. Disagreements between the human trafficking experts in non-political positions and higher-ranking staff in political positions are not disclosed to the public except on a case-by-case basis of leaks.

There is clear evidence of a disjuncture between TIP Report rankings and narratives. The Government Accountability Office (GAO) has cited concerns that the report narratives seem to contradict the associated rankings and stated tier criteria (Government Accountability Office 2016). Additionally, the GAO found that ‘for countries that changed tier from one year to the next, most narratives did not provide an explicit explanation as to why the rankings changed’ (Government Accountability Office 2016). This also suggests that narratives are not simply used as justifications for politicized rankings. Similarly, an investigation by Reuters found that internal disagreements between the recommendations made by trafficking experts in J/TIP and diplomats' preferred rankings often led to the diplomats' rankings being used, but did not indicate any evidence that the narratives were altered from expert recommendations (Szep and Spetalnick Reference Szep and Spetalnick2015).

The bias is therefore introduced in what Haschke and Arnon (Reference Haschke and Arnon2020) call ‘the second stage’ of the data-generating process, in which the narratives are converted into rankings. Here we evaluate only the bias introduced in this mapping, rather than address what information is included in the initial drafting of the narratives. It is important to note that while a single entity, the State Department, ultimately publishes the TIP Reports, the process that generates the reports involves multiple parties that have competing views. Thus it is unclear how closely the TIP Reports will mirror the trajectory of other human rights measurements created by independent third parties.

The second mechanism potentially biasing the rankings in addition to an opaque creation process are the unique political consequences of rankings. These consequences create incentives to pre-emptively leverage rankings as political rewards or punishments. Report rankings, in contrast to most other standards-based GPIs, are legally tied to bilateral outcomes between the United States and its partners. In 2000, Congress passed the first comprehensive federal legislation to combat human trafficking, which states that the US government must withhold all non-humanitarian aid from Tier 3 states, including military assistance. Furthermore, US representatives at the International Monetary Fund and the World Bank are instructed to vote against or deny loans to any Tier 3 state. Since many states depend on such assistance and US allies tend to follow its lead on these votes, rankings become consequential for state outcomes and foreign policy.

A specific example of concern regarding bias in the report comes from practitioners who have complained about the TPP's role in creating politically biased rankings. The TPP is a now-defunct proposed trade agreement between twelve nations including the United States. The United States first agreed to enter negotiations in 2008, but most took place between 2010 until the process ended with US withdrawal in 2017. TPP negotiations coincided with an increased interest in economic ties to many Asian states (including Thailand, for example). NGOs have cited this concern about Asian report rankings while testifying to Congress. Speaking to a House Committee on Foreign Affairs Subcommittee, Freedom House's president and former J/TIP director Mark Lagon warned of bias by pointing to cases like Malaysia. He said that ‘the major controversy of the 2015 Report is the unjustified upgrade of Malaysia… the upgrade seems to coincide more with desires to make Malaysia more eligible to join the TPP than the merits’ (Lagon Reference Lagon2019). Human Rights First, an NGO, has also spoken out about bias in Malaysia's TIP Report (Human Rights First 2017). The organization is supporting a legislative reform proposed by Senator Marco Rubio (Rubio Reference Rubio2019); Senior Associate Annick Febrey argues that ‘when politics interfere and allow even one or two countries’ rankings to be inflated, the credibility of the entire report suffers’ (First Human Rights 2017). A survey of the practitioner and academic work indicates that contestation occurs mainly around the reports’ rankings, rather than narratives (Szep and Spetalnick Reference Szep and Spetalnick2015).

We derive two expectations. First, consistent with the extant literature, we expect changing standards of accountability to occur in how the narratives map to rankings over time. Secondly, diverging from the current literature's findings, we expect that political bias will be more likely to manifest in highly salient cases due to differences in the data-generating processes and the political consequences of the rankings, which, as tested in this project, can be observed in Asian states around the time of the TPP negotiations. To be clear, we are testing one possible manifestation of political bias; others are feasible but are outside the scope of this project.

Research Design

To test our expectations, we use a rolling window forecasting model based on a supervised machine-learning technique (Greene, Park and Colaresi Reference Greene, Park and Colaresi2018). The algorithm examines how the lexical features of the TIP Report narratives are mapped onto TIP Report rankings. The function is defined as y _it = f _t(x _it) where y _it refers to a TIP ranking for country i ∈ (1, …, N) at time t ∈ (1, …, T). x _it denotes a vector of lexical features. f (⋅) is a function that maps lexical features onto rankings. Once the function is trained on a specific set of reports, or the training set, we use it to predict the rankings of different reports. We then compare the accuracy of the predictions based on different training periods of the data. This approach allows us to test for changes over time in the mapping of narratives to rankings. We do this first on the global sample to test for changing standards, and then we subset on cases for which we would expect political bias. If the results remain similar using both training sets, this would indicate that the main bias is changing standards of accountability.

We first split the data into a changing (rolling) in-window training set and a constant (fixed) out-window testing set. The in-window training set spans the early period reports (2001–2012), in 5-year increments of reports, to train the function to predict rankings based on the narratives' lexical features. We then test the predictions on the constant out-window testing set (out-of-sample testing).

Figure 1 illustrates the model. In Window 1, the narratives from 2001–2005 (a 5-year window) train the algorithm to predict the rankings. The trained algorithm then predicts the rankings of the most recent 5 years of reports, 2012–2017 – the out-window testing set. We then record the accuracy of the out-window prediction of the 2001–2005 in-window training set. This process is repeated eight times, and each iteration moves the in-window training set one year forward. In Figure 1 these are Window 2, Window 3 and so forth.

Figure 1.

Fixed rolling window model

The main evaluation metric to examine is the out-window accuracy. This measures the proportion of cases that the algorithm predicts correctly based on a specific in-window training set. Each of the eight 5-year in-window training sets produces a measure of out-window accuracy as reported in Figures 3 and 4 in the top panels. If the algorithm improves out-window accuracy over time, this would indicate that report standards are changing over time, since later reports more accurately predict contemporary reports than earlier ones. Put differently, if the out-window accuracy from the early in-window training sets is lower than the out-window accuracy from later in-windows, this is likely because the mapping of the narratives into rankings has substantively changed, which is what the algorithm is detecting. Analyzing the State Department's Human Rights Reports onto Political Terror Scale (PTS) rankings, Greene, Park and Colaresi (Reference Greene, Park and Colaresi2018) find an increase of roughly 15–20 per cent in out-window accuracy and conclude that standards of accountability are changing. This large change in out-window accuracy serves as our benchmark for testing whether standards have changed in the TIP Report.

To test whether political bias is introduced into the report-to-rank mapping, we subset the sample to cases in which the reports are expected to be politically salient for the United States – Asia during and after the initiation of the TPP negotiations. If accountability standards are changing across the board for all countries, we should observe a similar pattern of out-window accuracy for this subset. Alternatively, if the out-window accuracy changes only around the period of the TPP, we can infer that the narrative-to-rank mapping changes in response to political considerations. Because we expect political cajoling to occur only around the rankings, and narratives to remain unchanged, the algorithm should detect unexpected deviations from the mapping of narratives to rankings that did not occur in the global sample. To be clear, we expect that additional forms of political bias may also exist; the particular subset of cases we test simply represents one manifestation identified by practitioners. This test allows us to determine the feasibility of the claims of political bias.

The second metric reported for both the global and Asia-only samples is the in-window accuracy. We train the algorithm within the in-window training set, and test the accuracy of prediction within the window (5-fold cross-validation). In-window accuracy is similarly the proportion of cases predicted correctly. This metric tests whether there are changes in the algorithm's ability to learn and correctly predict over time. The third metric is bias, which is calculated as the average of differences between predicted and observed values. If the algorithm predicts all the observations correctly, bias is equal to zero.

We use a number of supervised machine-learning algorithms: Logistic Regression (LR), Naive Bayes (NB), Random Forest (RF), Support Vector Machine (SVM), and a majority vote ensemble classifier with a bag-of-words representation of lexical feature vectors (VC). We run a total of twenty-five models (each classifier with unigram, bigram, and trigram and either term frequency or term frequency inverse document frequency weighting).Footnote ¹ For each of them we set aside the narratives from the 2013–2017 reports for an out-window testing set. We created a number of 5-year rolling windows.Footnote ² For example, W1: 2001–2005, W2: 2002–2006, … W8: 2008–2012 are used for in-windows. Each of these is tested using the fixed out-window testing set (2013–2017) after being trained for each rolling in-window.

If bias is introduced through changing standards alone, we expect both the global and Asia-only samples to show a large and consistent increase over time in the out-window accuracy. If bias is introduced through both changing standards of accountability and political biases, we expect to see an overall increasing trend of out-window accuracy in both samples, with larger variation around the out-window accuracy of in-window training sets around the period of the TPP (particularly post-2010). If only political bias is present, we expect to see a non-changing trend line of out-window accuracy in the global sample, but marked changes in the Asia-only sample, specifically around the TPP period.

Analysis

Changing Standards of Accountability in TIP Report Rankings

Figure 2 shows the changes that have occurred in the TIP Reports since they began in 2001. First notice the scope of reports: the number of countries covered expanded rapidly in the first five years, from 82 to 149, and by 2009 the reports covered nearly all countries. Secondly, the length of the reports has expanded dramatically. The median length nearly quadruples in the first eight years, and the median word count for later reports is consistently over 1,400. Additionally, the number of tiers has increased from three to four (1, 2, 2-watch list, and 3). Another important change is the professionalization of the reports over time. Early reports were short and included spotty coverage of countries, with little investment in production value. Later years include an introduction by the secretary of state and are more visually polished in their presentation and now include an additional tier option, 2-watch list.

Figure 2.

Changes in Coverage and Median Word Count of TIP Reports

Note: Shown are the annual median word counts of the TIP Reports, since 2001, and the number of countries covered. The left y-axis reports the median word counts and corresponds to the bar graphs. The right y-axis reports the number of countries covered, and corresponds to the thick black line.

Figure 3.

Accuracy and bias across windows – Global Sample

Figure 4.

Accuracy and bias across windows - Asia only

These figures indicate that reports have increased in length and geographic coverage, which is consistent with Clark and Sikkink (Reference Clark and Sikkink2013) and Fariss (Reference Fariss2014), who argue that increased information leads to changes in standards of accountability. The implication is that, because the narratives contain more information, a country that scored a 2 in 2002 would be qualitatively different from the narratives of the reports that led to a ranking of 2 in 2012.

Figure 3 presents results that include every report issued between 2001–2017. The top panel reports the out-window accuracy over time, the middle panel reports in-window accuracy and the bottom panel reports bias. If standards of accountability are changing over time in TIP Reports, then the early reports are substantively different from later reports. As a result, the accuracy of predictions should change when the mapping function changes over time. In short, if standards of accountability are changing, we would expect to see a large increase (or decrease) in out-window accuracy over time.

As can be seen in the top panel, the out-window accuracy does increase slightly, but it is relatively stable over time. Examining the majority vote classifier, out-window accuracy improves over the entire period by roughly 3 per cent. Recall that in comparison, Greene, Park and Colaresi (Reference Greene, Park and Colaresi2018) find that out-window accuracy improves by almost 20 per cent when examining a different set of human rights reports. The relatively small change in out-window accuracy in our results indicates that the standards of accountability in TIP Reports haven't changed much over time, if at all.

In the middle panel we report the in-window accuracy. Recall that this indicates whether we are over-fitting (or under-fitting) the models due to the level of difficulty of each in-window prediction. The middle panel also shows a relatively stable line, indicating that the models fit reasonably well and, more importantly, do so consistently. In the bottom panel we report the bias, which measures the average difference between the predicted and observed rankings. As the models improve performance (out-window accuracy increases), the differences between the predicted and actual rankings shrink. Thus the average biases get closer to zero. In the early period almost all classifiers are marked with bias, but they all improve over time and converge to zero. While the bias does approach zero, this early bias is quite low as well.

Have the accountability standards in the TIP Reports changed over time? These results indicate that this is not likely the case, as the reports’ mapping of narratives to rankings remains fairly consistent over time. This is particularly interesting compared to the analysis by Greene, Park, and Colaresi (Reference Greene, Park and Colaresi2018), who find that the PTS rankings, which are based on the State Department's Human Rights Reports, displayed significant changes in standards over time due to a shift in the informational environment. The TIP Reports, in contrast, don't show the same indications of overall change.