A NEW VERSION OF DYNAMIC IRT

The two most commonly used measures of human rights are the Political Terror Scale (PTS) and the CIRI Physical Integrity Index. Both data projects assign numerical scores to countries based on information included in annual reports produced by the US Department of State and Amnesty International. According to Fariss, these scores do not accurately record changes in human rights over time. Fariss presents the problem as a “changing standard of accountability.” Human rights scores may be inconsistent over time, because: (a) human rights reports have gotten longer, and more information may have led coders to make more negative assessments of human rights practices; (b) coders may have applied more stringent standards in more recent years; and (c) there may be new types of critiques included in more recent reports (Clark and Sikkink Reference Clark and Sikkink2013; Fariss Reference Fariss2014; Hafner-Burton and Ron Reference Hafner-Burton and Ron2009). For counterarguments and contrary evidence, see Richards (Reference Richards2016) and Haschke and Gibney (Reference Haschke, Gibney, Backer, Bhavnani and Huth2017).

Fariss suggested that possible biases in human rights data could be identified and corrected by estimating a latent index of human rights abuses. The assumption behind such an index is that, while the true level of human rights abuses is latent (i.e., unobserved), it is correlated with observable indicators of human rights. Various statistical techniques, ranging from factor analysis to IRT, would allow one to estimate a latent index based on observable values of several available indicators. Dynamic versions of IRT assume that criteria for recording the indicators could change over time. Fariss (Reference Fariss2014) introduced a unique version of IRT to estimate latent human rights scores.

Combining two types of indicators, giving some weight to each type, should produce latent scores somewhere between the scores obtained when using each type separately. However, in Fariss's specification, that is not the case. When we replicated his analysis (using Fariss's computer code) we found that the reported upward trend in human rights depended almost entirely on the inclusion of the mass killing indicators. No indicators of lesser human rights violations were necessary. We replicated the analysis replacing the actual values of all indicators of lesser human rights violations with randomly generated data and obtained an identical trend (Figure 2).Footnote ¹ When we repeated Fariss's computations using only the five indicators of mass killing, again we obtained latent scores that show an improving trend (Figure 3) similar to the trend reported by Fariss (Reference Fariss2014, 308). In contrast, when we replicated the analysis including only indicators of lesser human rights violations, there is no upward trend in the calculated latent scores (Figure 3). A simple (bivariate) OLS regression shows that scores generated using only records of mass killing can explain 88% of the variation in Fariss's scores.

Figure 4 illustrates that the trends in Fariss's latent scores for many specific countries also are similar to the trends in latent scores produced only from records of mass killing. For more country examples, see our online appendix (Figures A.1.1– A.1.4).

FIGURE 4.

Fariss's Scores (unfilled circles) Compared to Latent Scores Estimated Using Only Five Indicators of Mass Killing, 1949–2010

MODELING CHOICE MATTERS

The customary way to use IRT is to treat all observable indicators similarly in their relationship with the latent variable. This is how dynamic latent IRT models were used previously (e.g., Martin and Quinn Reference Martin and Quinn2002; Schnakenberg and Fariss Reference Schnakenberg and Fariss2014; Wang, Berger, and Burdick Reference Wang, Berger and Burdick2013). As applied to the debate in human rights, this approach would test the argument about the changing standards of accountability in human rights records by assuming that potentially all indicators are subject to such changes. Thus, they all could have variable intercepts. This approach leaves the possibility of a fixed intercept to be endogenously generated in the estimation.

When Fariss's model combines the two types of data in a single estimator, it treats the two groups of indicators differently (Fariss Reference Fariss2014, 305–306). It sets the mass killing indicators to follow a logistic regression with a fixed intercept (cut point) but allows the indicators of lesser human rights violations to follow an ordered logistic regression with variable intercepts for every year. Thus, the latent variable has to fit actual observations of the mass killing indicators without allowing a possible adjustment to the intercepts. With the indicators of lesser human rights violations, on the contrary, it is much easier for the algorithm to fit the latent variable as there are several dozen additional parameters (time specific intercepts) that could also adjust. Consequently, variation in the mass killing indicators generates the improving trend of the latent variable. This is why the actual values of the indicators of lesser human rights violations do not matter much.

Fixing the intercepts for mass killing indicators is necessary to obtain the scores showing an improvement in human rights. A model where the intercepts for all items vary across time produces latent variable estimates similar to those from a model where none of the intercepts vary. When we rerun Fariss's analysis allowing all indicators to have variable intercepts (in all other ways relying on Fariss's original computer code), we obtain the trend displayed in Figure 5, showing no human rights improvement. These results are robust to the choice of indicators included in the estimation— from four individual CIRI components to all 13 available indicators.

FIGURE 5.

Means of Dynamic Latent Physical Integrity Estimates, 1949–2010

Key: Unfilled Circles: Replication of Figure 3 in Fariss (2014, 308).Filled Circles: The scores are calculated with all indicators assumed to have a similar relationship to the latent variable; that is, all intercepts are allowed to vary. The assumption is that there also has been a change in the standards for recording mass killings.

MASS KILLINGS AS A BASELINE?

Fariss (Reference Fariss2014, 301) assumed that instances of mass killings, genocides, and political executions by oppressive regimes could “act as a consistent baseline” by which to compare the levels of variables measuring lesser human rights violations. As we demonstrated above, the crucial assumption is that there has been no change in the “standards of accountability” in records of mass killings. We prefer an alternative, less restrictive assumption–that there has also been a change in the standards for recording mass killings—as a starting point for thinking about ways to combine the two types of evidence.

The difference in the changes in the standards of accountability (S₁ – S₂) between the two types of records should be treated as an empirical question. It is likely that both lesser human rights violations and mass killing events are recorded more accurately now than in the past. There is a higher likelihood now that mass killings in remote places will be recorded. Coding rules for recording mass killings may be changing. Coders may have applied more stringent standards in more recent years. And coding rules across mass killing recording projects may be becoming more or less consistent with one another.

Though Fariss's model distinguishes between events-based and standards-based data, it is important to recognize that even mass killing indicators are standards-based. To record mass killings, scholars must make judgment calls that require coding rules determining such things as whether, and under what circumstances, to include (a) relatively low death toll events (b) deaths due to interstate and civil war, and (c) killings by nongovernmental actors such as paramilitaries.

Harff and Gurr (Reference Harff and Gurr1988, 365) developed relatively restrictive, explicit coding rules, only counting events in which “(a) many noncombatants were deliberately killed, (b) the death toll was high (in the thousands or more), and (c) the campaign was a protracted one.” Like the PTS and CIRI coders, they relied on the annual reports issued by Amnesty International and the US State Department, among other sources, to identify their cases. Other mass killing scholars like Rummel (Reference Rummel1994) applied less restrictive coding rules and recorded the greatest number of mass killings (Figure A3), even counting the United States as committing mass killings based on civilian wartime deaths in Korea and Vietnam.

Even if the standards for recording mass killings had been more consistent over time and among coders, those extreme events should not be weighted so heavily that they become a proxy or substitute for more direct measures of lesser human rights violations. Mass killings and lesser forms of human rights violations may not even be indicators of the same underlying concept—respect for physical integrity human rights. The most general definition of the right to physical integrity is “freedom from 1) state-imposed deprivations of life, 2) physical harm at the hands of state agents, and 3) state-imposed detention” (Hill Reference Hill2016). Lesser forms of physical integrity human rights abuses happen in almost every country every year. Mass killings are rare events, occurring mostly in failed and authoritarian states.

More commonplace forms of human rights violations may be following one trend while the worst forms are following another (Gutierrez-Sanin and Wood Reference Gutiérrez-Sanín and Wood2017). Mass killings may have declined because most governments have learned to selectively use the “lesser” forms of repression to achieve their objectives, while avoiding the worst and most notorious forms. Yet, Fariss's modeling specification forces the occurrence of the rare events of mass repression to strongly affect the human rights scores of all countries. Why should mass killing in Myanmar, for example, have any effect on the calculation of the scores of countries such as Denmark?

As a consequence of putting so much weight on rare mass killings, for many countries, the trends in Fariss's scores are nonintuitive, especially those produced through backward data extrapolation and imputation for the 1949–1975 period. These trends would not stand up to the level of scrutiny applied by Clark and Sikkink (Reference Clark and Sikkink2013), who criticized PTS and CIRI scores for particular countries to illustrate the possibility of an information paradox in recording human rights violations.

Particularly questionable are the trends in the scores between 1949 and 1975. No records of lesser human rights violations are available for this period. Moreover, for the majority of country-years between 1949 and 1976, the actual codes for all 13 human rights indicators before extrapolation were zero. Yet all country-years were assigned different latent human rights scores.Footnote ²

Despite research showing that democratic, economically developed countries have better human rights records (e.g., Poe and Tate Reference Poe and Tate1994), authoritarian, less developed countries often rank higher than well-established democracies. Fariss's scores place the United States in 1953 at the same level as North Korea. The US scores for the 1950s are well below the scores for South Africa, which, at the time, was building its apartheid regime. Between 1949 and 1970, the scores for the United States are significantly below the scores for Afghanistan and Soviet-satellite Mongolia (Figures A.2.1– A.2.5).