Respecting the facts

The assumptions on which some formal modeling rests are often so at variance with empirical reality that model results are dismissed out of hand by those familiar with the facts. The problem is not just unreal assumptions, for one way to build helpful models is to begin with stylized and perhaps overly simple assumptions, test the model's predictions, and then modify the assumptions consistent with a progressively more accurate model of reality. Yet these follow-up steps are too often not taken or left unfinished, with the result being a model that does little to enhance understanding or to advance the discipline.

One justification for “theories of unreality” is that realistic models are often so complex as to be of limited value. There is merit to this defense. An important function of formal modeling is to assist in identifying crucial quantitative and qualitative effects from those that are of minimal importance. However, the drive for simplicity can be taken too far. This conflict between realism and analytical tractability is not new or only a problem in the discipline of political science. Economics is instructive in this regard. In the late 1960s and early 1970s there was a revolution in macroeconomic research, which put great emphasis on the microfoundations of macroeconomic outcomes. Yet, as George Akerlof recently noted, “[T]he behavioral assumptions were so primitive that the model faced extreme difficulty in accounting for at least six macroeconomic phenomena. In some cases logical consistency with key assumptions of the new classical model led to outright denials of the phenomena in question; in other cases, the explanations offered were merely tortuous.”¹⁷

The use of simplifying assumptions is in principle a virtue and remains so when such simplifications do no harm to overall predictive accuracy. However, this does not mean that formal modeling should proceed without regard to glaring factual contradictions in its foundational or situation-specific assumptions. Rather, formal modelers must be especially careful to make sure that they test their models in situations that go beyond the circumstances that suggested the models, for it is there that simplifying assumptions are likely to lead to difficulties.

In this situation, the role for EITM is to enhance the formal analysis by adding the necessary empirical results that lead to more accurate representations. When this approach is adhered to, researchers can ensure that simplifications do not compromise scientific rigor by leading to models that only predict the phenomena that led to their development in the first place.

Theorizing by proverb

For case study analysis, Riker's admonition still applies when practice degenerates into theorizing by proverb. While such studies can be richly illuminating, it is sometime hard to know where empirical tests leave off and the researcher's perspective or biases begin. Nor can one know what salient facts might be left out, or whether apparently strong findings represent idiosyncrasies of time and place rather than powerful general tendencies.

When a researcher goes from an empirical puzzle to a theory and then to hypothesis testing using the same observations, there is no guarantee that the observed relations support the theory. Indeed, they cannot be said to support the theory if it was inductively derived from the case studied. Yet, case studies often produce very useful theoretical propositions that perhaps reflect V.O. Key's observation of 50 years ago that “uniformities of behavior, at least in the aggregate, do turn up with astonishing regularity.”¹⁸

Consistent with Key's assertions, we think a researcher's task is to take the reliably measured aspects of behavior and then attempt to provide valid generalizations. If a researcher can point to a set of conditions or circumstances where a variable X occurs and a variable Y changes in response, then it is possible to construct theories that serve vital purposes. For policy makers, who rely on theories and findings to take action, it is not just a question of seeing a specific consequence to their decision and action; rather, it is also about knowing the circumstances in which a policy choice is likely to produce a desired outcome, as well as when it is wise to take an action.

In non-systematic case study analysis there is a distinct danger that in generalizing to a specific causal relation, the researcher will create proverbs rather than theories. Herbert Simon warned long ago that proverbs, for all their uses as rhetorical devices, fail to foster scientific progress:

The danger exists that unless carefully and systematically undertaken, the case study method lacks sufficient power to separate out the many plausible rival explanations. Many factors may cause variation in a dependent variable, not just the plausible mechanism for which researchers might effectively argue. When we question explanations rather than accept them because of surface plausibility, we are able to find out which of the many potential factors matter most and how causes are conditional by context.

There are other problems as well.²⁰

William Riker provides guidance on this matter. In his book, Liberalism Against Populism, he examines the concept of “democracy.” Rather than settle on a single definition, he asserts, “we cannot go to a unique authoritative source for a definition.”²¹

To take Riker's approach one step further, we are arguing (using EITM) that when a researcher formalizes a model and then collects the necessary data to link the model with empirical outcomes, s/he has taken a crucial step toward identifying causal linkages. That, after all, is the ultimate goal of scientific endeavors.

Current and common applied statistical practices

When applying statistics, analysis too often turns into garbage-can models and jerry-built statistical contraptions with little if any relation to the theory. In A Primer of Statistics for Political Scientists, Key makes the case for the utility of quantitative analysis:

However, in the current research environment where data mining, garbage-can specifications, and statistical patches dominate, the ability to harness the attributes noted above—particularly generalizability—is compromised.

To be more specific, the following ratio is the subject of much attention by applied statistical analysts because it is the basis for which “theories” survive or perish:

This ratio is commonly referred to as a “t-statistic.” It is the “truth” that most applied statistical analysts are concerned with, and it can be confounded by influences that shift the numerator (b) in unforeseen ways. The denominator, the standard error [s.e.(b)], also is susceptible to numerous forces that can make it artificially large or small. In either case, avoiding false rejection of the null hypothesis (Type I error) or false acceptance of the null hypothesis (Type II error) is imperative. While the concern with Type I and Type II errors should be of prime importance, that unfortunately is not usually the case. Instead, the focus is on the size of the t-statistic and whether one can get “significant” results.

The first tendency in trying to achieve “significant” results is the practice of data mining. Some political scientists put data into a statistical program with minimal theory and run regression after regression until they get either statistically significant coefficients or coefficients that they like. This search is not random and can wither away the strength of causal claims.

A second practice is that many studies degenerate into garbage-can regression or garbage-can likelihood renditions. By a garbage-can regression or likelihood we mean a practice whereby a researcher includes, in a haphazard fashion, a plethora of independent variables into a statistical package and gets significant results somewhere. But a link with a formal model could help in distinguishing the variables and relations that matter most from those that are ancillary and, probably, statistical artifacts. More often than not there is little or no attention paid to the numerous potential confounding factors that could corrupt statistical inferences.

The first and second practices lead to the third—statistical patching (i.e., the use of weighting procedures to adjust the standard errors [s.e.(b)] in the t-statistic ratio above). Data mining and garbage-can approaches virtually are guaranteed to break down statistically. The question is what to do when these failures occur. We think the answer to this question involves returning to the drawing board and developing new and more accurate specifications (with the aid of a formal model). However, this is not the usual practice.

Consider again the t-statistic above and the incentive to achieve significant results. One can do this by using statistical patches that have the potential to deflate the standard error and inflate the t-statistic, which, of course, increases the chance for statistical significance. Unfortunately, with the advances in computing power and the simplification of statistical software packages, this practice is only a click on the “enter” key away.

There are elaborate ways of using error-weighting techniques to “correct” model misspecifications or to use other statistical patches that substitute for a new specification. For example, in almost any intermediate econometrics textbook²⁴

In principle, there is nothing wrong with knowing the Omega matrix for a particular statistical model. The trouble comes in how one uses it. Consider that Omega matrices remove or filter residual behavior. The standard error(s) produced by an Omega matrix should only serve as a check on whether inferences have been confounded to such an extent that a Type I or Type II error has been committed.

Far too often, however, researchers treat the Omega weights (or alternative statistical patches) as the result of a true model. This attitude hampers scientific progress because it uses a model's mistakes to obscure flaws. Achen points out the scientific problem with these empirical practices:

The recent controversy over the United States' presidential election models illustrates the problem with current empirical practices. Despite the fact that Tse-min Lin has demonstrated the overall lack of robustness of the most widely used models, there has been little movement in reformulating key features of these models.²⁶

Case study analysis can also illuminate weaknesses in current applied statistical practice as they pertain to the 2000 presidential election. Henry Brady notes flaws in John R. Lott's statistical analysis of Florida voter turnout for the 2000 presidential election.²⁷

Using back-of-the-envelope “case-study” kinds of calculations, including census data, uniformity of voting during the last hour (when the results were declared), media exposure on who would have heard the early call, and other factors, Brady's analysis shows that the figure is more likely to be between 28 and 56 votes lost for Republicans. This is an example where “causal-process observations demonstrate that it was highly implausible for the media effect suggested by Lott's analysis to have occurred.”²⁹

If one were to summarize the problem here, one would conclude that the intellectual drift from the virtues of empirical practices means that statistical technique has come to dominate the practices used to help identify causal linkages. But statistical technique alone cannot test generalizations of observed political behavior. Once again, the solution is to find ways to link statistical techniques with formal theory:

Growth in partisan identification

Party identification research has clearly occupied an important place in political science.³³

We know that within advanced industrial democracies, there is generally not a high level of political and social awareness or political and social participation. The primary accountability mechanism—voting, for example—is usually limited to well-educated and prosperous segments of the society. Moreover, the typical voter tends to be ill-informed about political, social, and economic issues and about candidates' positions on these issues. Few voters take part in politics in ways other than voting, and their voting behavior seems mainly determined by group loyalties, such as social class, religion, ethnic affiliation, and above all, party identification.

How did we learn this? During the 1940s and 1950s the development and application of new social research techniques such as survey research (and subsequent computer analysis) provided new and sometimes startling insights into political behavior. When micro-analytic research methods began to be used in political science, it became possible to study the dynamics of individual voting behavior.

The rational actor model of representative democracy—the notion that citizens make rational decisions, which are the basis of their voting behavior—was challenged by what was discovered. Many paradoxes emerged. For one, political attitudes and voting behavior are determined, in part, by the social milieu into which one is born and the context in which one is raised. Although there are good historical reasons why certain religious, ethnic, or geographical groups vote as they do, in succeeding generations those political affiliations become fixed and are transmitted from one generation to another. They cease to be the result of rational analysis or current events.

Realizing this, political scientists began to study voting behavior more in relation to political socialization and less in relation to issue conflicts. In 1969, in his seminal article, “Of Time and Partisan Stability,” Philip Converse advanced the theory that strength of party identification (and voting behavior) is primarily a function of intergenerational transmission plus the number of times one had voted in free elections.

Converse and Georges Dupeux had found in their earlier comparative study of France and the United States that 75 percent of Americans identified with a political party while only 45 percent of French did so.³⁴

In his subsequent research, Converse presented a theoretical framework in which he assumed that few of the fathers (in France) identified with a party. Proceeding with that conjecture, he reasoned that if the 50 percent of French voters who did not know their father's party identification became party identifiers, then the later generations would be more like Americans in that their partisan identification rates would be significantly higher. In short, France would become more like the United States, Great Britain, and Norway in party identification in ensuing generations and less like Germany, Italy, and earlier generations in France. To support this conjecture Converse made use of a mathematical technique: the Markov chain (see the Appendix).³⁶

With the assistance of the Markov chain model, the theory has the following dynamic: In the first election in a country, almost no one would identify with a party. By the time the second-generation voters were of voting age, 50 percent of them would express identification. Then, gradually, party identification would rise to a stable level of about 72 percent. Converse assumed that the relatively low level of party identification in France resulted from the fact that the vote was extended later there than in the United States, and that French women were not given the vote until 1945.

Converse then noted that as individual voters become older, they identify more strongly with one party. This is not strictly a result of age, but of how long the person is exposed to a party by being able to vote for it. Combining these two findings, he came up with the theory that the strength of a voter's party identification can be predicted from two factors: The number of years the person was eligible to vote, and the likelihood that the individual's father had identified with a party.³⁷

[bull ] When elections are first held in a country, little party identification is observed. (The pattern found in mature democracies—that older people have stronger party identification than younger people do—is not observed in the early days of a democracy.)

[bull ] If elections are interrupted, then a country's level of party identification declines.

[bull ] “If the transition rates for all countries are roughly the same as for France and the United States, then party identification levels in all electoral democracies should converge over a few generations toward a single value of about 72 percent.”³⁸

To summarize, Converse started with a fundamental puzzle that had important societal consequences, and then analyzed data. He then used a mathematical apparatus to generalize the case beyond the two-country comparison. At its most rudimentary level, he combined all the fundamental components of EITM-like research to enhance knowledge. In the ensuing years, a new generation of research has built on this beginning to provide more elaborate formal explanations about how individuals learn about political parties and events and adjust their partisan attitudes.³⁹

The Phillips curve

Beginning in the early 1960s, macroeconomic policy makers (or their advisors) constructed statistical models to determine the effects of monetary and fiscal initiatives on unemployment and output. The scientific basis for this emphasis on policy “fine tuning” centered on the research of A. W. Phillips. Phillips showed empirically that there was an inverse relationship between nominal wages and unemployment: higher unemployment was associated with lower wages, while higher wages were associated with lower unemployment. This relation was extended to incorporate a trade-off between inflation and unemployment. From the late 1950s to the late 1960s most economists assumed that there was a stable trade-off between unemployment (or output) and inflation. In fact, this stable relationship could be graphically demonstrated on what is now called the Phillips curve.⁴⁰

This assumption of a stable relationship had powerful appeal to policymakers. One could observe the corresponding rates of inflation and unemployment on the Phillips curve and formulate an appropriate monetary and fiscal policy to stimulate or contract the economy. The United States' experience with low unemployment and inflation in the early to mid-1960s suggested this was sound policy.

This optimism was to be short lived. Policy analysts noted by 1970 that their conditional forecasts were incorrect. The negative empirical relation between inflation and unemployment (or output) that Phillips found, and that was implicit in the fine-tuning policies, disintegrated in the 1970s. The prevailing conditions of high inflation and high unemployment, which came to be known as stagflation, defied the characterization offered by Phillips.

The policy failure was preordained, in large part by the failure to reconcile the empirical relations with standard models in economics. In the latter part of the 1960s, Friedman and Edmund Phelps, using both formal and non-formal theoretical arguments, demonstrated that the underlying Phillips curve assumptions were inconsistent with basic economic theory.⁴¹

In particular, Friedman and Phelps both emphasized that the Phillips curve is inaccurate when the public's expectations of inflation are taken into consideration. They argued that a stimulative policy could lower unemployment for a brief time if workers set their wage demands too low. This occurs if workers underestimate future inflation, but Friedman and Phelps reasoned that workers could not be fooled for long. They would eventually correct this mistake. During the transition to correcting the inflation expectation error, however, unemployment falls because wages have not kept pace with inflation, and employers can better afford the cheaper labor during this period.

The scientific consequence is that expectation errors on inflation are important in determining the level of unemployment. Friedman reiterates this point and then draws the policy implications:

As a result, there could be no stable or predictable Phillips curve trade-off. The policy implications and social implications were equally clear. If policy makers, for example, attempted to reduce the existing rate of unemployment, the result would be more volatile swings in monetary policy and, by implication, in output, prices, and unemployment. Indeed, such policies would eventually be self-defeating and would instead create a combination of higher unemployment and higher inflation, or what came to be known as stagflation.

In the last 20 years, this new theory has been the dominant view held by policymakers and policy advisors of both political parties in the United States. While some can ascribe the elimination of stagflation in the United States to good luck, there is also evidence that scientifically informed policy practices were a factor.⁴³

All of this is not intended to applaud the discipline of economics at the expense of political science. It is simply to note that scientific progress in economics is testimony to the power of merging formal and empirical analysis. Economists have made numerous contributions in developing theories robust enough to guide policy actions. Still, many issues in the macroeconomic policy area remain unresolved. Perhaps the most interesting to political scientists are those that center on the interaction of the public, elected officials, policy making institutions, and the ways in which these interactions can be structured to yield desirable policy responses. Political science can make progress in solving the conundrums that these interactions pose by constructing and progressively refining models that have well-identified relations with a direct empirical link.

Testing strategic interaction in international relations

A third example linking formal modeling with empirical analysis explores various aspects of relevance to the conduct of foreign policy and the use of military force. In particular, Curtis Signorino examined “the machinations of state leaders trying to achieve their foreign policy goals through sometimes peaceful but often violent means.”⁴⁴

Of central importance are the strategic behaviors and the strategic interactions between the competing states. As Signorino points out, “States do not act in a vacuum. Decisions to engage in arms production, enter into alliances, and go to war are not independent of the expected behavior of the other states in the system. Their calculations are based on what they expect other nations are currently planning to do or how they may respond to particular actions.”⁴⁵

For this type of research question, where strategic interaction is involved, game theoretic modeling offers important attributes. Additionally, Signorino was interested in linking game theoretic predictions with empirical tests. Signorino adopted an EITM approach with the following justification:

To link the formal game theoretic model with an empirical test, Signorino used Quantal Response Equilibrium (QRE) analysis. Originally developed by Richard McKelvey and Thomas Palfrey,⁴⁷

This second feature is of some importance. The inclusion of errors, while not only consistent with the idea of agents having bounded rationality (which is assumed), also allows for feasible statistical estimation.⁴⁸

Signorino did a series of tests to determine if QRE improved predictive accuracy. One test summarized here centers on his comparison of a strict empirical model (Logit) with data (generated via Monte Carlo) based on QRE. With his Monte Carlo experiments, he found that when the empirical model fails to incorporate strategic behavior, the statistically significant results give incorrect policy advice. Signorino pointed out the policy ramifications of following a naive, strictly empirical approach:

However, if this policy prescription is followed it would increase the probability of war (and not reduce it) because the raw empiricism does not capture the true underlying probability of war in the Monte Carlo analysis.

Signorino also extended his Monte Carlo “experiments” to other more sophisticated models that include “balance of power” concerns or whether two nations jointly value war. He found that these improvements in sophistication, although estimated without consideration of strategic interaction, also provide poor policy advice. He concluded this portion of the analysis with the following thoughts that pertain to the issue of blindly using strictly empirical procedures and ignoring the richness a formal model can provide. This choice contributes to noncumulation:

Regulatory policy delay

Daniel Carpenter's research on regulatory policy offers a different type of policy relevant example where EITM is undertaken. Carpenter poses the question, “Why do bureaucrats delay?” Why do regulatory choices made under identical administrative procedures exhibit highly varying decision times? He studied the histories of 450 new drugs (“new chemical entities”) reviewed by the United States Food and Drug Administration (FDA) between 1977 and 2000. Guided by an optimal stopping model—how can we predict the time horizon for approval of drugs?⁵²

Carpenter found differences in the review time. Moreover, he discovered that factors that should not, given the FDA's mandate, make a difference in review time seemed to matter a great deal. For example, when Carpenter counted how many Washington Post stories mentioned a specific disease in any given year, and then computed how quickly new drugs for that disease won FDA approval, he found that the frequency of disease mentions seemed to relate directly to approval time for specific drugs. This occurred regardless of the seriousness of the ailment, its incidence in the population, the cost of treating it, the availability of other medicines targeting the disease, and other variables that measured the potential value of the drug.

Along with the influence of the print media, Carpenter also reveals equity issues that should concern policy makers. His model and his empirical tests together produce a set of interesting research findings with important social implications. The interaction between the optimal stopping model and his empirical results illuminate not just the degree of bureaucratic delay, but also the forces responsible for it. Consequently, this research not only contributes to an understanding of the linkages between the causes of, and implications of, delay in the drug approval process, but also provides findings for policymakers entrusted with formulating health policy.⁵³