Media and Political Treatment of the Market

Perhaps the defining characteristic of the proposed market saga was the lack of any high-level political debate. Practically every politician that weighed in on the matter opposed it. Wyden referred to the project as “make-believe markets trading in possibilities that turn the stomach,”¹

while Dorgan labeled it “unbelievably stupid.”² Senator Hillary Clinton (D-N.Y.) was widely quoted as calling the program “a futures market in death.”³

Unlike other debates, the criticism did not stop at party lines. Senator John Warner (R-Va.) called the program “a rather egregious error of judgment,” and Senator Pat Roberts (R-Kan.) said that it “defies common sense. It's absurd.”⁴

Even Deputy Defense Secretary Paul Wolfowitz joined in the criticism, claiming that he had only learned about the plan himself by reading the newspaper and stating that “I share your shock at this kind of program. We'll find out about it, but it is being terminated.”⁵

The response by the press was similarly unbalanced, although not quite to the extent of the response by politicians. A search of Lexus-Nexus from the end of July until late August turned up 115 pieces on the policy analysis market (hereafter PAM).⁶

Using a very blunt coding scheme, we broke down articles into three categories: “opposed,” “neutral,” and “in favor.” We cast our “in favor” net as wide as possible, including any pieces that had something positive to say about the proposed market. We also broke down articles into news or opinion pieces, with the latter comprising editorials, op-eds, letters to the editor, and columns. News articles were assigned to the “opposed” or “in favor” category only if they had a strong tone.

As Table 1 demonstrates, only 16 articles were in favor of PAM. This stands in marked contrast to the 56 pieces that opposed the project. While there were 43 articles in the “neutral” category, the vast majority of these were news articles. Of the 67 opinion pieces, 49 were opposed to PAM, with only 14 presenting arguments in favor.

This pattern was even more pronounced across the 25 editorials, with titles such as “The Pentagon's Sick Take on Risk Strategy”;⁷

“Dr. Strangelove Caught in Bizarre Betting Parlor”;⁸ “A Brainless Scheme”;⁹ and, quoting Dorgan, “Unbelievably Stupid.”¹⁰ Indeed, only one paper—the New York Post—presented an editorial in favor of PAM.¹¹

The primary media attacks paralleled those made by politicians, with the system being criticized as “grotesque,”¹²

“morally reprehensible”¹³ and the result of the fact that “social scientists and the policy wonks who funded them [had grown] deaf to ethics and moral responsibility.”¹⁴ Some argued that such a market could actually encourage terrorist attacks by adding a profit motive: “Faced with a large pot of money and having few scruples, Gordon Gekko types might be tempted to influence events, even in friendly countries. Alliances don't matter when there's a buck to be made.”¹⁵

Although in the minority, a number of pundits did point to specific concerns about the market's ability to function as desired rather than simply condemn it wholesale. Both Bill Day of the San Antonio Express-News¹⁶

and Jeff Brown of the Houston Chronicle¹⁷ wondered whether participants in the proposed market possessed enough information concerning the likelihood of future terrorist attacks to make it an effective mechanism for predicting such events. Senator Dorgan, writing in the Washington Post, noted that markets have also had spectacular failures, such as the stock market's inability to foresee the collapse of the Enron Corporation. Economist William Sharpe raised the question of whether the market would have been truly competitive due to the presence of inside information. In an interview with the Boston Globe he noted, “I'd worry that someone betting on when Weapons of Mass Destruction will be discovered in Iraq might in fact already know where such weapons are.”¹⁸

One of the most interesting features of the various attacks on PAM was that hardly any contained a detailed description of the history of the program. The most notable exception was an article by Bradley Graham that appeared on the front page of the August 1 edition of the Washington Post. To judge from the vast majority of press coverage, PAM was designed for the sole purpose of allowing people to bet on the likelihood of future terrorist attacks. Graham reported, however, that

Graham also identified another interesting feature of PAM. The original plan of at least one of the companies hired to build the market, Net Exchange of San Diego, had been to limit market participation to “intelligence analysts and others in the U.S. government. But the idea of setting up an internal market ran into legal prohibitions against moving money among agencies funded under separate congressional appropriations.”²⁰

It is likely that PAM will go down in history as the government program with the shortest lifespan from entering the public consciousness to complete and utter termination. Clearly, Dorgan and Wyden had struck both political and public nerves. The question remains, however, whether they had actually helped public policy by doing so.

Analyzing the Institution

So what issues would have arisen in a better debate? In this section, we present three topics for assessment: the ability of markets to predict future events; the danger of manipulation of markets to affect non-market goals by market participants; and the effect of “noise” in markets on their efficiency. While none of these topics is as sexy as the prospect of Gordon Gekko types building a nest egg by correctly predicting mass destruction, all three point to important questions for assessing the functionality of the proposed PAM and, consequently, solutions for making a hypothetical PAM in the future work better.

First, the idea of using a futures market to generate predictions is not new. Following Friedrich Hayek,²¹

the predominant view among economists is that market prices efficiently summarize the private information of individual traders. Hayek's perspective has motivated the creation of markets that trade commodities that pay off if a particular event occurs. For example, since 1988, the Iowa market has traded commodities corresponding to U.S. elections. In their analysis of this market, Robert Forsythe and his co-authors²² conclude that “the market worked extremely well, dominating opinion polls in forecasting the outcome of the 1988 presidential election.”

As some of the participants in the debate over PAM noted, most field experiments of this form have been quite satisfying.²³

However, political markets do not always perform well, and it is instructive to see which ones perform poorly. Ben Jacobsen and his co-authors study a political stock market in the Netherlands and find the market's predictions to be much less accurate in multiparty races than in American elections.²⁴ The authors rule out many potential reasons for the poor performance and forward one explanation, which speaks directly to the proposed market on terrorism: If traders over-weigh private information, then small parties will be overvalued and larger parties will be undervalued. In proportional representation systems with many parties, this effect is more pervasive than in U.S. elections. A market with futures contracts that pay off if terrorist activities occur is closer to a typical European proportional representation election than an American two party election. So while the few proponents of PAM that offer supporting evidence from the Iowa market are hitting on an important point, closer inspection of these field experiments suggests that success in the United States may overstate the predictive value of markets in terrorist settings.

While the media tended to draw connections with the Iowa political markets, potentially more relevant comparisons existed in the experimental markets at Hewlett Packard (which, incidentally, were created by some of the same scholars who were involved in the DARPA project). The H.P. markets traded futures related to the sales of the company's products and its overall profitability. In contrast to the Iowa markets, which were open to the public, the H.P. markets were expressly designed to involve a small number of carefully chosen traders. “They were selected specifically from different parts of the business operation because they were thought to have different patterns of information about the targeted event. These patterns of information, including market intelligence, specific information about big clients, and pricing strategies, were in need of aggregation.”²⁵

Traders could also track the trades made by individuals (although their real identities were suppressed). Thus, if a particular trader was repeatedly buying an asset, the other traders could observe this. Kay-Yut Chen and Charles Plott²⁶ demonstrate that the HP markets tended to outperform official forecasts and, in some cases, were remarkably accurate.

Early discussion of the terrorism market also focused on the consequences of having open registration for participation, which would potentially allow terrorists to profit from their activities. Senator Barbara Boxer (D-Calif.), in particular, feared this scenario because “knowing they were planning an attack, [terrorists] could have bet on the attack and collected a lot of money.”²⁷

Some suspected that the real goal of the market was to gain information by luring knowledgeable collaborators out of the fold with the temptation of earning a quick profit, with one editorial claiming that “It was supposed to be a sting operation: Presumably, terrorists would know where they'd be attacking and what leaders they'd be gunning for, and would tip their hand on the Internet in hopes of making a killing on the killings.”²⁸

But on closer examination, concerns about terrorists making money seem far less pressing than the concern that they would use the market to foil the intelligence community. As Dorgan noted, “the market invited manipulation.”²⁹

Given the willingness of terrorists to expend resources—including lives—on their missions, it is not far-fetched to imagine that they would be willing to take a financial loss to mislead the Pentagon (Sowing havoc among the infidels: PRICELESS). At the least, this could lead to valuable resources being wasted; in the worst case scenario, it could increase the likelihood of a successful attack.

Finally, a related issue pertains to the relationship between the number of traders, the number of informed traders, and the amount of money available to traders. Information aggregation requires that individuals have information. The markets that predict electoral outcomes have been successful precisely because there is information (polls, news reports, et cetera) upon which participants can draw. The H.P. markets were successful because the traders possessed relevant information. But only a small handful of people are likely to have any advance knowledge of a terrorist attack. Consequently, in an open market, most traders will be doing little more than guessing. Even worse, the “noise” produced by these bets will dilute whatever useful information might have been generated by the few traders with relevant knowledge. While markets tend to be less susceptible than elections or polls to this dilution effect, the market force to close arbitrage opportunities will not be sufficient to offset the dilution effect if the cap on the amount of money an investor can stake is too small.

Consider as an example the game show Who Wants to Be a Millionaire? Contestants on the program are allowed to poll the audience for help in choosing their answer to a trivia question. Imagine if the host of the show posed a math problem for which only three mathematicians knew the correct answer. If the contestant could question just these mathematicians, she would learn the appropriate response. However, if the contestant surveyed the entire audience—even if it contained these mathematicians—then the opinions of those who did not know the answer would overwhelm the few knowledgeable experts, and the aggregate results of the survey would be practically useless. Terrorist attacks that depend on the element of surprise are, by their very nature, similar to questions whose answers are only known by a select few.

However, if instead of polling the audience the contestant was offered an alternative lifeline—create a market and watch the trading—the result would be different. If one of the mathematicians was in the audience, she would continue to buy shares of the option that represented the correct answer until the price reflected her subjective confidence that she knew the answer (or until she hit the cap on trading). If the cap on trading were small, then the mathematician's knowledge would not produce a noticeable spike in prices. If the cap were large, however, the mathematician's knowledge would be reflected in the price of the commodity. Alternatively, with a small cap and more mathematicians, the market would also do better. This analogy suggests that for hard problems in which a very small percentage of the population has real information, ensuring that a large percentage of market traders are potentially informed or ensuring that spending caps are not too small are important issues to consider.

From this brief analysis of the incentives, we can draw several conclusions. First, a PAM limited to those with expert opinions would likely have done a better job anticipating future developments than one which invited broad public participation. Second, some form of entry requirements designed to exclude likely terrorists would minimize the manipulation concerns. Third, allowing traders to cast rich enough bids so that one confident person can influence the market will help reduce the dilution effect. Fourth, contracts on a small number of events may limit the market's exposure to problems that surfaced in the multi-party proportional representation field experiments.

Based on this analysis, the PAM proposed by DARPA—which would have had open participation; the potential for large numbers of contracts on very specific events; and, most likely, low caps on trading—clearly had serious flaws that would have inhibited its value as a predictive market. But by thinking about how the structure/rules of the market might influence the behavior of traders and, ultimately, the market's effectiveness, institutional analysis yields several amendments to the proposal. With these changes, it seems possible that a PAM might provide the intelligence community with useful information. However, the value of such an amended PAM should not be assessed in a vacuum. Instead, we should contrast it with alternative means for filtering information.

Comparing the Market to an Alternative

Earlier, we touted the potential relevance of comparative institutional analysis. One of the striking features of the brief debate over the PAM was that this comparative dimension was almost completely absent. Since no one viewed the market as a replacement for the current intelligence community, the relevant question to consider is whether it is likely that PAM might offer information that would be missed by current intelligence gathering procedures.

Bruce Berkowitz and Allan Goodman present a lucid description of the status quo intelligence gathering institutions:

Most scholars would agree that hierarchies like those found in the intelligence community are capable of missing available information. Typically, information only makes its way up the chain if, at each level of the hierarchy, the supervisor chooses to pass it on. Accordingly, an unpopular speculation can easily be blocked by one of the many links in the chain. Moreover, even if the relevant information reaches the top of the hierarchy, the actual evaluations and policy prescriptions are made by a small set of individuals.

The recent Congressional inquiry into the 9/11 terrorist attacks provides an apt example of the potential shortcomings of such methods of information aggregation. The report notes that, as a whole, the intelligence community “did have information that was clearly relevant to the September 11 attacks, particularly when considered for its collective significance.”³¹

In particular, various parts of the intelligence community had received information suggesting that aircraft might be used as weapons, that the possibility of terrorist attacks in the United States was increasing, and that Osama bin Laden was sending students to the United States for aviation training. Despite these warning signs, the report concludes that “Some significant pieces of information in the vast stream of data being collected were overlooked, [and] some were not recognized as potentially significant at the time and therefore not disseminated…. For all these reasons, the [i]ntelligence [c]ommunity failed to fully capitalize on available, and potentially important, information.”³² By contrast, markets allow individuals the ability to act on relevant information unfettered by the constraints of whether someone higher up the chain of command thinks the information is important. As The Economist noted when discussing the H.P. markets, “markets of this sort offer a much better way of gleaning valuable information. That is because traders have a strong incentive to get their predictions right (rather than say what they think their managers want to hear).”³³

As noted above, as long as caps are high enough, even one individual can significantly affect the price of a commodity. In the existing hierarchical world, agents at both FBI headquarters and the New York field office chose to ignore the Phoenix Electronic Communication warning of the “inordinate number of individuals in investigative interest” seeking flight training in Arizona³⁴

and, consequently, the information went nowhere. Had a market been in place, however, the authors of the Phoenix memo might have been able to affect the price of a related futures contract, something that would have been noticeable throughout the intelligence community, including to those agents who had been warning about the possibility of aircraft being used in terrorist attacks since 1994.

James Surowiecki³⁵

makes a similar comparison:

Clearly, U.S. intelligence gathering capabilities in recent years have not been beyond reproach. Would a better designed PAM have helped supplement these capabilities? Due to the policy debate that emerged in July of 2003, we'll probably never know the answer to that question.