The Pentagon, global warming, and the problem with reporting on gaming and scenario-based planning

A leaked report from the Pentagon on the subject of global warming has caused a stir recently. The article in the Guardian/Observer is headlined:

Now the Pentagon tells Bush: climate change will destroy us

· Secret report warns of rioting and nuclear war
· Britain will be 'Siberian' in less than 20 years
· Threat to the world is greater than terrorism

Pretty scary sounding stuff. And coming from the Pentagon? That seems to give it even more credibility. But this incident is a pretty good example of a real problem that can affect gaming and other scenario-based forms of strategic planning. From the text of the article:

Climate change 'should be elevated beyond a scientific debate to a US national security concern', say the authors, Peter Schwartz, CIA consultant and former head of planning at Royal Dutch/Shell Group, and Doug Randall of the California-based Global Business Network.

An imminent scenario of catastrophic climate change is 'plausible and would challenge United States national security in ways that should be considered immediately', they conclude. As early as next year widespread flooding by a rise in sea levels will create major upheaval for millions.

I haven't read the report itself. It does not appear to be available beyond the snippets contained in press coverage. But several points about the above quote should stand out, and have been completely lost in the coverage. First, Peter Schwartz is best known for his work on scenario-based planning, as I mentioned here when discussing his book, The Art of the Long View. I don't know anything about Doug Randall, but I believe the Global Business Network where he works was started by Schwartz after he left Royal Dutch/Shell. That, and the language in the next paragraph about the "imminent scenario of catastrophic climate change" suggest that the report itself was written as a form of scenario-based planning document, probably along similar lines to the sort of thing found in Schwartz's book.

If that is the case, the press is getting the story very wrong. A scenario, in this process, is a tool for considering possible futures and open planners' eyes to potential shifts away from their main-line analysis, not for predicting specific events. By considering multiple plausible scenarios, the planning organization can work to be prepared for any one of them, and can get a better understanding of how certain actions that seem desirable in one scenario might prove problematic in another. These scenarios are researched and detailed, but are not scientific documents or predictions, and it is wrong to present them as such. They are, in the words of the book's introduction, "designed (one hopes) to bring forward surprises and unexpected leaps of understanding." They are also generally presented in groups, so that the planning organization can look across different plausible futures.

So, what does that mean for this report? It sounds like the events picked up on by the Guardian/Observer were part of one particular scenario within a broader document. Certainly, the point of the report might have been that there needs to be some serious consideration given to the potential for disasters like the ones in the scenario, but that's a far cry from saying that these specific things will happen. Things like them could happen, and it behooves any huge organization engaged in long-term planning (such as the Pentagon) to consider what the actions they take today would look like in a variety of futures, including one in which climate change effects dramatic shifts on world security concerns. That seems like a reasonable statement. But that's not the story the press told in this case.

This highlights a concern that could arise in the context of gaming out possible futures, or in other efforts to use scenario-based planning. It can be important to consider the extreme case, and looking for radical discontinuities and unexpected shifts is difficult work, so a process involving scenario-based planning could produce plenty of dire-sounding projections that might be reported as forecasts rather than what they really are: parts of a larger project of strategic planning. This lack of context is a problem, because it might inhibit the use of these helpful planning tools for fear of leaks or bad press. It's important for the Pentagon and the rest of the national security community to be open to multiple possibilities about the future, and this is an effort that should be encouraged. This is not to say that I think there's a simple fix here; it's natural for the press to jump on things that seem controversial or explosive, but it might be just that kind of controversial scenario that is necessary to promote a better understanding of the deep uncertainty that surrounds the future. This seems like a genuine problem for this kind of planning effort.

Orbis Forum on Political and Military Gaming

In 1984, the journal Orbis published a series of articles on the state of political-military gaming. The exchange is unfortunately not available for free online, but it appeared in the winter 1984 issue, which is probably widely available at university libraries. As a group, these articles are an excellent source of reflections on gaming by some prominent practitioners and theorists.

This is a long post, and it covers four separate articles. If I had known it would take me so long to get around to finishing it, I would have split it up. Lesson learned.

Lincoln Bloomfield was a pioneer in political-military gaming, particularly in his work organizing research-oriented political exercises (referred to as POLEXes) for senior-level decisionmakers at MIT in the late 1950's and throughout the 1960's. All of the POLEX research results are apparently unclassified, but I can't find any reference to them on the web. In his Orbis contribution, Bloomfield discusses the history, practice, and realism of political gaming, based on his own experience. His explanation of how free-form gaming requires experienced participants in order to generate useful policy results is particularly well laid out:

No satisfactory model yet exists of the national security decisionmaking system of the United States (or any other country), nor of the larger system of interactions and perceptions that connects them. Thus one either specifies a crude, oversimplified model or relies on the complex model inside the heads of experienced professionals. How successfully a simulation emulates reality depends on the extent of the players' knowledge of the structures, routines, and probable responses of decisionmakers. Great pedagogic, but little policy value results from putting inexperienced individuals in the shoes of decisionmakers.

Even today the national security decisionmaking process is not sufficiently understood as to negate the value of free-form gaming. I would suggest that it never will be. (More on this in a future post.) There has been some debate over issues related to the last part of Bloomfield's remark above, and in the 50's and 60's several researchers (Guetzkow and Hermann are the two I can think of offhand) used inexperienced participants in games to study various aspects of crisis behavior, then tried to generalize conclusions based on a number of runs of the same game. Whether or not those efforts are considered to have produced valid results, exercises designed as experiments in that way bore little resemblance to the high-level games Bloomfield is basing his conclusion on. It is reasonable to agree that given Bloomfield's type of political game, he is correct in asserting that inexperienced participants who do not have carefully calibrated internal models of the world situation would not produce particularly meaningful policy-oriented results.

Among the points Bloomfield makes that have a special relevance to the epistemology of gaming is this:

Games do not predict future events or policy outcomes and can be misleading for specific contingency planning purposes. But they can indicate in detail how a future situation might develop and, even more important, why. A little-noticed fact is that in such games he controlling prediction has already been made: the situation the players face is not of their making, but rather is decided by the game designer. Both MIT and government games relied on scenarios prepared by experts asked to depict a specified future situation in a way that would be accepted by other experts as plausible. A prediction was thereby made that partially determined the game results. Thus, such POLEXes have been biased by game designers, whether for experimental, bureaucratic, or merely frivolous reasons.

These remarks remind me of some of H.A. DeWeerd's concerns about the contextual basis for scenarios, previously discussed here.

Although Bloomfield argues that "In a properly designed POLEX, a knowledgeable non-national can often approximate the policy options likely to be considered by another country's decisionmakers," he also notes that:

Given our national tendency to ascribe Western "rational" mindsets to others, crossing cultural barriers to investigate probable foreign reactions is a valuable and neglected use of gaming. A comparable value derives from putting military officers in the shoes of civilians and vice versa. (A survey of several hundred officials who participated in MIT games reported "role-exchange" as the most useful aspect of the games.)

I suppose that the idea here is that worthwhile results can be generated by non-nationals playing a country's decisionmakers, but only if there is an attempt to understand the cultural context in which those decisionmakers would operate. That may be an element, in Bloomfield's thinking, of proper POLEX design.



Paul Bracken (professor of both political science and management at Yale) contributed an article titled "Deterrence, Gaming, and Game Theory." Among other things, this article includes a short discussion of the distinction between gaming and game theory. This is an issue, because there is a tendency to lump the two together, which is part of the problem with gaming terminology. While Bracken's distinction relies on a rather specific definition of "gaming," it illustrates the difference between the two very well:

Gaming and game theory are sometimes confused with each other, because of their similar names. Gaming refers to an exercise in which opposing teams of human players are confronted with a situation or problem and work out responses to the problem and to the moves of the opposing team. The use of human players to simulate specific political or military decisionmakers is the key to gaming. Such role playing gives the game its richness while distinguishing it from a mathematical model or computer simulation. However, models and simulations often are included in political-military games as supporting tools to facilitate human decisionmaking. When there is extensive use of such models and simulations in a role-playing game, or when computerized communication and control systems are included, the entire exercise is sometimes referred to as a "man-machine simulation."

Game theory is a body of primarily mathematical theory concerning decisionmaking. Although often applied to real problems, game theory needs no justification of practicality. It offers a useful vocabulary for making subtle distinctions and precise definitions about phenomena that arise in gaming and decisionmaking.

Despite this important distinction, Bracken highlights the ways in which gaming and game theory can support each other as analytic tools. Both are tools dealing with the study of strategic interaction, and it is no accident that game theorists like Thomas Schelling and Martin Shubik (and presumably Bracken himself) were drawn to the study of gaming.

When many interacting moves are extended over time, the details of information processing, communication, decentralization, and sequencing are of the highest importance in carrying out a strategy. Gaming, especially with a large number of participants on each side, can make this process transparent. This is less likely when strategic analysis is compacted into verbal stories that rely on a handful of words like "counterforce" and "countervalue."

Specifically, Bracken recommends large-scale games that can encompass the complexity of interactions within the nuclear command and control structure: "The play of a large political-military exercise points to the importance of process, informational patterns, and institutional structure." These are precisely the points he felt needed most examination within the deterrence context. Today's context is greatly different from the nuclear confrontation of 1984, but the utility of large scale games for examining organizational/institutional factors would seem to remain. Bracken was skeptical of pol-mil crisis games that only included a few participants for their lack of emphasis on institutions, at least if they resulted in overly complicated or intricate plans for nuclear warfighting that did not take communication and organizational issues into account.



Garry Brewer (another professor of political science as well as management at Yale) contributed an article titled "Child of Neglect: Crisis Gaming for Politics and War." As might be expected from the title, this is a defense of the crisis game as an analytic tool, and a call for its increased use after a decade of very limited application.

These games never prove anything in a narrow scientific sense. They help to portray the complexities of international conflict; their role-playing aspects provide insights into the special problems of command and control; and they are important educational experiences, providinng participants an opportunity to become aware of facts associated with possible conflicts. Discovery is emphasized and highly valued. Positions, expectations, perceptions, facts, and procedures typically are challenged and improved as the game proceeds. Controllers and referees, who are often experts in particular areas, may question a decision or prevent individuals from making certain moves, but their actions are also open to challenge and debate. Thus, imagination and innovation play central roles in the drama of the manual game. The game also allows players to challenge the initiating scenario, including its explicit and implicit assumptions.

In discussing the original pol-mil crisis games of the 50's and 60's, Bracken notes that:

Political-military games directly addressed a number of important questions:
1) What political options could be imagined in light of the military situations portrayed, and what likely consequences would each have? How, in other words, is force related to political ends?
2) Could political inventiveness be fostered by having those actually responsible assume their roles in a controlled, gamed environment? Would the quality of political ideas stimulated be as good or better than those garnered by more conventional means?
3) Could the game identify particularly important, but poorly understood, topics and questions for further study and resolution? What discoveries flow from this type of analysis that might not from other forms and methods?
4) Could the game sensitize responsible officials to make potential decisions more realistic, especially with respect to likely political and policy consequences?

The flexibility and transparency of the scenario and its relevant assumptions in the crisis gaming technique is particularly important to Bracken:

It is not widely appreciated, but all computer-bassed models and games have embedded scenarios and associated assumptions. The technical problem, however, is that changing scenarios or assumptions means changing many complex instructions contained in the model's embedded codes - something no model builder relishes, especially considering the probable headaches he has already suffered in getting the device to run.

Finally, Bracken makes a necessary point about maintaining a broader view of analysis, incorporating whatever techniques best suit the material, while remaining open to alternative approaches:

A fundamental purpose of manual gaming is to encourage creative, innovative thinking about problems that defy treatment with more conventional analytic approaches and methods. This basic goal has not been achieved to the extent that it could and should be. Furthermore, political-military crisis games are best perceived as key elements in a generalized problem solving process. At present, the analytic community shows an unfortunate tendency to believe that a specific model or analysis will provide answers to a given problem. This is unfortunate for several reasons. The most essential: any given analysis or model can represent only one version and vision of reality. More is needed, and the inherent strength of the manual game in this respect calls it to our attention.

Lloyd Hoffman, Jr. worked at the War Gaming and Simulation Center at NDU, which I assume was replaced by the National Strategic Gaming Center (discussed previously here) at some point, but I haven't looked into the history. Suffice to say, there is no entity with that name at present. His contribution was titled "Defense War Gaming," and it provides a survey of what sort of gaming activity was going on in the Defense Department in 1984. While this is of great historical interest, I won't quote him at length here on that subject. Hoffman covers a wider scope of war gaming than the other authors in the series, delving into field exercises, tactical models, and other such areas.

DeWeerd: Political-Military Scenarios

Another RAND monograph on scenario design by H.A. DeWeerd, but this one is available for free online (through the Defense Technical Information Center (DTIC)'s Scientific and Technical Information Network (STINET)) , unlike the one discussed previously here. Written in 1967, this one is a very basic primer on scenario construction, discussing some of the details a designer needs to consider. The Peter DeLeon piece I described here is still my favorite single scenario design article, but this is a useful look at the process.

Two key concepts in this piece are credibility and relevance, and DeWeerd describes the tension that can arise between them when a game is intended to shed light on a specific research question. At times, it may be necessary to favor relevance to research objectives over scenario credibility, though DeWeerd recommends that in any such circumstances the designers make clear to participants what sacrifices in terms of credibility have been made.

DeWeerd: A Contextual Approach to Scenario Construction - and - Schwartz: The Art of the Long View

A bit more on scenario design.

H.A. DeWeerd wrote A Contextual Approach to Scenario Construction while he was working at RAND, for submission to the journal Simulation and Gaming. It was printed in the December 1974 issue (Vol. 5, No. 4), but it does not appear to be available for free online, either from RAND or the journal itself. It's only 10 pages of text, so it's not worth buying, but DeWeerd is an expert in scenario development (and is cited as such by his student, Peter DeLeon, in the piece I wrote about here).

DeWeerd's main point is that the way scenarios for games, research, or thought experiments are frequently designed back-to-front, with the desired problem or crisis as a starting point, working backwards to fill in contextual details to make the scenario as plausible as possible. DeWeerd contends that a better way to develop scenarios would be to construct the context first, based on current trends, and then look for the most likely scenarios that could take place in the posited context.

This approach is not too far removed from that described by Peter Schwartz in The Art of the Long View. Schwartz developed his techniques for scenario-based planning at Royal Dutch/Shell, and later started Global Business Network. His basic idea is that by developing alternate scenarios of future developments, the possible consequences of present day decisions can be evaluated under a range of conditions. When Schwartz refers to a scenario, he is essentially describing what DeWeerd would call a context.

While Schwartz's method works for considering present day decisions, DeWeerd's proposal for contextual-based scenario design is somewhat more problematic. As he notes, most scenario design is done with a specific problem in mind. As applied to gaming, this is almost always the case. The sponsor of a game has something they want to explore, and they describe it to whatever group organizes the event. Unless the context development takes place on the sponsor's side, before the problem for the scenario is selected, there is very little opportunity for the gaming group to follow DeWeerd's advice. Instead, the retroactive assembly of contextual details to bolster the credibility of the scenario is the only option. Is there another way? Perhaps if the gaming process were integrated well into a broader strategic planning process.

More on the Harvard Symposium on Preemptive Action

I attended the symposium at Harvard Law School I mentioned in the previous post. The panels were great, with high-powered speakers and interesting discussions. However, I was disappointed in the scenarios presented. My guess is that they were written by as legal hypotheticals by law school students, who perhaps then ran them past Richard Clarke to see if they were vaguely realistic. The scenarios were short and handed out to the audience, so I'm going to reprint most of both of them here, to illustrate.

Scenario 1: A high-profile Al Qaeda leader begins operations in Yemen in mid-2006. He is very generous and has been paying local militias to defend and work for him, so the local government is aware that he is there. In general, due largely to bin Laden's fiscal generosity, the local government has been supportive of him. The Yemeni national government, on the other hand, has no idea that he is in the country.

In January, based on reliable human intelligence, the United States uncovers this information. however, it appears as if the Yemeni government does not have a lot of control over the region in which he is located. The United States would like to intervene militarily to capture the Al Qaeda leader, but believes that if it tells the Yemeni government about the intervention, the local government will find out and tip him off so that he can escape. Moreover, the United States is not entirely sure that Yemen would be willing to cooperate and permit military action.

This scenario was followed by a brief comment about the 2002 Predator drone Hellfire missile attack on a car in Yemen carrying six Al Qaeda members (including one of the leaders of the group responsible for the bombing of the USS Cole in October 2000), which was undertaken with the approval of the Yemeni government.

Although this was labeled scenario #1, the panel started its discussion with the scenario listed below instead:

Scenario 2: On February 27, 2007, seven Americans (tourists, businessmen, etc.?) were taken hostage on the border between Colombia and Ecuador by an unknown terrorist/rebel group. Last night, on March 1, the responsible group broadcast online a beheading of one of the hostages and threatened that more beheadings would follow if the United States did not agree to stop funding coca eradication programs in the region. The group provided no other relevant information and its demands are thus far relatively vague.

Although both Ecuador and Colombia have expressed public outrage at the hostage situation and have demanded the release of the hostages, neither government is willing to take action. Ecuador is hesitant to move in because they say the group is actually in Colombia, and Colombia refuses to move in because they think the group is technically in Ecuador. In reality, neither country has total control over the jungle region. Additionally, each is probably incapable of safely rescuing the American hostages. For a variety of reasons, both countries object to allowing American troops to intervene. In each country, the domestic political situation weighs heavily against permitting American intervention. Moreover, both countries claim that allowing American troops to rescue the hostages would constitute a violation of their sovereignty.

There is no evidence that either government is actively supporting the terrorist/rebel group nor is there any evidence that either government has extensive knowledge of the group. The United States has reliable intelligence about the location of the hostages and thinks that a military intervention to rescue them would be successful.

The last paragraph of the scenario talks about the "variety of interesting questions related to preemptive action and international law" that the hypothetical situation raises. In fact, there appeared to be very few interesting questions raised for the panelists by this scenario. They basically agreed that international law did not prohibit the U.S. from using military force without the consent of the two countries (though for different reasons, the discussion of which was interesting), and the precedent of the Entebbe Raid was discussed at length.

Beyond that, the panelists all stressed that this sort of decision would not be made by international legal experts, but by policymakers concerned with much more than just what international law did or did not allow. That was a valuable point, and the panel emphasized it during their discussions of both of the scenarios. That makes sense, since the chair of the panel, Antonia Chayes, teaches a well-regarded class at the Fletcher School on just that subject (one which I wish I had found time to take during my time there).

Several panelists suggested that there would be tremendous public pressure on the administration to "do something" in the Colombia/Ecuador scenario after the kidnapping and the beheading video. What the scenario did not mention is that there are a number of American citizens currently held hostage in Colombia already. Three contractors working under the Department of Defense were taken captive by the FARC in 2003 after their plane crashed. A Colombian colleague and another American citizen were shot, execution style, and their bodies were left at the crash site. While it didn't pack quite the punch of an internet-broadcast beheading of a tourist, in the event there was little public outcry. In the last two years, about 18 American citizens have been kidnapped in Colombia, mostly by terrorist groups. While the beheading would be a new wrinkle, and might generate more attention, the idea that there would be a massive public outcry in the wake of the scenario just doesn't seem likely.

Another issue that the panel rightly addressed was the offhand way the scenario assures that "the United States has reliable intelligence about the location of the hostages and thinks that a military intervention to rescue them would be successful." The panel in general urged skepticism about any claims of this sort, given how difficult it is to make a meaningful and accurate assessment of intelligence data, and given the challenges posed by mounting an operation deep in the Amazon rainforest. The Amazon rainforest that the scenario suggests is the location of the hostages is an incredibly impenetrable area. The canopy foliage in the rainforest can be so thick that GPS will not function through it. Locating anything or anyone from the air is ridiculously difficult, and getting from the air to the ground through the dense foliage presents its own challenges. The scenario tried to wish away such concerns in the interest of focusing on the legal issues involved, but it did so in such a ham-handed way that the panel was obliged to comment on their skepticism of any such confidence. And what no one mentioned but I found very odd about the scenario was that if the U.S. had sufficient intelligence data to know where the hostages were, presumably they could generate map coordinates for the location sufficient to prove to Colombia or Ecuador on which side of the border the terrorists were holed up.

It is true that both in the case of Colombia and the case of Yemen, there are areas within the country that are not under the control of the central government. In Colombia, history and geography are the main reasons why, with a combination of a state that has historically never truly controlled its borders or territory and nearly impassible terrain that makes it extremely difficult for any central authority to assert itself in large portions of the country. I know much less about the situation in Yemen, but my understanding is that there are areas over which, for reasons largely related to tribal dynamics, the central government exerts authority in name only. It was in such an area where the Predator drone attack in 2002 took place.

A term that comes up occasionally in gaming literature is the tendency of participants to "fight the scenario" at times. This is usually held up as something to be avoided, both through careful scenario construction (with an eye towards credibility) and through explicit requests to the participants themselves to refrain from such activity and simply try to play the game as written. This is one of the reasons a debriefing session is so important after a game: participants who had objections to certain aspects of the scenario can raise them at the debrief, where they will still be considered by the group. In a short session like this panel discussion, however, it was precisely the right time to fight this sort of scenario, which the participants did to some extent. It was still a high-quality discussion, thanks to the panelists, but it could have gotten off to a better start with a better set of scenarios.

Harvard Symposium: Legal Perspectives on Preemptive Action

Here's an event that I'm planning to attend tomorrow afternoon at Harvard Law School. The key paragraph from my perspective:

We are working with Richard Clarke to create two unique hypothetical situations for the second panel of the day. Using the same format as the first panel, this panel will address preemptive strikes against non-state actors without the approval of the sovereign nation in which the non-state actors are located. Although the scenarios have not been finalized, they will have to do with a counterterrorism strike against an unwilling sovereign and will be as realistic as possible. An information sheet outlining the scenarios will be circulated before the panel.

It will be interesting to see how they structure the scenarios. Two professors from the Fletcher School will be on the panel, Antonia Chayes and Michael Glennon, both of whom are fascinating speakers. I realize the odds are slim that anyone will read this in time here and decide to attend, but what the heck.

DeLeon: Scenario Designs, an Overview

More from RAND. There has been surprisingly little written about scenario design, and the best piece I have found so far is Peter deLeon's Scenario Designs, an Overview. It's not even 20 pages, and it was written in 1973, but it covers a lot of important ground. And it's available for free online in .pdf format.

Some points I found particularly relevant:

"Although the political/military game can portray a nation's response as the result of the interplay between a number of actors... it cannot hope to capture the full richness of the bureaucratic models that have been increasingly coming to the fore... [he specifically mentions Graham Allison's Models II and III, the Organizational Behavior and Governmental Politics models from Essence of Decision, which I hope to discuss in more detail in a future post].... However, it can help isolate the decision process from extraneous bureaucratic noise."

DeLeon also describes two ways the control team can involve the participants in the action of the scenario. The first, which he calls the "scenario goad," utilizes the scenario to place the teams in a decision setting where they are in effect forced to act. This forecloses options the teams might have had earlier in a crisis, and posits the building crisis as a part of the scenario itself. The second, which he calls the "control goad," turns decision making over to the teams in the earlier stages of a crisis, allowing them more freedom of action. However, if a team proves to be too "reticent or cautious" this requires the control team to "structure and manipulate the game environment in order to provide... incentives" for the team to act in such a way as to advance the game situation. I suppose the literary analogy here is that the "scenario goad" is starting the participants' game experience in media res, while the "control goad" is more of an ab ovo beginning. DeLeon sees the "scenario goad" as a more useful tool for less skilled or less knowledgeable participants (he mentions university students), while more skilled/knowledgeable participants (professionals) might "bring enough prior knowledge to the game that they can promote their own actions and, with the prompting of control, intensify the crisis."

One possibility DeLeon mentions is providing "a menu of options or actions that a team might choose to exercise while playing the game. The explicitness of these 'handles' is largely dependent upon the sophistication of the players. However... the scenarist must be careful not to bias his scenario unconsciously so as to make some options more attractive than the others. Also, such a menu might possibly limit a team's capability and incentive to devise new and novel options. That is, the scenario should present a situation that permits - indeed, encourages - the exploration of a wide variety of levers, old and new."

For a short paper, there's a lot here.