Pre-mortems and the Day After Game

The "pre-mortem" is a technique proposed by Gary Klein and promoted by Daniel Kahneman to legitimize dissent and identify previously unconsidered risks in projects or plans. Kahneman is a psychologist who is essentially the father of behavioral economics, and who won the Nobel Prize in Economics in 2002 for developing prospect theory with Amos Tversky, so when he gets interested in a decision-making tool it's a good sign that there is some real value to it. In an interview, Kahneman described it like this:

'I learnt a great idea from a friend of mine, Gary Klein. He recommends what he calls a 'pre-mortem'. It's a very clever idea. You get a group of people who have made a plan, it's not completely final, but they have a broad plan. And then you bring them together for a special session, 45 minutes is usually enough. You tell them, take a sheet of paper and now imagine the following: a year has passed, we have implemented the plan and it is a disaster.

'Now, write down on that sheet of paper what happened. Why did the plan turn out a disaster?

'It's a brilliant idea. Because you have a group of people who are now encouraged to think of difficulties and problems with the plan. And that solves the problem of legitimising dissent very elegantly, and it's easy to implement.

Simple and elegant. So much so that there isn't much written about it online. How much more is there to say? Kahneman himself discusses the pre-mortem in this video interview (the whole discussion is worthwhile, but the pre-mortem section starts with about 12:38 remaining). As one commenter paraphrased part of Kahneman's argument:

In organizations where the members are competitive, you expect people to think quite hard about the flaws in the idea and what could go wrong. In a room of twenty people you might expect three or four new ideas that can be used to readjust and improve the proposed plan of action.

The whole concept of the pre-mortem reminds me of a type of game that RAND developed called the Day After Game. This is a much more structured exercise than a pre-mortem, but a pre-mortem seems limited to considering specific plans that are already well-developed, while the Day After Game can be applied to more general problems or issues. The Day After Game is divided into three periods. In the first period, participants are confronted with an unfolding crisis and asked for their response. In the second, the "day after" (hence the name of the technique) the crisis hits is explored, with the consequences mapped out and the effectiveness (or more likely ineffectiveness) of the measures proposed in the first period considered. But it's the third phase that really separates the Day After Game from other crisis exercises, because then participants are asked to step back to before the crisis event and consider what could have been done between the present day and the hypothetical future date of the crisis in the first period to improve the likelihood of mounting an effective response. By considering the severity of the day after, it becomes a mechanism for focusing attention on the "day before." In this respect, the Day After Game is a means of generating "prospective hindsight" in a similar fashion as the pre-mortem exercise.

This methodology was originally developed as a means of surveying the Washington policy community by running a series of games with many repetitions, in an attempt to identify the major schools of thought about policy alternatives that would likely dominate discussion in Washington in the near term. Over time it seems to have evolved into something more focused on generating "prospective hindsight." The RAND report on the first use of this technique (examining nuclear proliferation) is available for free in .pdf format in three volumes. One other Day After Game report (on cybersecurity) is publicly available as well. There is a good description of the methodology followed in each report. RAND has apparently used the technique in a number of other contexts, as well.

I'm a little surprised that this type of game hasn't made many inroads (to my knowledge) into academic or classroom settings. It's a simple enough concept. There isn't any strategic interaction, since Day After Games are run with all the participants responding as a group to the scenario, but that's part of what makes it easier to execute.

Eliot Cohen on punditry (and the implications for gaming)

I missed this when it was published, but Eliot Cohen had a great op ed in the Wall Street Journal in January, shortly after stepping down as senior adviser to SecState Rice. It doesn't directly touch on gaming, but it provides some very useful suggestions for pundits and policy commentators on the outside of government looking in:

Most commentators have a radically imperfect view of what's going on. Those on the inside, including at the very top, know more, though less than one might think. Government resembles nothing so much as the party game of telephone, in which stories relayed at second, third or fourth hand become increasingly garbled as they crisscross other stories of a similar kind ("That may be what the Russian national security adviser said to the undersecretary for political affairs on Wednesday, but it's not how the Turkish foreign minister described the Syrian view to our ambassador to NATO on Thursday.") Add to this the effects of secrecy induced by security concerns, as well as by the natural desire to play one's cards close to one's vest, and the result is a well-nigh impenetrable murk of policy making.

But it's even murkier on the outside. "Occasionally an outsider may provide perspective; almost never does he have enough knowledge to advise soundly on tactical moves," Henry Kissinger once remarked. Or as the White House correspondent of one major national newspaper once confided to me, "We really don't have a clue what's going on in there."

What, then, is a pundit to do? The best commentary has an impact, less because it offers new ideas (most ideas have been considered, however incompletely, on the inside) than because it clarifies problems or solutions that the insiders have only vaguely or incompletely considered. A tight, well-written, and carefully reasoned examination of a policy problem will bring into focus an issue that the officials have not had the time, or often the literary skill, to capture precisely. That kind of analysis is very much worth reading.

Invariably, a pundit will prescribe solutions. In doing so, he should follow the advice of the late Raymond Aron, the wisest French policy intellectual of modern times: Never criticize a policy unless you can convincingly depict a better course of action. Aron, like many of the greatest commentators on policy, had virtually no experience in government, but great empathy for those in a position to decide. Empathy -- the capacity for imagining what it is like to be the other -- is an essential quality for the thoughtful pundit. Policy makers, of course, prefer sympathy, which is soothing, unnecessary and often harmful.

There's more in the full op ed. It's not hard to see how this could relate to gaming. In a sense, the question of how games can have a policy impact is implicated in this line of thinking. If a game "clarifies problems or solutions that the insiders have only vaguely or incompletely considered" or provides the opportunity for outside commentators to develop the empathy with decision-makers that Cohen describes, that could be a valuable contribution. It is extremely rare for high-level decision-makers to participate in games; the time just isn't available. Cases like the Sigma II game in 1964 are few and far between. Therefore, the utility of games from a policy perspective is at best indirect. Cohen's piece provides some food for thought on maximizing the usefulness of games to current policymakers.

(Found this piece thanks to Wiggins at Opposed Systems Design)

Update: Rex Brynen has more thoughts on this, including an excellent example from his own experience (and some very kind words) at PaxSims. See also the comment by Wiggins.

Game theory courses (and some learning by doing... in Starcraft)

Game theory doesn't have anything directly to do with the political/military gaming this blog is about, but it's a useful tool in conceptualizing strategic interaction, which can come in very handy. Academic Earth has posted video of the lectures from two different elementary game theory courses, one from Yale, the other from Berkeley. The Berkeley course only has one lecture posted so far, but the Yale course appears to be complete. However, the Berkeley course offers something unique: it is the first class to be taught from the perspective of the video game Starcraft. The strategic interaction within the game forms the basis for the course's explanations of the basics of game theory, and familiarity with the game is highly recommended for students taking the class. I'm not sufficiently familiar with Starcraft to fully appreciate this, but it's a fascinating concept. The class website is here, with more information. It's enough to make me consider trying to hunt down a copy of the game.

For a more complete explanation of the difference between gaming and game theory and an example of how game theory and gaming together can complement each other as analytic tools, see the section of this post describing an article by Paul Bracken that utilizes both.

Sigma I-64 and Sigma II-64: gaming Vietnam

In 1964 the JCS hosted two particularly interesting wargames, Sigma I-64 and Sigma II-64. The Sigma wargames were designed to consider the issues surrounding the escalation of U.S. commitment in Vietnam. H.R. McMaster's article in the current issue of World Affairs mentions these games in the context of what they revealed about the weakness of "Graduated Pressure," a strategy of carefully calibrated use of force meant to efficiently change the behavior of the adversary:

The failure of Graduated Pressure was foretold even before it was implemented. In 1964, two eerily prophetic Pentagon war games exposed fatal flaws in the strategy. In those war games, Southeast Asia experts played the role of the North Vietnamese government. In response to limited bombing designed to signal American resolve, those experts decided to infiltrate large numbers of North Vietnamese Army soldiers into the Central Highlands of South Vietnam. This, in turn, impelled the commitment of American troops to the South. The war games concluded that the combination of enemy sanctuaries in North Vietnam, Cambodia and Laos, the enemy’s ability to sustain itself on meager provisions, its strategy of emphasizing political and military actions to avoid strength and attack weakness, and limitations on the application of American military power, would mire the United States in a protracted conflict with little hope for success. The game ended after five years of fighting with five hundred thousand troops committed in South Vietnam. Bundy, however, found the conclusion to be “too harsh.” Rather than force a reexamination of strategy, the results of the SIGMA I and SIGMA II war games appear, in retrospect, as a roadmap that civilian and military leaders followed along the path to failure in Vietnam.

McMaster goes on to note that "The SIGMA war games had no effect on American policy or strategy in Vietnam." Why is that? On the surface, the games were extremely well positioned to have a major impact. Certainly, there was a need for sober reexamination of the underlying assumptions of U.S. strategy. Harold Ford cited a CIA officer’s comment on Sigma I-64:

“Widespread at the war games were facile assumptions that attacks against the North would weaken DRV capability to support the war in South Vietnam, and that such attacks would cause the DRV leadership to call off the VC. Both assumptions are highly dubious, given the nature of the VC war.”

Gaming with a broad cross-section of participants can serve to identify and challenge the assumptions they bring with them, and in this case there were participants in the Sigma games that did not share the predominant view of the conflict. Sigma I-64 was played by “working level CIA, State, and military officers.” Sigma II-64 also involved high-level policy officials, including National Security Adviser McGeorge Bundy, DCI John McCone, General Curtis LeMay, General Earle Wheeler, and A/S Defense John McNaughton, Deputy Secretary of Defense Cyrus Vance, Deputy CNO Admiral Horacio Rivero, Jr., A/S State William Bundy, and CIA deputy director for intelligence Ray Cline. This is an exceptionally high-level group to find participating in a wargame, but it does not appear that the Sigma wargames of 1964 had any direct impact on policy, despite their dire warnings about the potential consequences of U.S. actions and despite the participation of numerous military and civilian national security leaders. Without a willingness to be persuaded on the part of decision-makers, the games could not make a policy impact. While the results of Sigma I and II were uncannily prescient, the failure of participants to take the results seriously is not the most significant issue; policy-makers are right to be skeptical of a game's predictive value. But by failing to move policy-makers to reexamine their assumptions from a fresh perspective, the Sigma games represent a colossal missed opportunity.

(Thomas Allen's 1987 book, War Games, has a chapter on the Sigma games (pp. 193-208). See also Harold Ford, CIA and the Vietnam Policymakers: Three Episodes 1962-1968, pages 57, 58, and 67.)

Moral dilemmas , gaming, and the Kobayashi Maru

Rex Brynen at PaxSims has an interesting post up wondering whether it would be helpful to provide training to peacekeepers in the form of Kobayashi Maru-like scenarios (scenarios for which there is no "good" answer, no way to win, and no way to avoid a severe negative outcome), as a way of preparing them for the wrenching moral choices they might encounter in the course of their work:

Science fiction fans will immediately recognize this as a reference to a Starfleet training simulation featured in the movie Star Trek II: The Wrath of Khan. In the Kobyashi Maru scenario, trainees were faced with impossible choices in an unwinnable situation: did they answer a distress call from a damaged frieghter, only to find their ship destroyed? Or did they ignore the call, only to see the freighter destroyed? It was meant to be a test of character, and an evaluation of how would-be officers confronted such dilemmas.

Now, readers who haven’t the slightest interest in Star Trek needn’t worry that this blog post will slip into excessive Trekkism. Rather, it occurs to me that there may be some value in designing peacebuilding/humanitarian assistance operations in which participants are confronted with situations that truly have no good answers. I mean this, moreover, not simply in that they face resource shortages and hence opportunity costs associated with actions (something that the Carena simulation does very well), but rather that no matter what they do, they are forced to confront gut-wrenching moral choices.

• Does one—for example—pull humanitarian workers out of a dangerous area, knowing locals will die? Or does one keep them there, knowing that no matter what security precautions are taken there is a significant risk of staff being killed?
• Do you authorize an airstrike against a high-value insurgent leader, knowing that there is a near-certainty of significant civilian casualties?
• Do you pay “taxes” to a local militia to enable access to a needy population—knowing that the doing so strengthens their capacity to engage in such predatory activities?
• Do peacekeepers fight in to protect civilians from massacre, even if they believe they lack the capability to win and might thereby be slaughtered as well? (Yes, I’m thinking here of Srebrenica, although it could equally be applied to some of the choices that MONUC has made in DR Congo.)

…and so forth. The point would be not so much which particular choice was made, but how it was made—providing an opportunity for participants to reflect on the the moral and practical calculus involved.

This got me back to thinking about the potential role of gaming in preparing participants for difficult moral challenges by means of acclimating them to decision-making in that sort of environment (something I looked at a bit in my thesis). In their 1977 book, Decision Making: A Psychological Analysis of Conflict, Choice, and Commitment, Irving Janis and Leon Mann described a similar phenomenon they called "outcome psychodrama," which would involve a sort of "emotional inoculation" for decision-makers ahead of time to better prepare them for the emotional states they might experience under a decision-making situation. They believed that gaming might be a particularly appropriate tool for outcome psychodrama "[f]or group discussions whose consequences are highly dependent upon the reactions and countermoves of other people." In contrast to the way most games are run, Janis and Mann suggested running the similar scenarios multiple times with the same participants, examining likely, favorable, and worst case results for each of the available alternatives.

As an example of the danger in ignoring the emotional factors in decision-making, consider Bernard Brodie’s description of the U.S. decision to defend South Korea in 1950 after having announced that it outside the U.S. defensive perimeter in the region just months earlier:

“President Truman and his Secretary of State had endorsed the well-known and obviously rational opposition of the military to becoming involved on mainland Asia. When the crisis came, all, including the military, immediately reversed their position. Apparently few had asked themselves: How will we feel, and how will we in consequence respond, if there is a flagrant attack upon this state and this government that we have ourselves set up and all but recently withdrawn from on the now disproved assumption that they were no longer in danger? Will we really stand by and let them go down?” (Bernard Brodie, War and Politics, 60-61)

Brodie suggested that this was the result of "the tendency of people in office to use formulae rather than imaginative rumination to in projecting their own behavior into the future." "Imaginative rumination" is something that free-form gaming can offer the opportunity to pursue. This goes further than the Kobayashi Maru no-win scenario, but seems like it could have significant value. I am not aware of games having been built for this purpose, but John Hanley discussed a number of participants from the Global War Game who reported having experienced powerful emotions surounding the sorts of "gut-wrenching moral choices" Rex proposes. In the Global War Game, these often revolved around the use of nuclear weapons. Hanley mentioned some participants reporting that "[e]ven simulated momentous decisions cost them some sleep."

New gaming blog, new article on NDU's gaming center

Rex Brynen and Gary Milante recently started PaxSims, a great new blog, "devoted to the development and implementation of peace and conflict simulations for education and training purposes." It's well worth checking out, and they are posting frequently.

Their most recent post is on a new article in Joint Forces Quarterly on the National Strategic Gaming Center at NDU. There's some great stuff about a new initiative by the NSGC to get the various gaming shops to engage more with each other and with the broader policy community with regards to gaming methodology. I wrote about a previous JFQ article on the NSGC here.

Nuclear War: Iran vs. Israel

This report from the Center for Strategic and International Studies is not strictly speaking a game report, but rather "the material used in a scenario analysis and interactive game that looks at some of the consequences of a future nuclear exchange between Israel and Iran, and the possible impact of its expansion to cover targets in Syria, Egypt, and the Gulf." Unfortunately, I can't find any details on the "interactive game," but the background materials are comprehensive and fascinating (if you're into nuclear weapons). The results from their analysis are presented as a quote from the movie Wargames: the only way to win is not to play. If anyone knows where I could find details about the "interactive game" they conducted, I would love to learn more about it.