Pre-mortems and the Day After Game

The "pre-mortem" is a technique proposed by Gary Klein and promoted by Daniel Kahneman to legitimize dissent and identify previously unconsidered risks in projects or plans. Kahneman is a psychologist who is essentially the father of behavioral economics, and who won the Nobel Prize in Economics in 2002 for developing prospect theory with Amos Tversky, so when he gets interested in a decision-making tool it's a good sign that there is some real value to it. In an interview, Kahneman described it like this:

'I learnt a great idea from a friend of mine, Gary Klein. He recommends what he calls a 'pre-mortem'. It's a very clever idea. You get a group of people who have made a plan, it's not completely final, but they have a broad plan. And then you bring them together for a special session, 45 minutes is usually enough. You tell them, take a sheet of paper and now imagine the following: a year has passed, we have implemented the plan and it is a disaster.

'Now, write down on that sheet of paper what happened. Why did the plan turn out a disaster?

'It's a brilliant idea. Because you have a group of people who are now encouraged to think of difficulties and problems with the plan. And that solves the problem of legitimising dissent very elegantly, and it's easy to implement.

Simple and elegant. So much so that there isn't much written about it online. How much more is there to say? Kahneman himself discusses the pre-mortem in this video interview (the whole discussion is worthwhile, but the pre-mortem section starts with about 12:38 remaining). As one commenter paraphrased part of Kahneman's argument:

In organizations where the members are competitive, you expect people to think quite hard about the flaws in the idea and what could go wrong. In a room of twenty people you might expect three or four new ideas that can be used to readjust and improve the proposed plan of action.

The whole concept of the pre-mortem reminds me of a type of game that RAND developed called the Day After Game. This is a much more structured exercise than a pre-mortem, but a pre-mortem seems limited to considering specific plans that are already well-developed, while the Day After Game can be applied to more general problems or issues. The Day After Game is divided into three periods. In the first period, participants are confronted with an unfolding crisis and asked for their response. In the second, the "day after" (hence the name of the technique) the crisis hits is explored, with the consequences mapped out and the effectiveness (or more likely ineffectiveness) of the measures proposed in the first period considered. But it's the third phase that really separates the Day After Game from other crisis exercises, because then participants are asked to step back to before the crisis event and consider what could have been done between the present day and the hypothetical future date of the crisis in the first period to improve the likelihood of mounting an effective response. By considering the severity of the day after, it becomes a mechanism for focusing attention on the "day before." In this respect, the Day After Game is a means of generating "prospective hindsight" in a similar fashion as the pre-mortem exercise.

This methodology was originally developed as a means of surveying the Washington policy community by running a series of games with many repetitions, in an attempt to identify the major schools of thought about policy alternatives that would likely dominate discussion in Washington in the near term. Over time it seems to have evolved into something more focused on generating "prospective hindsight." The RAND report on the first use of this technique (examining nuclear proliferation) is available for free in .pdf format in three volumes. One other Day After Game report (on cybersecurity) is publicly available as well. There is a good description of the methodology followed in each report. RAND has apparently used the technique in a number of other contexts, as well.

I'm a little surprised that this type of game hasn't made many inroads (to my knowledge) into academic or classroom settings. It's a simple enough concept. There isn't any strategic interaction, since Day After Games are run with all the participants responding as a group to the scenario, but that's part of what makes it easier to execute.

Lempert and Schwabe - Transition to Sustainable Waste Management

Another RAND monograph, but this one deals with an application of gaming techniques to a domestic public policy question (available in pdf form here). The goal was to investigate what factors might turn out to be important in the shift from a product packaging industry that primarily used "virgin" materials and generated large amounts of solid waste to a system based on recycling of packaging material. Participants from government and industry were assigned roles within the context of a model, and were asked to make decisions on investment or regulation related to packaging. Based on these decisions, the model then worked out the market clearing prices and quantities for each type of product or service, established profits and losses for the simulated activities, and presented the results to the participants for another round of regulation and investment.

This paper provides a detailed look at the way a model can provide the structure for a game, as well as showing the value added that gaming out a model can produce.

DeWeerd: Political-Military Scenarios

Another RAND monograph on scenario design by H.A. DeWeerd, but this one is available for free online (through the Defense Technical Information Center (DTIC)'s Scientific and Technical Information Network (STINET)) , unlike the one discussed previously here. Written in 1967, this one is a very basic primer on scenario construction, discussing some of the details a designer needs to consider. The Peter DeLeon piece I described here is still my favorite single scenario design article, but this is a useful look at the process.

Two key concepts in this piece are credibility and relevance, and DeWeerd describes the tension that can arise between them when a game is intended to shed light on a specific research question. At times, it may be necessary to favor relevance to research objectives over scenario credibility, though DeWeerd recommends that in any such circumstances the designers make clear to participants what sacrifices in terms of credibility have been made.

DeWeerd: A Contextual Approach to Scenario Construction - and - Schwartz: The Art of the Long View

A bit more on scenario design.

H.A. DeWeerd wrote A Contextual Approach to Scenario Construction while he was working at RAND, for submission to the journal Simulation and Gaming. It was printed in the December 1974 issue (Vol. 5, No. 4), but it does not appear to be available for free online, either from RAND or the journal itself. It's only 10 pages of text, so it's not worth buying, but DeWeerd is an expert in scenario development (and is cited as such by his student, Peter DeLeon, in the piece I wrote about here).

DeWeerd's main point is that the way scenarios for games, research, or thought experiments are frequently designed back-to-front, with the desired problem or crisis as a starting point, working backwards to fill in contextual details to make the scenario as plausible as possible. DeWeerd contends that a better way to develop scenarios would be to construct the context first, based on current trends, and then look for the most likely scenarios that could take place in the posited context.

This approach is not too far removed from that described by Peter Schwartz in The Art of the Long View. Schwartz developed his techniques for scenario-based planning at Royal Dutch/Shell, and later started Global Business Network. His basic idea is that by developing alternate scenarios of future developments, the possible consequences of present day decisions can be evaluated under a range of conditions. When Schwartz refers to a scenario, he is essentially describing what DeWeerd would call a context.

While Schwartz's method works for considering present day decisions, DeWeerd's proposal for contextual-based scenario design is somewhat more problematic. As he notes, most scenario design is done with a specific problem in mind. As applied to gaming, this is almost always the case. The sponsor of a game has something they want to explore, and they describe it to whatever group organizes the event. Unless the context development takes place on the sponsor's side, before the problem for the scenario is selected, there is very little opportunity for the gaming group to follow DeWeerd's advice. Instead, the retroactive assembly of contextual details to bolster the credibility of the scenario is the only option. Is there another way? Perhaps if the gaming process were integrated well into a broader strategic planning process.

DeLeon: Scenario Designs, an Overview

More from RAND. There has been surprisingly little written about scenario design, and the best piece I have found so far is Peter deLeon's Scenario Designs, an Overview. It's not even 20 pages, and it was written in 1973, but it covers a lot of important ground. And it's available for free online in .pdf format.

Some points I found particularly relevant:

"Although the political/military game can portray a nation's response as the result of the interplay between a number of actors... it cannot hope to capture the full richness of the bureaucratic models that have been increasingly coming to the fore... [he specifically mentions Graham Allison's Models II and III, the Organizational Behavior and Governmental Politics models from Essence of Decision, which I hope to discuss in more detail in a future post].... However, it can help isolate the decision process from extraneous bureaucratic noise."

DeLeon also describes two ways the control team can involve the participants in the action of the scenario. The first, which he calls the "scenario goad," utilizes the scenario to place the teams in a decision setting where they are in effect forced to act. This forecloses options the teams might have had earlier in a crisis, and posits the building crisis as a part of the scenario itself. The second, which he calls the "control goad," turns decision making over to the teams in the earlier stages of a crisis, allowing them more freedom of action. However, if a team proves to be too "reticent or cautious" this requires the control team to "structure and manipulate the game environment in order to provide... incentives" for the team to act in such a way as to advance the game situation. I suppose the literary analogy here is that the "scenario goad" is starting the participants' game experience in media res, while the "control goad" is more of an ab ovo beginning. DeLeon sees the "scenario goad" as a more useful tool for less skilled or less knowledgeable participants (he mentions university students), while more skilled/knowledgeable participants (professionals) might "bring enough prior knowledge to the game that they can promote their own actions and, with the prompting of control, intensify the crisis."

One possibility DeLeon mentions is providing "a menu of options or actions that a team might choose to exercise while playing the game. The explicitness of these 'handles' is largely dependent upon the sophistication of the players. However... the scenarist must be careful not to bias his scenario unconsciously so as to make some options more attractive than the others. Also, such a menu might possibly limit a team's capability and incentive to devise new and novel options. That is, the scenario should present a situation that permits - indeed, encourages - the exploration of a wide variety of levers, old and new."

For a short paper, there's a lot here.

Jones: On Free-Form Gaming

Another RAND note, this time from 1985. William Jones wrote On Free-Form Gaming based on his decades of experience working on gaming at RAND.

This is a 55 page document, but it is so dense with practical insights into gaming procedures that it is almost impossible to summarize adequately. Anyone looking for a better understanding of gaming techniques is well advised to find a copy somehow. It is not available for free online at this time, but it is available through interlibrary loan for those with access. The price at RAND is steep for 55 pages ($20.70), but quite possibly worth it.

Jones focused mainly on two-team games, like the classic crisis games (mentioned previously here and here, but his suggestions and insights pertain to any team structure, and he discussed several alternative structures in the course of the note. One interesting detail is that his vision of free-form gaming involved set move periods, requiring each team to produce a comprehensive set of instructions by the end of the move period. There was no opportunity to take action on the part of the teams (besides making specific requests for information to the control team) during the move period, meaning that "game time" effectively did not advance during the teams' deliberations. After the move period, the control team would assess what the timeframe and situation would be at the start of the next move period. That required them to assemble all of the teams' move papers, decide what the results of the requested actions would be, decide on when to advance the game time until (based on their view of the next interesting point for study), and produce new scenario papers for the next move period.

In contrast, using computer systems (like the ICONS project's web-based software, but more on this another time), exercises today can more easily allow teams the option for action during moves, implicitly or explicitly advancing "game time" during the move period. That probably creates a very different atmosphere within the teams. It also opens the door to diplomatic interaction between teams, something very difficult to manage if move orders are only processed at the end of each move period.

This is an excellent paper. Rather than try to go into further detail now, I'll refer to it when discussing things in the future, since there is something related to almost every aspect of pol/mil gaming.

UPDATE: Alert reader persis notes in the comments that this paper is now available for free download at RAND. Go get it!

The Simulation Heuristic

I finished another RAND monograph recently (Implications of Modern Decision Science for Military Decision-Support Systems), which is too long and too dense and too useful to sum up in a single post. It's available for free online, but I wound up buying a copy from Amazon after realizing how relevant it was for me (Amazon was cheaper than buying it through RAND this time).

The first section deals with the history and development of decision science. It's a great, concise summary of the psychological aspects of decision-making theory, including a description of the split between what the authors refer to as the Heuristics and Biases Paradigm and the Naturalistic School of decision science. More on this in the future, but something that struck me was the description of what some call the simulation heuristic, and which the monograph refers to as "imaginability" (as a subset of memory biases) or the "availability heuristic". The basic idea is that people see things as more likely if they can visualize them more easily. That echoes Robert Levine's criticism (referred to here) that a game could lead participants to believe that the events depicted were probable, as opposed to merely plausible. In the simulation heuristic formulation, it is the very act of making it easier to imagine an event that increases the perception of its probability.

In a sense, that might not be a bad thing, in that one of the benefits to simulation and gaming can be to make a given circumstance seem more real, for the purposes of spurring people to action, or taking a threat seriously, or provoking more thoughtful analysis. But Levine's point stands to a certain extent; what about the other, unsimulated, ungamed possibilities that are accorded less priority because the simulated case is assigned a higher probability than is warranted? The simplest answer I can see right now is that rather than doing less gaming, perhaps the simulation heuristic should in fact lead to doing more gaming, as a tool of exploratory analysis, with an eye towards keeping in mind the vast multitudes of possible futures (more on this sometime soon, I hope). Another answer might just be that gaming is not necessarily by itself a good indicator of the likelihood of a given set of events. That seems like a no-brainer, but perhaps it's difficult to keep that in mind when presented with a realistic game. How do the pros deal with this issue?

Schwabe: An Introduction to Analytic Gaming

William Schwabe wrote an entry on gaming for the Encyclopedia of Operations Research and Management Science in 1994 that was also released as a RAND paper. Unfortunately, the entry/paper is not available online at this time. It's only 2 or 3 pages in the encyclopedia, and the RAND paper is only 7 pages long (double spaced), but it does a good job of laying out the basics of gaming as a policy research tool. It's a bit short to quote at length, but here are several points:

Few methods have been so inadequately named, prompting ridicule from skeptics and attempts by adherents to call it something more serious sounding or descriptive, such as "operational gaming," "simulation gaming," "free-form gaming," and, in defense analysis, "war gaming" and "political-military gaming."
....

Unlike many other techniques of analysis, gaming is not a solution method. The output of a game is not a forecast, solution, or rigorous validation. The output of a good game is increased understanding.
....

Gaming has often not been as well integrated into studies using other methodologies as might be warranted. Gaming is but one form of analysis to inform policy, managerial, or operational decisions.

In addition, Schwabe quotes Clark Abt's 1970 book, Serious Games, in his definition of a game in his first paragraph. It's a book I'll have to pick up at some point (and apparently it was just reprinted in paperback in 2002), but for now:

Clark Abt (1970) defines a game as "an activity among two or more independent decision-makers seeking to achieve their objectives in some limiting context."

Note the importance of multiple decision centers, just as Schelling noted (as mentioned previously here). Schwabe quoted some of the same portions of Schelling's section in Crisis Games 27 Years Later that I did in his description of what games can produce that other research methods cannot.

For a very basic look at gaming as a research tool, this is a remarkably successful 7 pages. I have the RAND report, not the encyclopedia entry, but I believe they are about the same. It wouldn't be worth the $18 RAND is charging for the paper, though, and it shows the importance of checking the page count before purchasing thinktank reports online. I got mine through interlibrary loan, and I expect that the encyclopedia is widely available at university libraries.

More on Crisis Games

To continue the thoughts in the previous post, Crisis Games 27 Years Later contains a number of criticisms of the gaming form by Robert Levine, to which Thomas Schelling and William Jones respond.

Some of Levine's concerns:

Gaming could lend a sense that a plausible scenario is in fact probable. This could take place by virtue of the chosen, simulated scenario becoming more "real" in the minds of the participants than all of the other, unexplored possibilities. (More on cognitive biases like this one in another post soon, once I finish the paper I'm reading now)

A combination of "ersatz history" (in the form of scenarios that depart greatly from present day events) and "ersatz people" (since game players do not share the same characteristics as the individuals they are simulating) introduce biases to the gaming process that compromise its utility in generating worthwhile hypotheses.

The results of games may be used as evidence, even when they have no evidentiary value. Specifically: "If a game result appears in language which can be read as a confirmed policy conclusion, there is a substantial chance that it will be so read."


For a more recent view of some similar issues, Robert Rubel's "The Epistemology of War Gaming" (mentioned in this previous post) talks about a broader category of games, and also has steps in mind to try and address some of these methodological challenges, rather than simply advocating that gaming be abandoned as a research tool.


Rather than enumerate Schelling and Jones' replies to these and other arguments point by point, here are a few examples of their ideas (the paper is well worth reading in its entirety):

Schelling defines games as necessarily involving "at least two separate decision centers... neither of which is privy to the other's planning and arguing, neither of which has complete access to the other's intelligence or background information, neither of which has any direct way of knowing everything that the other is deciding on." This definition excludes many of the exercises we would categorize as games, and includes many that do not appear to be games at all, but this is because Schelling is focused on the strategic interaction between the two (or more) decision centers:

"What this mode of organization can do that can not otherwise be done is to generate the phenomena of understanding and misunderstanding, perception and misperception, bargaining, demonstrations, dares and challenges, accommodation, coercion and intimidation, conveyance of intent, and uncertainty about what each other has already done or decided on. There are some things that just cannot be done by a single person or by a team that works together."

"One is to judge how 'obvious' something is. An analogy is the 'hidden face' in the picture. If I draw a face with a hidden picture, there is no way for me to tell how hard it is to see the face except to show the picture to somebody. I can't not see the picture because I put it there, and the hidden face has the quality that once you've seen it it is awfully hard to recapture your innocence and not see it."

Jones responds to the charge of "ersatz history" thus (keep in mind that he was writing in 1964):

"[P]erfectly plausible, useful, and not improbable game scenarios of a situation three or four years in the future can be had by simply initiating a game using today's newspapers -- interjecting only tomorrow's equipment. Arab-Israeli controversy has existed for some time and it seems probable that it will be in existence in 1970. Turkey and Greece have not seen eye-to-eye on the Cyprus situation for a considerable period and, unfortunately, this controversy is likely to persist.... Notice here that I am not saying that games, played from scenarios which are extrapolations of today's problems, are perfect for determining tomorrow's force posture and weapons characteristics. I am saying that I can think of worse procedures (re-fighting the last war, for example) and I know of no other way in which I have even such a limited amount of confidence."

There is a good deal more in this paper than I can tackle in a blog post, but many of the ideas and concerns expressed resonate today, which is why RAND reprinted it in 1991, 27 years after it was written.

Crisis Games and RAND

In 1964, Robert Levine, Thomas Schelling, and William Jones circulated a series of drafts through the RAND Corporation internal mail system, discussing the strengths and weaknesses of crisis gaming. In 1991, feeling that the issues laid out remained relevant, Levine republished the entire exchange as a RAND paper: Crisis Games 27 Years Later: Plus C'est Deja Vu. RAND has released this paper (along with some others from their 50 years of research and policy memoranda) for free in pdf form. It's well worth reading in its entirety. Thomas Schelling won the Nobel Prize in economics in 2005, and his defense of crisis gaming as a technique is fascinating.

RAND did a lot of pioneering work in crisis gaming, which seems to have come on the scene during the Cold War as a way of examining the issues surrounding crises between the US and the USSR (I will confirm this at some point in the future, but the earliest references I have seen so far to crisis gaming date to the 1950's). The study of crisis was a priority, given the risks of nuclear war that attended a misstep. The various Berlin crises, the Cuban Missile Crisis, and the surprise attack that began the Korean War were all influential in shaping the view that crisis management was an important skill to develop. Cold War crises had some very appealing features when it comes to gaming: there were two main adversaries whose interactions (due to their superpower status) were much more important than the roles of any other states (whose actions could be devised by the control team), the stakes in any given confrontation were high and the time frames often short (due to the presence of large nuclear arsenals), which ensured that the highest level decision makers would be involved, superseding normal policymaking structures, and the novelty of the nuclear standoff, with unprecedented destructive power involved, which gave a sense to policy theorists that all bets were off and that this type of crisis event needed to be studied outside the context of previous conventional military confrontations. This last point was also a major argument made by the newly ascendant civilian policy analysts in favor of rethinking conflict in ways opposed by the military establishment.