I met Alex Soojung-Kim Pang, a self-described "historian and futurist of science," at a recent conference. He has some interesting ideas about the potential use of gaming in communicating scenarios (of the type generated by scenario planning processes) to decision-makers. And I owe him an email.
His blog, The End of Cyberspace, would likely be of interest to readers of this blog. Most posts are collections of links, but what links they are! Of particular note, these two posts deal mostly with tabletop exercises for disaster preparedness and emergency management, but they also extend to decision-making research, including some intriguing National Science Foundation grant awards. One of the links I'm looking forward to exploring in more depth is the FEMA online independent study course on emergency preparedness exercise design, which includes a section on their vision of tabletops.
Of course, another blog anyone who reads this should also be reading is PaxSims, and not just because Rex Brynen compared my return to blogging with the camp exploits of my childhood hero, though that has helped make him one of my favorite people.
Showing posts with label decision science. Show all posts
Showing posts with label decision science. Show all posts
Saturday, July 17, 2010
Friday, July 9, 2010
Perception and misperception in gaming
Jon Alterman has written a short and thought-provoking piece for a CSIS newsletter about his experience on the U.S. team in a recent multi-cell policy game. Worth reading in full, but here's an excerpt:
The U.S. team swiftly leapt into a series of actions intended to direct the actions of its allies and blunt the efforts of its foes. In the second move, things got worse, and the U.S. side tried even harder to marshal its forces, artfully deploying its military and diplomatic assets. By the third move, the situation continued to worsen in many respects, but the U.S. team saw light at the end of the tunnel. We had a plan, and our allies were looking to us for leadership. Equally importantly, they were all acting precisely as we had hoped, abandoning the troublesome sorts of freelancing that had marked their earlier moves. We thought we had played the game well.When we all gathered after the final move, however, it was clear how much we had misjudged the situation. Opponents talked about how easily U.S. moves were blunted or ignored. Allies were beside themselves that the United States had missed numerous opportunities to consult with them and raised tensions needlessly. But most importantly, they charged that the U.S. team had fundamentally misjudged the motivations of their actions. The U.S. team had congratulated itself on its ability to integrate all of the instruments of national power—in contrast to allies that could either convene summits or issue statements or host American military forces, but rarely more than one of those and almost certainly not in a sustained, purposive and coordinated way. Yet allies explained that perceptions of their own national interests drove their decisions, and that U.S. actions rarely shaped those decisions. It is true, the U.S. team had moved military aircraft and issued statements to and fro, but the other players did not find it very impressive. They had their own calculus. In their telling, it was as if the U.S. team was trying to take credit for the sun rising in the East. If anything, they said, the U.S. team’s actions had made it harder for them to comply with U.S. wishes.
There's a lot here. First, this is the sort of discussion that is missing in many public depictions of gaming events. Alterman was constrained by the event sponsor's desire to keep details of the scenario private, but the generalized insight he describes is probably of at least as much value to him and the other participants as any specifics of the game outcome.
My sense is that games are a largely unexploited avenue to bring participants to a better understanding of their own perceptions and misperceptions. In Alterman's example, the stark contrast between the U.S. team's sense of agency and the view of other actors will likely stick with him to a greater extent than hearing someone lecture about the pitfalls of psychological decision-making biases. Thomas Schelling viewed the RAND crisis games of the 50's and 60's as a tool uniquely suited to examining perceptual factors (discussed briefly here), but I'm not aware of much work since then on the subject, nor have I seen much application of social psychology to gaming, with a couple of exceptions that I should blog about sometime.
The role of the debrief session is key in all of this, as it was for Alterman's experience. Unfortunately, it's much easier to say how important debriefing is than it is to concisely define what it is about a debrief that makes it effective. How would you set up a game and the subsequent debrief to address perception and decision-making biases directly? Multiple cells, for starters, would be a key factor, as Schelling would have said, with closed information conditions. The debrief would have to be substantial enough to give each cell the opportunity to understand what was going on in each of the other cells, and the debriefer would need to be focused on drawing out the differing perceptions of the participants. Beyond that, I'm really not sure.
Thursday, March 19, 2009
More on risk, uncertainty, and Nassim Taleb
To follow up on my previous post on Nassim Taleb and his work, there is a critical distinction that is often overlooked between risk and uncertainty.* Risk is quantifiable. Whether or not it has been properly measured, it refers to something that is measurable. Uncertainty is not quantifiable. Risk can be "bought down" or hedged in ways that uncertainty cannot. A standard example of risk is the sort of game you find at a casino, like roulette, where the odds are clearly known. The state of affairs that will result from a war (the outcome or outcomes), on the other hand, is uncertain, relying on too many variables, complex interactions, and unknown unknowns to be meaningfully quantified. The outcomes of risks have known probability distributions. Not so with uncertainties. Here's an example of these concepts in action, in the context of baseball.
This conceptualization of risk and uncertainty is sometimes mapped onto the "known unknown" vs. "unknown unknown" divide, with known unknowns characterized as risks and unknown unknowns as uncertainties. But the distinction between known unknowns and unknown unknowns is based on the knowledge of the observer, while the distinction between risk and uncertainty is in some respect a difference in "knowability." There are plenty of "known uncertainties" in the "known unknown" category, where we are perfectly capable of identifying something we don't know which nonetheless has a probability distribution we do not or cannot know.
Part of Taleb's argument with regards to the financial crisis could be framed in these terms. Financial managers thought they had transformed some types of uncertainty into risk that could be reliably estimated through new statistical techniques. But they were operating with assumptions about the underlying probability distribution of events that were unwarranted, meaning that all they had managed to do was conceal significant uncertainties (unquantifiable indeterminacies) that ultimately came back to bite them. (See this Wired Magazine article for a fascinating description of how this came about.) The more generalized form of this argument is that we tend to operate as though we are facing risks, rather than uncertainties, in part as a result of our psychological biases. (Taleb discussed a variant of this a bit in the podcast I referred to in the earlier post.)
This may all seem rather esoteric, but in a world where there is a fresh appreciation for the limits of statistical knowledge there needs to be a way to act under uncertainty. The elemental caution reflected in Taleb's advice on operating in what he would call the "fourth quadrant" and what I would call conditions dominated by uncertainty is as reasonable a response as any to decision-making in this type of environment. When faced with less-structured problems, dominated by uncertainties and unknown unknowns, highly structured analytic tools are frequently neither appropriate nor helpful. Gaming can be one of the least-structured analytic approaches, which limits its outputs but allows it to constructively address issues characterized by deep uncertainty.
* These definitions do not conform to popular usage of either term, but they are generally used in this way in policy analysis and represent a helpful way to characterize different types of indeterminacy. Risk in this case is not limited to costs or negative events, but instead applies to probabilistic outcomes both positive and negative.
This conceptualization of risk and uncertainty is sometimes mapped onto the "known unknown" vs. "unknown unknown" divide, with known unknowns characterized as risks and unknown unknowns as uncertainties. But the distinction between known unknowns and unknown unknowns is based on the knowledge of the observer, while the distinction between risk and uncertainty is in some respect a difference in "knowability." There are plenty of "known uncertainties" in the "known unknown" category, where we are perfectly capable of identifying something we don't know which nonetheless has a probability distribution we do not or cannot know.
Part of Taleb's argument with regards to the financial crisis could be framed in these terms. Financial managers thought they had transformed some types of uncertainty into risk that could be reliably estimated through new statistical techniques. But they were operating with assumptions about the underlying probability distribution of events that were unwarranted, meaning that all they had managed to do was conceal significant uncertainties (unquantifiable indeterminacies) that ultimately came back to bite them. (See this Wired Magazine article for a fascinating description of how this came about.) The more generalized form of this argument is that we tend to operate as though we are facing risks, rather than uncertainties, in part as a result of our psychological biases. (Taleb discussed a variant of this a bit in the podcast I referred to in the earlier post.)
This may all seem rather esoteric, but in a world where there is a fresh appreciation for the limits of statistical knowledge there needs to be a way to act under uncertainty. The elemental caution reflected in Taleb's advice on operating in what he would call the "fourth quadrant" and what I would call conditions dominated by uncertainty is as reasonable a response as any to decision-making in this type of environment. When faced with less-structured problems, dominated by uncertainties and unknown unknowns, highly structured analytic tools are frequently neither appropriate nor helpful. Gaming can be one of the least-structured analytic approaches, which limits its outputs but allows it to constructively address issues characterized by deep uncertainty.
* These definitions do not conform to popular usage of either term, but they are generally used in this way in policy analysis and represent a helpful way to characterize different types of indeterminacy. Risk in this case is not limited to costs or negative events, but instead applies to probabilistic outcomes both positive and negative.
Wednesday, March 18, 2009
The uncertain world of Nassim Taleb
A trader for 20 years, Nassim Taleb became fascinated by decision research as a result of watching his colleagues and the financial industry in general. Two books (so far) have resulted from this fascination: The Black Swan and Fooled by Randomness. I haven't had the chance to read either just yet, but I recently found this podcast of an interview with Taleb, in which he covers some of the same ground he goes over in his books. He is not an eloquent speaker and is occasionally hard to follow, but I enjoyed this conversation greatly. Russ Roberts struck me as a skilled interviewer, though I'm not sure how accessible some parts of the discussion will be to those without some economics-related background (give it a listen anyway... it's thought-provoking and worthwhile).
Taleb's work is an example of the recent trend of books about decision-making research which are aimed at a (relatively) broad audience. Blink, by Malcolm Gladwell, is the best example of this phenomenon I can think of offhand. In addition to demonstrating a heightened sensitivity to the role played by psychological decision-making biases, Taleb is particularly concerned with the use of statistics and related tools for decision-making purposes for which they are ill-suited, in his opinion. He has a point. About 55 minutes into the podcast, he and Roberts talk a bit about Taleb's distinction between what he calls "ludic" and "non-ludic" (also referred to as "ecologic") environments. Ludic decision-making environments are those where the "rules" are known, with no ambiguity, such as in a standard board game. Non-ludic environments are those where the rules are unknown and/or don't apply. Instead of a lack of ambiguity, non-ludic environments are characterized by "real world uncertainty." One of Taleb's basic concerns is that people seem to approach many non-ludic situations as thought they were operating in a ludic environment, using decision-making tools (such as certain applications of statistics, but also including innate psychological decision-making biases and heuristics) with a ludic basis and which do not necessarily provide the right kind of decision support for a non-ludic environment. Another way of saying this might be that people too often ignore the existence of unknown unknowns, to their peril.
This article by Taleb (found via Mapping Strategy, which has some intriguing thoughts on the subject as well) enumerates more of his ideas, focusing on his case against the misuse of statistics rather than the decision-making biases he spends most of the podcast talking about. It's interesting stuff. Our limited ability to plan for and consider the future based solely on backwards-looking models suggests that other methods are necessary. Mapping Strategy sees this as an endorsement of alternative tools like scenario planning, and I see a role for gaming to play in trying to understand our messy, non-ludic, ecologic world.
Taleb's work is an example of the recent trend of books about decision-making research which are aimed at a (relatively) broad audience. Blink, by Malcolm Gladwell, is the best example of this phenomenon I can think of offhand. In addition to demonstrating a heightened sensitivity to the role played by psychological decision-making biases, Taleb is particularly concerned with the use of statistics and related tools for decision-making purposes for which they are ill-suited, in his opinion. He has a point. About 55 minutes into the podcast, he and Roberts talk a bit about Taleb's distinction between what he calls "ludic" and "non-ludic" (also referred to as "ecologic") environments. Ludic decision-making environments are those where the "rules" are known, with no ambiguity, such as in a standard board game. Non-ludic environments are those where the rules are unknown and/or don't apply. Instead of a lack of ambiguity, non-ludic environments are characterized by "real world uncertainty." One of Taleb's basic concerns is that people seem to approach many non-ludic situations as thought they were operating in a ludic environment, using decision-making tools (such as certain applications of statistics, but also including innate psychological decision-making biases and heuristics) with a ludic basis and which do not necessarily provide the right kind of decision support for a non-ludic environment. Another way of saying this might be that people too often ignore the existence of unknown unknowns, to their peril.
This article by Taleb (found via Mapping Strategy, which has some intriguing thoughts on the subject as well) enumerates more of his ideas, focusing on his case against the misuse of statistics rather than the decision-making biases he spends most of the podcast talking about. It's interesting stuff. Our limited ability to plan for and consider the future based solely on backwards-looking models suggests that other methods are necessary. Mapping Strategy sees this as an endorsement of alternative tools like scenario planning, and I see a role for gaming to play in trying to understand our messy, non-ludic, ecologic world.
Tuesday, February 24, 2009
Moral dilemmas , gaming, and the Kobayashi Maru
Rex Brynen at PaxSims has an interesting post up wondering whether it would be helpful to provide training to peacekeepers in the form of Kobayashi Maru-like scenarios (scenarios for which there is no "good" answer, no way to win, and no way to avoid a severe negative outcome), as a way of preparing them for the wrenching moral choices they might encounter in the course of their work:
This got me back to thinking about the potential role of gaming in preparing participants for difficult moral challenges by means of acclimating them to decision-making in that sort of environment (something I looked at a bit in my thesis). In their 1977 book, Decision Making: A Psychological Analysis of Conflict, Choice, and Commitment, Irving Janis and Leon Mann described a similar phenomenon they called "outcome psychodrama," which would involve a sort of "emotional inoculation" for decision-makers ahead of time to better prepare them for the emotional states they might experience under a decision-making situation. They believed that gaming might be a particularly appropriate tool for outcome psychodrama "[f]or group discussions whose consequences are highly dependent upon the reactions and countermoves of other people." In contrast to the way most games are run, Janis and Mann suggested running the similar scenarios multiple times with the same participants, examining likely, favorable, and worst case results for each of the available alternatives.
As an example of the danger in ignoring the emotional factors in decision-making, consider Bernard Brodie’s description of the U.S. decision to defend South Korea in 1950 after having announced that it outside the U.S. defensive perimeter in the region just months earlier:
Science fiction fans will immediately recognize this as a reference to a Starfleet training simulation featured in the movie Star Trek II: The Wrath of Khan. In the Kobyashi Maru scenario, trainees were faced with impossible choices in an unwinnable situation: did they answer a distress call from a damaged frieghter, only to find their ship destroyed? Or did they ignore the call, only to see the freighter destroyed? It was meant to be a test of character, and an evaluation of how would-be officers confronted such dilemmas.
Now, readers who haven’t the slightest interest in Star Trek needn’t worry that this blog post will slip into excessive Trekkism. Rather, it occurs to me that there may be some value in designing peacebuilding/humanitarian assistance operations in which participants are confronted with situations that truly have no good answers. I mean this, moreover, not simply in that they face resource shortages and hence opportunity costs associated with actions (something that the Carena simulation does very well), but rather that no matter what they do, they are forced to confront gut-wrenching moral choices.…and so forth. The point would be not so much which particular choice was made, but how it was made—providing an opportunity for participants to reflect on the the moral and practical calculus involved.
- Does one—for example—pull humanitarian workers out of a dangerous area, knowing locals will die? Or does one keep them there, knowing that no matter what security precautions are taken there is a significant risk of staff being killed?
- Do you authorize an airstrike against a high-value insurgent leader, knowing that there is a near-certainty of significant civilian casualties?
- Do you pay “taxes” to a local militia to enable access to a needy population—knowing that the doing so strengthens their capacity to engage in such predatory activities?
- Do peacekeepers fight in to protect civilians from massacre, even if they believe they lack the capability to win and might thereby be slaughtered as well? (Yes, I’m thinking here of Srebrenica, although it could equally be applied to some of the choices that MONUC has made in DR Congo.)
This got me back to thinking about the potential role of gaming in preparing participants for difficult moral challenges by means of acclimating them to decision-making in that sort of environment (something I looked at a bit in my thesis). In their 1977 book, Decision Making: A Psychological Analysis of Conflict, Choice, and Commitment, Irving Janis and Leon Mann described a similar phenomenon they called "outcome psychodrama," which would involve a sort of "emotional inoculation" for decision-makers ahead of time to better prepare them for the emotional states they might experience under a decision-making situation. They believed that gaming might be a particularly appropriate tool for outcome psychodrama "[f]or group discussions whose consequences are highly dependent upon the reactions and countermoves of other people." In contrast to the way most games are run, Janis and Mann suggested running the similar scenarios multiple times with the same participants, examining likely, favorable, and worst case results for each of the available alternatives.
As an example of the danger in ignoring the emotional factors in decision-making, consider Bernard Brodie’s description of the U.S. decision to defend South Korea in 1950 after having announced that it outside the U.S. defensive perimeter in the region just months earlier:
“President Truman and his Secretary of State had endorsed the well-known and obviously rational opposition of the military to becoming involved on mainland Asia. When the crisis came, all, including the military, immediately reversed their position. Apparently few had asked themselves: How will we feel, and how will we in consequence respond, if there is a flagrant attack upon this state and this government that we have ourselves set up and all but recently withdrawn from on the now disproved assumption that they were no longer in danger? Will we really stand by and let them go down?” (Bernard Brodie, War and Politics, 60-61)Brodie suggested that this was the result of "the tendency of people in office to use formulae rather than imaginative rumination to in projecting their own behavior into the future." "Imaginative rumination" is something that free-form gaming can offer the opportunity to pursue. This goes further than the Kobayashi Maru no-win scenario, but seems like it could have significant value. I am not aware of games having been built for this purpose, but John Hanley discussed a number of participants from the Global War Game who reported having experienced powerful emotions surounding the sorts of "gut-wrenching moral choices" Rex proposes. In the Global War Game, these often revolved around the use of nuclear weapons. Hanley mentioned some participants reporting that "[e]ven simulated momentous decisions cost them some sleep."
Saturday, February 16, 2008
Thesis: Decision-Making and Gaming
Here is my Fletcher thesis, looking at how decision-making theories from political science and psychology can be used to better understand free-form games. I can't help but think of this as a work in progress, which is why it has taken me a while to get comfortable with the idea of posting it in its present form. The bibliography (which I may revise and post separately) might be the most useful part of the paper for people interested in this sort of thing.
The Application of Decision-Making Theories to Free-Form Gaming
Comments and questions are very welcome.
(Google Documents seems to have done something strange to the footnotes. Anyone who is interested can email me or leave a comment and I'll send a file in .pdf or .doc format directly.)
The Application of Decision-Making Theories to Free-Form Gaming
Comments and questions are very welcome.
(Google Documents seems to have done something strange to the footnotes. Anyone who is interested can email me or leave a comment and I'll send a file in .pdf or .doc format directly.)
Wednesday, February 14, 2007
The Simulation Heuristic
I finished another RAND monograph recently (Implications of Modern Decision Science for Military Decision-Support Systems), which is too long and too dense and too useful to sum up in a single post. It's available for free online, but I wound up buying a copy from Amazon after realizing how relevant it was for me (Amazon was cheaper than buying it through RAND this time).
The first section deals with the history and development of decision science. It's a great, concise summary of the psychological aspects of decision-making theory, including a description of the split between what the authors refer to as the Heuristics and Biases Paradigm and the Naturalistic School of decision science. More on this in the future, but something that struck me was the description of what some call the simulation heuristic, and which the monograph refers to as "imaginability" (as a subset of memory biases) or the "availability heuristic". The basic idea is that people see things as more likely if they can visualize them more easily. That echoes Robert Levine's criticism (referred to here) that a game could lead participants to believe that the events depicted were probable, as opposed to merely plausible. In the simulation heuristic formulation, it is the very act of making it easier to imagine an event that increases the perception of its probability.
In a sense, that might not be a bad thing, in that one of the benefits to simulation and gaming can be to make a given circumstance seem more real, for the purposes of spurring people to action, or taking a threat seriously, or provoking more thoughtful analysis. But Levine's point stands to a certain extent; what about the other, unsimulated, ungamed possibilities that are accorded less priority because the simulated case is assigned a higher probability than is warranted? The simplest answer I can see right now is that rather than doing less gaming, perhaps the simulation heuristic should in fact lead to doing more gaming, as a tool of exploratory analysis, with an eye towards keeping in mind the vast multitudes of possible futures (more on this sometime soon, I hope). Another answer might just be that gaming is not necessarily by itself a good indicator of the likelihood of a given set of events. That seems like a no-brainer, but perhaps it's difficult to keep that in mind when presented with a realistic game. How do the pros deal with this issue?
The first section deals with the history and development of decision science. It's a great, concise summary of the psychological aspects of decision-making theory, including a description of the split between what the authors refer to as the Heuristics and Biases Paradigm and the Naturalistic School of decision science. More on this in the future, but something that struck me was the description of what some call the simulation heuristic, and which the monograph refers to as "imaginability" (as a subset of memory biases) or the "availability heuristic". The basic idea is that people see things as more likely if they can visualize them more easily. That echoes Robert Levine's criticism (referred to here) that a game could lead participants to believe that the events depicted were probable, as opposed to merely plausible. In the simulation heuristic formulation, it is the very act of making it easier to imagine an event that increases the perception of its probability.
In a sense, that might not be a bad thing, in that one of the benefits to simulation and gaming can be to make a given circumstance seem more real, for the purposes of spurring people to action, or taking a threat seriously, or provoking more thoughtful analysis. But Levine's point stands to a certain extent; what about the other, unsimulated, ungamed possibilities that are accorded less priority because the simulated case is assigned a higher probability than is warranted? The simplest answer I can see right now is that rather than doing less gaming, perhaps the simulation heuristic should in fact lead to doing more gaming, as a tool of exploratory analysis, with an eye towards keeping in mind the vast multitudes of possible futures (more on this sometime soon, I hope). Another answer might just be that gaming is not necessarily by itself a good indicator of the likelihood of a given set of events. That seems like a no-brainer, but perhaps it's difficult to keep that in mind when presented with a realistic game. How do the pros deal with this issue?
Subscribe to:
Posts (Atom)
