Scenario Planning Literature Review

Scenario Planning Literature Review DRAFT FOR REVIEW U.S. Army Corps of Engineers Institute for Water Resources October 2004 Views, opinions an/or ...
Author: Eric Powell
3 downloads 2 Views 444KB Size
Scenario Planning Literature Review DRAFT FOR REVIEW

U.S. Army Corps of Engineers Institute for Water Resources October 2004

Views, opinions an/or findings contained in this report are those of the author(s) and should not be construed as an official Department of the Army position, policy or decision unless so designated by other official documentation.

Scenario Planning Literature Review DRAFT FOR REVIEW Prepared by Charles Yoe, Ph.D. In Association with Planning and Management Consultants, Ltd PMCL@CDM A CDM Company 2845 South Illinois Avenue P.O. Box 1316 Carbondale, IL 62903 (618) 549-2832 A Report Submitted to: U.S. Army Corps of Engineers Institute for Water Resources Casey Building 7701 Telegraph Road Alexandria, VA 22315-3868 Under Contract # DACW72-99-D-0005 Task Order # 130 October 2004

Contents Tables .................................................................................................................................. v Section 1 Introduction ...................................................................................................... 1-1 Section 2 Methodology .................................................................................................... 2-1 Section 3 Scenario Planning Overview............................................................................ 3-1 Section 4 Scenario Planning Literature............................................................................ 4-1 4.1 Scenarios in Corps Context ............................................................................................. 4-1 4.2 Early History of Scenario Planning ................................................................................. 4-2 4.2.1 What Is It?........................................................................................................................... 4-2 4.2.2 Why Do It? ........................................................................................................................... 4-3 4.2.3 Who Is Doing It? .................................................................................................................. 4-5

4.3 Purpose of Scenario Planning ......................................................................................... 4-6 4.3.1 Scenario Planning Is Strategic ............................................................................................ 4-6 4.3.2 The Shell Experience .......................................................................................................... 4-7

4.4 Forecasts Are Usually Wrong ......................................................................................... 4-9 4.5 Constructing Scenarios .................................................................................................. 4-10 4.6 Adapting Scenario Planning to the Corps ..................................................................... 4-16 4.7 Selected Issues .............................................................................................................. 4-19 4.7.1 The Role of Stakeholders .................................................................................................. 4-19 4.7.2 The Corps Culture of Uncertainty ...................................................................................... 4-20

Section 5 Scenarios in Water and Other Resource Contexts ........................................... 5-1 Section 6 Expert Opinion, Subjective Probability and Sensitivity Analysis in Related Literatures .................................................................................................................. 6-1 6.1 Aerospace Sector ............................................................................................................. 6-1 6.2 Military Intelligence ........................................................................................................ 6-3 6.3 Probabilistic Risk Analysis ............................................................................................. 6-7 6.3.1 6.3.2 6.3.3 6.3.4

Subjective Data: Spread ...................................................................................................... 6-9 Subjective Data: Dependence ............................................................................................. 6-9 Subjective Data: Reproducibility ........................................................................................ 6-9 Subjective Data: Calibration ............................................................................................. 6-10

6.4 Policy Analysis.............................................................................................................. 6-11 6.5 Sensitivity Analysis ....................................................................................................... 6-13

Section 7 Bibliography .................................................................................................... 7-1



iii

Contents

iv



Tables Table 4-1

Early Users of Scenario-Driven Planning ............................................. 4-5

Table 4-2

General Electric Scenario Process ........................................................ 4-11

Table 6-1A NASA Risk Assessment Matrix ............................................................. 6-2 Table 6-1B NASA Risk Assessment Matrix ............................................................. 6-3 Table 6-1C NASA Risk Assessment Matrix ............................................................. 6-3



Table 6-2

Reliability and Accuracy Ratings ........................................................... 6-5

Table 6-3

A Kent Chart for Estimating Terms and Degrees of Probability ...... 6-6

v

Tables

vi



Section 1 Introduction Uncertainty is the reason for scenario planning. We live in a world with an increasingly rapid pace of change. Environmental challenges to organizations and their goals are developing progressively faster. The novelty, complexity and speed of these changes have increased the likelihood of strategic surprises. It is not too much of a stretch to suggest these changes have guaranteed strategic surprises. Scenario planning is one technique developed in the latter part of the twentieth century for dealing with the pervasive uncertainty that confronts modern decision makers. Scenario planning is not forecasting. In fact scenario planning is rooted in the proposition that all forecasts are wrong. This approach to planning considers scenarios to be instrumental in understanding and anticipating environmental trends. It relies on the construction of alternative scenarios as a form of sensitivity analysis performed on the most likely scenario. As such, the scenario planning approach is conceptually quite compatible with the U.S. Army Corps of Engineers (Corps) current planning process as detailed in the Economic and Environmental Principles and Guidelines for Water and Related Land Resources Implementation Studies (P&G) of March 10, 1983 (U.S. Water Resources Council [WRC], 1983) and the Planning Manual (Yoe and Orth, 1996). Whether it ultimately proves to be pragmatically compatible or not remains to be seen. This literature review summarizes the results of the Task 1 - Literature Search and Review for the subject scope of work. The general purpose of this task (see textbox) was to identify scenario planning techniques of the most possible interest to the Corps Civil Works Planning Process. This review comprises six sections. Section Two briefly describes the methodology of this literature search. Section Three provides an overview description of two main thrusts in the use of scenario planning in the literature. This overview draws on knowledge gained from the literature review but does not cite the literature. Its purpose is to provide the uninitiated reader with a basic understanding of the topic so that he/she can better appreciate the literature review summary that follows. Section Four reviews the literature related directly to scenario planning as a methodology. Section Five reviews the literature related to the use of scenarios in water resources and other policy-making areas. Section Six provides an abbreviated review of some related literature that addresses the use of opinion and subjective probability to address uncertainty in policy decisions.

TASK 1 – Literature Search and Review The contractor will conduct a review of academic and private sector literature, documents and research, on scenario planning and scenario-based decision-making. The contractor will highlight applications of scenario planning techniques, which have the most promise for use in relevant Corps studies and will document results of the literature search. The documentation will be of such form so ii can be included in sections of the research report, describing the history of scenario planning and its usage in water resource applications. The government will provide comments on this literature review documentation to the contractor. A one-day meeting will be held following the literature review to discuss previous scenario planning techniques, which show the most promise for application in Corps Civil Works projects.



1-1

Section 1 Introduction

1-2



Section 2 Methodology This literature search focused on the seminal articles in the field, methodological articles, textbooks and the much broader field of applications of scenario analysis. Virtually all of the seminal articles have been reviewed along with many of the early methodological articles. These articles have been used to provide the historical background and the description of scenario planning that follows in the Section Three. These articles are found among the extensive bibliography which accompanies this review. The scenario planning literature is mature enough to have produced several good textbooks which were very useful for the general methodological descriptions that follow. A search for specific water resource applications of scenario analysis proved somewhat disappointing. Although there is no shortage of articles addressing the use of scenarios in water resource and other resource related applications, they are not generally consistent with the methodological descriptions of scenario planning. Consequently, the review is divided along two lines. The first, describes a planning method called scenario planning. This is the basic literature that presents a framework for thinking about decision-making under uncertainty in a (strategic) planning context. It is “big picture“ and generally applicable to the Corps larger planning process concerns. Then there is a related but different literature that describes the use of scenario analysis in what may be loosely described as decision-making contexts. This second branch of the literature tends to focus much more narrowly on specific models and techniques for generating scenarios which were not always or often used in what one might call a scenario planning process. A third body of literature was investigated for its general interest to the Corps planning process. This might be called uncertainty. The uncertainty literature is vast and it was not reasonable to review it for this task. However, scenario planning and many of the scenario applications rely on expert opinion and subjective probabilities. These two subjects were reviewed in order to provide some access to this literature of potential interest. The search itself made extensive use of several search engines. These included the services of the Loyola-Notre Dame Library and the Libraries of the University of Maryland System. In addition, extensive use was made of two commercial search engines. One of these was Ingenta. The other was Questia. They are described briefly in the following textbox because of their potential interest to researchers among this review’s readers. Searches were run on a wide variety of keywords, authors and titles. The resulting bibliographies were screened for relevance by their abstracts. The most promising articles were then obtained from the electronic libraries of the College and University as well as from the commercial services themselves. The best of these are cited in the pages that follow. The bibliography includes all those articles and publications that made the first screening cut. To the extent possible, this literature review has attempted to deemphasize the specific details of the articles in favor of drawing on conceptual issues and methodologies that would have value in moving the Corps planning process in a direction of more effectively dealing with significant uncertainties encountered in planning.



2-1

Section 2 Methodology

Ingenta www.ingenta.com Since its launch in May 1998, Ingenta has developed and grown to become the leading Web infomediary empowering the exchange of academic and professional content online. Ingenta supplies access to: 6,000+ full-text online publications and 27,000+ publications. It serves a growing global audience of 260+ academic and professional publishers and 14,000+ academic, research and corporate libraries and institutions, incorporating 25 million users worldwide. Questia www.questia.com Questia is the first online library that provides 24/7 access to the world’s largest online collection of books and journal articles in the humanities and social sciences, plus magazine and newspaper articles.

2-2



Section 3 Scenario Planning Overview Uncertainty is the key to scenario planning. Scenario planning has developed as a systematic approach to coping with uncertainty in strategic planning and policy-making contexts. The Corps planning process can itself be described as a variation on the scenario planning theme. The Corps planners identify several future scenarios and choose a plan based upon the differences in key values and variables that are identified in a comparison of these scenarios. Every Corps planning process identifies a most likely future condition without any specific action taken by the Corps and its planning partners. This serves as a benchmark against which other alternative future scenarios are compared. For each plan formulated, the Corps planners develop a most likely future condition with that plan implemented. Plans are evaluated by comparing such values as expected annual flood damages, habitat units, water quality, costs and the like in their “without-project“ and “with-project“ future condition scenarios. Planning can be differentiated from ordinary problem solving by its future focus, its planning horizon. The future is fundamentally uncertain and scenario planning is one technique for addressing this uncertain future. The two major threads in the scenario planning/scenario analysis literature are summarized in the context of the Corps planning process and jargon. Subsequent sections consider the topic in its own context and jargon. The first and major thread of scenario planning can be described as a process that develops several without-project conditions rather than a single most likely alternative future without a project, as the Corps normally does. This method, developed for strategic planning by industry, recognizes large uncertainties in the future. Different realizations of the future could lead to quite different views about the best actions to take in the present. The uncertainties are addressed by describing different scenarios for each relevant future state of the world. Then, rather than to choose a plan based on its differences between a withoutand with-project conditions comparison as the Corps currently does, a plan would be evaluated against each of the future scenarios (i.e., the multiple without-project conditions). The plan that performs best across all future without-project conditions is deemed the best plan. A stylized example is offered to illustrate the concept. Consider a hypothetical deep draft navigation project where the future is quite uncertain. Commodity tonnage is considered as an indicative example of the process, in fact there would be multiple variables and values considered. Suppose there are four candidate scenarios with a reasonable chance of being realized if no improvements to the harbor are made. Scenario A is a constant amount of tonnage based on current capacity levels. Scenario B is tonnage growing at 3 percent annually, assuming capacity would be expanded to accommodate the growth. Scenario C is based on normalization of trade with Cuba which could triple or quadruple tonnage within three years. Scenario D is a worsening of geopolitical tensions that result in a bunker mentality that cuts trade in half. These are four different futures. The first two are variations on a status quo theme. The last two are radical but quite plausible departures.



3-1

Section 3 Scenario Planning Overview

The Corps process would be to identify one of these futures as the most likely and then to evaluate all planned improvements against it. One thread of scenario planning would evaluate each plan against Scenarios A, B, C and D. The plan that had the best overall performance would be the recommended plan. This oversimplified description of this thread overlooks a great many details but it does no great harm in return for the general understanding of the process. A second thread of the scenario planning/scenario analysis literature focuses on one or a few uncertain variables in either or both of the without- and with-project conditions. So if the first thread sees the future as black and white options, this second thread sees more shades of gray. The many shades of gray depend on the specific values the uncertain variables take. For example, the actual rate of growth of commerce in Scenario B above may be uncertain. The basic scenario is set as one of growth but actual future, hence project benefits for example, will depend on the actual rate of growth that is realized. The approach here is to use one or more of the many methods devised to address this more limited uncertainty in a more or less agreed upon scenario. These methods could include classical forecasting techniques, subjective probability elicitations, probabilistic risk analysis (PRA) and so on. This relies on a without- and with–project condition comparison similar to the current Corps planning process. The critical difference is that some degree of uncertainty analysis is explicitly introduced into this planning process. In summary, the greatest difference between the Corps current planning process and scenario planning/scenario analysis is that the latter explicitly address uncertainty and it does so in the development of scenarios. Many Corps planning studies make explicit investigations of uncertainties and risks encountered, but these are often done as part of the evaluation of differences between two most likely alternative future scenarios. In practice, when compared to the second thread described previously, this may be a distinction without a difference. The possibility that addressing uncertainties in scenario definition rather than in evaluation may make a difference in some instances is not precluded at this point, however.

3-2



Section 4 Scenario Planning Literature 4.1 Scenarios in Corps Context The Corps planning process is presented in the Planning Guidance Notebook (ER 1105-2-100, 2000) and it is described in detail in The Planning Manual (Yoe and Orth, 1996). Scenarios have always played a significant role in the Corps planning process. Planning can be figuratively thought of as standing at a juncture in the present and trying to choose the most desirable alternative future from among the many alternative futures that could occur. Planning relies on descriptions of existing and historic conditions as well as scenarios that describe the future as best we can. In best planning practice these future conditions describe objective realities and their subjective social contexts as honestly as possible. Scenarios are developed for a specific study area based on different sets of assumptions about actions society will take or will not take as a result of the planning process. The future is uncertain. It can unfold in many different ways. And the future will change, based upon the actions society takes in the present. If a planning partnership between the Corps and its non-federal partners decides to take no action at all following a planning study, the future will unfold in one way. This scenario is called the “without-project condition“ by the Corps planners; it refers to a future without any specific action taken by the partnership to alter the path of the future. If the planning partnership finds that the without-project condition future scenario is undesirable it may decide to take action in the present in order to alter that future. This scenario is called the “with-project condition“ because it identifies the future that is expected to occur if a specific action is taken by the partnership. The scenario that accompanies a specific action plan will be different from the “without-project condition“ scenario. Identifying differences in important situations, events, values, effects, conditions and resources by comparing these two future scenarios provides the evidence that forms the basis for decision making in the planning process. The fundamental problem confronting planners as they develop these scenarios is that the future is uncertain. It is impossible to forecast the future without any action or with a specific action with complete accuracy; but these conditions can always be forecast with more or less accuracy. What is essential in forecasting future scenarios is that the fundamental uncertainty of the task be recognized, acknowledged and dealt with in an open, honest and transparent manner. Increasing complexity and an increasingly rapid pace of change are shaping the twenty-first century world. The spread of an Internet worm affects computers and commerce around the world in a matter of hours. Middle East peace negotiations seem to defy resolution. The frenetic efficiency of Wall Street trading and its instantaneous response to world events give witness to our global economy. It has become increasingly difficult to anticipate future conditions with any degree of certainty. The pace of change continues to accelerate with globalization of the world’s economy and technological advances that affect everything from crop growth and tow boat horsepower to information flows. In addition, the Corps increasing involvement with larger and larger scale projects has complicated the forecast of



4-1

Section 4 Scenario Planning Literature

future conditions with and without a project. Landscape scale projects such as the Upper Mississippi River Navigation Study, the Comprehensive Everglades Restoration Plan and Coastal Louisiana Wetlands 2050 embrace far more complex natural and social systems to consider in forecasts than do smaller localized projects. These are but a few examples of systems that are hard to control and even harder to predict. The very complexity of these systems makes it difficult for any actor to know what actions to take; much depends on what others do and how their strategies change over time. The many actors in this world interact in intricate ways that continually reshape their collective future. In such complex and rapidly changing systems, the actors keep revising their strategies, trying to adapt to shifting circumstances. As they do, they constantly change the circumstances to which other participants are trying to adapt and the pace of change accelerates even more as telecommunications and other technological advances reduce the lag time between action and reaction. Change has forced us to clear new paths in business, in education, in research, in science and engineering. And it may be forcing us to consider new approaches to decision making and planning. With the complexity and rapid change comes uncertainty. It is ubiquitous. With uncertainty comes the necessity for decision-making that is flexible and agile in adapting to change. This literature review has been undertaken to assist the Corps in its search for flexible and agile decision support tools. The specific focus of this review is the scenario planning literature. Although scenario planning as a discipline is nearly half a century old, its relevance is newly affirmed by the growing prevalence of the need to make significant decisions in the face of large uncertainties. This review of the scenario planning literature begins with a brief history of the scenario planning process.

Planners Explore the Future “The capacity to tolerate complexity and welcome contradiction, not the need for simplicity and certainty, is the attribute of an explorer.“ Heinz Pagels, Perfect Symmetry.

4.2 Early History of Scenario Planning 4.2.1 What Is It? This section of the review begins by repeating an important distinction made previously. The future is uncertain. There is a difference between an uncertain variable and an uncertain scenario. There are a great many techniques for addressing the uncertainty in a single critical variable. These include classical forecasting techniques, risk assessment, sensitivity analysis and a variety of other techniques that are not addressed further in this literature review. Scenario planning is a method that is more suitable for addressing the uncertain, rapidly changing and challenging future in a much broader context. “Scenario“ literally means an outline or synopsis of a play. The word scenario is derived from the Italian word scena, scene, that comes from the Latin scaena and it dates from about 1878. Herman Kahn introduced the word to its planning context, roughly a description of possible actions or events in the future, at the RAND Corporation in the 1950s. The first

4-2



Section 4 Scenario Planning Literature

applications of scenarios in a planning context are thought to have been in the military strategy studies done by RAND for the U.S. Government. By the 1960s the Wharton School’s H. Ozbekahn had used scenarios in an urban planning project for Paris, France. The theoretical foundations of scenario forecasting, an important component of scenario planning, were principally developed in the 1970s. Royal Dutch Shell is regularly credited with popularizing and modernizing the use of scenario planning for strategic planning in the early 1970s (Wack, 1985a, 1985b). In fact, Wack asserts it was Royal Dutch Shell that came up with the idea of scenario planning. French (Godet, 1987) and German (Brauers and Weber, 1988) planners have also made early use of these methods. The use of scenario-driven planning spread in the 1970s and by the 1980s it seems to have emerged as a distinct field of study with an extensive literature. Known variously as scenario-driven planning, scenario forecasting, scenario analysis and scenario planning, by the 1980s these approaches had developed a range of sophisticated techniques for addressing uncertainty inherent in a rapidly changing world that arises from alternative futures. Part of the spread of scenario analysis in the United States is attributed to their early use by military strategists who subsequently took jobs in other government agencies and industry, taking their techniques with them (Becker, 1983). A Scenario“. . . is intended to describe a possible but by no means There have always been significant differences of certain set of future conditions. A opinion about what scenarios are, how they can scenario can present future and should be prepared and how they can and conditions in two different ways. . . .a should be used. Several formal methods of scenario snapshot in time, that is, conditions construction were used at the RAND Corporation at some particular instant in the and in the literature. Examples include Delphi future. Alternatively . . . the evolution of events from now to some point of techniques and cross-impact matrices. Some time in the future. . . a “future examples will be provided later in this review. A history.“. . . The latter approach is synthesis of scenario methods began in the 1970s generally preferred . . . because it that drew together in a single framework a variety provides cause and effect of perspectives, including those of professional information.“ (Becker, 1983 p. 96) planners, analysts and line managers (Georgantzas and Acar, 1995). Huss and Honton (1987) described scenario-driven planning as a hybrid of many disciplines which, unlike techniques that simply extrapolated from the past, encouraged planners and managers to think more broadly about the future. They describe a variety of approaches to scenario-driven planning that fall under three major categories: intuitive logics, trend-impact analysis and cross-impact analysis.

4.2.2 Why Do It? Scenario is a frequently used word that probably has a different meaning to everyone who uses it. It is important to be clear how the term is used in the current context. Consider a without-project condition for a flood problem that had different possible rates in which land in the watershed would be developed and converted to impervious surface. Consider a deep draft navigation project with alternative growth rates for commodities. Consider an ecosystem restoration that could produce differing marsh salinities. Each of these examples



4-3

Section 4 Scenario Planning Literature

includes a critical uncertain value that could result in potential wide variations in the future. These differences are not scenarios as the term is used here. In the somewhat dated language of the literature these might be called ”reference projections,” that is, piecemeal extrapolations of past trends (Ackoff, 1981). Ackoff distinguishes these reference projections from the overall reference scenario resulting from putting them together. In other words, the without- (or with-) project condition scenario consist of more than a forecast of a key variable. A scenario is the cumulative result of numerous factors. Scenario-driven planning is a systematic approach to the increasingly important responsibility of general management of advantageously positioning today’s organizations firm in a rapidly changing and complex global environment. In this sense it might be described as using scenarios to achieve a well-structured process of managing uncertain strategic situations. The future neither aligns nor reveals itself for our convenience and as such it is ill-structured for organizational decision-making paradigms. Unlike projections, scenarios do not indicate what the future will look like so much as what the future could look like. Scenario construction stimulates creative ways of thinking that help decision makers and stakeholders break out of established patterns of assessing situations and plans so that they can better adapt to the future. Scenarios are most appropriate under conditions where complexity and uncertainty are high (Schoemaker, 1993). The demand for scenario-driven planning in business has been increasing. Acar notes two reasons for this. First, there is abundant evidence that the strength of the U.S. economy is declining or at least stagnating. Second, using scenarios as a strategic tool has provided a handsome return on the investment it requires. The benefits of scenario planning far outweigh the costs of doing so. He concludes that firms need more people capable of generating strategic change scenarios. Because of its multidisciplinary nature, Acar notes it has been successfully applied in a variety of applications, namely, capital budgeting, career planning, competitive analysis, crisis management, decision support systems (DSS), macroeconomic analysis, marketing, portfolio management and product development. Although scenarios have been used in strategic planning principally to forecast future corporate environments, scenario-driven planning is increasingly of interest to functional managers in diverse business areas. Becker (1983) identifies at least three purposes for scenarios: (1) to estimate if various policies and actions can assist or prevent the conditions of a scenario from coming about; (2) to assess how well alternate policies and strategies would perform under the conditions depicted; and (3) to provide a common background for various groups or individuals involved in planning within an organization. This latter point refers to the so-called “corporate scenario“ that provides a common point of departure for all of an organization’s strategic planning. The Corps has struggled with the notion of a corporate scenario for navigation commerce off and on over the years.

4-4



Section 4 Scenario Planning Literature

4.2.3 Who Is Doing It? A survey by Linneman and Klein (1983) showed how scenario-driven planning was emerging as a common and useful tool among the Fortune 1,000 Industrials. They found only a handful of firms used scenarios in 1974. That number doubled in the next two years. By 1977, 47 firms used multiple scenarios. In 1981, 50 percent (108) of their respondents reported using scenarios. Not all industries adapted this technique with equal vigor; the greatest concentration of users was found in the aerospace and process industries. Table 4-1, adapted from Acar shows some of the industries benefiting from scenario planning. A primary lesson to be taken from this early experience is that TABLE 4-1 a variety of business areas have EARLY USERS OF SCENARIO-DRIVEN benefited by using strategic PLANNING scenarios to help them frame problems and situations. This Industry Sample Literature suggests that scenario-driven Aerospace and Millett and Randles (1986) planning needs to be modified Telecommunications some to suit the nature of the Agriculture Helgason T. and Wallace (1991) “business.“ Thus, the Corps would be well advised to take the Imundo (1986) Banking best of what scenario planning Prebble and Reichel (1988) has to offer and field modify for Chemicals Zentner (1987) their needs. This is not a tool that requires rigid adherence to a Data Processing Schultz (1986) method or set of rules of use. The Gross (1984) Jones (1985); Wack other lesson taken from these Petroleum (1985a and b); Wright and Hill early applications is that (1986) managers should learn to Public Utilities Ports (1985) construct their own scenarios. Experience shows that only then will they use them in strategy design. This suggests that although the Corps planners may be ultimately responsible for the development of scenarios, decision makers need to have more than a passing acquaintance with them. The literature repeatedly makes the point that company-wide capability is important to the success of scenario planning. But equally prevalent is the point that consultants and scenario experts should not impose their own models and scenarios on managers. That suggests to this reviewer that the Corps decision makers, i.e., those responsible for the Corps planning process, need to take a more active role in defining and understanding the scenarios used in planning. The role for stakeholders, especially the non-Federal planning partner is likely to be quite unique as well. Top management support is needed to build an organization-wide capability of scenario planning. Organization-wide capability and top management support are pervasive requirements in the literature. Schwartz (1991) describes these two elements as a pair of switches that can turn a firm’s managers and executives into partners in taking the long view. If both switches are on, then a firm will benefit from scenario-driven planning. The benefit is not more accurate forecasts but “better decisions about the future. “



4-5

Section 4 Scenario Planning Literature

There are fundamental differences between the corporate culture where scenario planning first flourished and the culture of a government agency with a military heritage. This raises some concern about such an agency’s ability to incorporate some of the more valuable lessons learned in the early years of scenario planning. Scenarios are valuable as long as they cause a new form of interaction among those who must decide and act. In the Corps context this would suggest new forms of interaction among decision-makers, planners and stakeholders. Decision makers who at times are removed from the details of the planning process would seem to need to take a more proactive role in an effective adaptation of scenario planning. Acar reports that Wack, who led the scenario planning effort for Shell, had less interest in predicting the future than in liberating management insight and inspiring the long-term view. In scenario planning, managers can re-perceive a strategic situation and discern their assumptions about the situation so that they can improve their decision quality. Successful scenario planning will require the Corps to establish an effective culture of uncertainty throughout the planning process. Schwartz (1991) says scenario thinking is an art, not a science, as he describes the macroenvironmental scenarios that allow a whole array of possible futures to be addressed before a firm’s managers evaluate their responses to them. There are still many methodological problems and difficulties with scenario planning, but experience and research are reducing these issues (Chandler and Cockle, 1982). Chandler provides some discussion of the differences among methods for generating scenarios, but he does not focus on scenario construction processes, a topic taken up later in this review. Klein and Linneman (1984) found trend extrapolation to be the most widely used forecasting technique among Fortune 1,000 corporations followed by scenario writing. But they also noted that most companies use a “very informal scenario writing approach, with little reliance on rigorous methodologies“ (p. 72). Bearing in mind that scenario-driven planning is more than a forecasting technique, attention turns to scenario generating techniques in section 4.4.

4.3 Purpose of Scenario Planning 4.3.1 Scenario Planning Is Strategic Scenario planning is for strategic decision making under uncertainty. A review of the extensive scenario planning literature shows increasingly more attention was paid to the methodology of scenario planning through the 1980s. Since that time the literature has continued to grow. The emphasis in the more recent literature has been on the use of scenarios as a tool for addressing uncertainty. Much of this literature is more oriented toward a discussion of the tools, models and methodologies used to generate scenarios rather than on the methodology of scenario planning itself. This section focuses principally on the methodology of scenario planning. Scenario analysis has been developed as a strategic planning tool to deal explicitly with overconfidence, reliance on one certain estimate for an uncertain future and anchoring in the present (Clemons, 1995). It is a conscious move away from forecasts that rely on trend extrapolation. Scenario planning in its early applications to private industry was used as a strategic planning tool. Firms would not only anticipate their future, through environmental scanning and scenario constructions, but also actively engage in creating it

4-6



Section 4 Scenario Planning Literature

Modern strategic planning has generated its own extensive literature that is not a focus of this review. Clemons describes applications of scenario planning designed to anticipate a company’s future environmental and operational uncertainties and to achieve consensus on the changes that need to be made in the present. Specific examples include financial risks— delays and cost overruns, lack of feasibility; technical risks—staying within the present while making the most of what technology can do; project risks—choosing the right project and completing the chosen project; functionality risks—completed systems may not have the right capabilities because planners misunderstood problems and opportunities or because the problems and opportunities have changed; and political risks—organizational resistance keeps the system from being completed or adopted. Scenario planning explores several alternative futures. Scenarios are: “Developed by blending data and analysis with intuition and creativity, scenario plots must ”hang together” like a well-crafted novel, stretch the imagination without going outside the bounds of believability and consistently address issues that are critical to decision makers,“ (Schriefer and Mercer, 1996). Although there is not a single monolithic definition of scenario analysis, there are some reasonably consistent characteristics of scenario analysis in most of its forms. Among them are the following. Scenario analysis does:

„ Acknowledge uncertainty and highlight the key, critical sources of uncertainty and ambiguity.

„ Develop a range of possible future scenarios for exploration, acknowledging that not all are equally likely and that the future may indeed have aspects from more than one scenario.

„ Develop a range of strategies and future indicators of which strategies may be most critical.

„ Acknowledges that future uncertainties may create discontinuities. Scenario analysis does not:

„ Hide uncertainty or ambiguity. „ Develop a single most likely answer. „ Develop a single strategy to which the firm can commit and the firm can pursue. This is a contingent or conditional plan, goes beyond phasing of a plan, the plan itself can change.

„ Obtain unavailable data or make decisions on available data that may not be relevant to the future planning process.

4.3.2 The Shell Experience Wack’s experience with Shell is worth a brief review in itself. His corporate planning group focused on the development of a first tier of six different scenarios about the global



4-7

Section 4 Scenario Planning Literature

environment. They used a fifteen-year planning horizon. Some of these scenarios suggested that an energy crisis might not be far away. Although many managers were skeptical of these warnings (they were not consistent with extrapolated trends), the energy crisis of 1973 gave Wack’s planners new credibility with Shell. The long-run global scenarios developed by Wack’s corporate planning team were disseminated to the planning departments of Shell’s operating companies. These helped inform the corporate environmental scenarios. Smaller business units were encouraged to use the corporation’s environmental scenarios in their own strategic planning. A second tier of short-term scenarios was developed for near-term planning at Shell. These scenarios differed in their time horizon and their focus on short-term economic and business cycle developments. These scenarios were developed in an effort to directly aid the planning of Shell’s smaller business units. The second-tier scenarios proved to be another important step in the company’s learning process. The third level of scenarios developed at Shell consisted of environmental change scenarios developed by individual business units. When the global scenarios did not focus on the factors critical to business unit managers, they were encouraged to develop their own scenarios. Some of Shell’s business unit managers developed their own global scenarios. The culture at Shell welcomed these differences in scenarios because they stimulated thought and were considered creative rather than destructive. Nonetheless, the corporation could ill afford related business units like chemical and refining developing conflicting scenarios and plans. As Shell gained experience, scenario-driven planning came to be accepted as a helpful alternative to traditional forecasting. Quantitative forecasts still had a role in planning; variables such as GNP, inflation and oil prices were still quantified. Their role was secondary to the overall scenario. Shell’s long-run scenarios remained qualitative in nature. Although they started with six scenarios, Shell managers planned without seriously considering the outlying scenarios. Consequently, Shell corporate planners eventually reduced the number of scenarios to two. In practice these two scenarios were often strongly opposing pairs chosen perhaps to provide a devil’s advocate position (Cosier, 1981a; 1981b; and Schwenk, 1984). The Shell approach, which might be important for water resource planners to note, was that it is impossible to predict the future exactly and dangerous to try. The effective operating principle was that neither scenario is right. But if you’re prepared for both, you’ll be ready to cope with the real world. The essence of the planning process then, was to develop management alternatives that perform best when compared to any of the possible scenarios. This is an important fundamental difference between scenario-driven planning as practiced by Shell and others and the Corps planning process. The Corps identifies a most likely scenario and formulates and evaluates plans against that scenario. Scenario-driven planning says the scenarios are wrong. Any of the scenarios is possible. Plans are then formulated and evaluated against all the possible future scenarios. Shell found it valuable to develop flexible strategies, capable of modification in case of rapid response from rival firms. Managerial plans and strategies were valued for their resilience in all possible combinations of global developments. This suggests that resilience or flexibility

4-8



Section 4 Scenario Planning Literature

might be an evaluation criterion for scenario-driven planning. Scenarios, as used in classical scenario planning, are not about predicting the future so much as perceiving and then reperceiving possible alternative futures. Consequently, adapting the Corps planning process to scenario planning would require a shift in practice from focusing on a most likely alternative future to considering a range of possible futures. Good scenarios challenge decision makers to examine carefully their biases and mindsets. The intent is to have decision makers say, “I can see how that might happen and what we should do about it if it does.“ Developing good scenarios helps organizations to learn, anticipate and plan to cope with uncertainty. Acar offers these summary observations on the process. “Scenario-driven planning has an overriding goal and an underlying mind-set to help companies confront looming challenges and render themselves efficiently adaptive. Its’ overriding goal is to enrich the way managers and their consultants think, learn and feel about strategic situations in the turbulent global environment.“ (footnote) It is in times of rapid and unexpected change that scenario-driven planning has leverage and can make the difference between good and poor decisions.

4.4 Forecasts Are Usually Wrong It is much easier to predict the effects of changes in the environment in which we operate than it is to predict the primary causes of change. It would be far easier to predict the changes in airline security than it would be to predict the World Trade Center attack. Forecasting is an indispensable tool for the Corps planners. But it is limited in its utility and range. Forecasts of future trends, such as fleet compositions and commodities, are seldom value free. They have often been offered as self-serving prophecies for the purpose of shepherding resources and efforts toward specific planning objectives. Scenario planning is a purposeful examination of a complete range of futures that could be realized. It is done to address the uncertainty inherent in planning. It is a process engaged in by decision makers and staff alike. In the Corps context, I suggest it would appropriately include stakeholders as well. When planners believe they can forecast with high certainty, or if they believe they are dealing with a single scenario (the case with traditional Corps planning) their approach to the future is called deterministic. Renn (2000) suggests that such deterministic views are a form of negation or suppression of ambivalence and uncertainty that were often produced by technocracies. In the past, citizens relied on experts to remove uncertainty and ambivalence from their lives. When alternative futures are possible or are believed to exist, planners are said to hold a probabilistic view of the future (Becker, 1983). When a range of future possibilities is used, multiple scenarios can generate important insights that would escape the use of a single scenario. Plans that lead to the more desirable futures or that fare better against all the possibilities are obviously the more attractive plans. Deterministic scenarios are rooted in the desire for a single right answer, anchored in the present, with overconfidence in knowledge and current models. These often provide dangerously conservative strategies. Scenarios enable managers to address strategic uncertainties. Rather than embracing a single view of the future, scenario planning embraces uncertainty (Clemons, 1995).



4-9

Section 4 Scenario Planning Literature

Scenarios mark an improvement over forecasts because they are data-based and managers feel better when decisions are data-based. It is not possible to collect “facts“ about the future. Scenario planning tests pseudo-facts for their likelihood, plausibility, fit, logical connections and links between the present reality and future conditions. Because decisions are rooted in mental models, they are prone to become out-of-date as the environment changes. Yet they remain understandably hard to give up. Scenarios challenge individual and organizational mental models. Scenarios enable decision makers to practice solutions under a wide variety of conditions. A technocracy or expertocracy is no longer an acceptable approach to handling uncertainty. The future is not deterministic and forecasts are usually wrong.

4.5 Constructing Scenarios The literature does not yet include a good, thorough methodological review of how to construct scenarios. In the absence of such a review, this section reviews several techniques found in the literature. General Electric (GE) used a scenario construction approach (Jauch and Glueck, 1988) based on a Delphi expert panel and both trend-impact and cross-impact analyses. The outputs of these techniques were used to develop a range of probable future scenarios. Delphi forecasting has been described as constrained guesswork, but its results are trusted because the panelists selected are experts in their fields. GE’s approach started with an initial determination of the key trends by their planning analysts. This was followed by ”constrained expert guesswork” by one or several panels of outside experts. Trend-impact analysis begins with an outside expert’s assessment of the Delphi panel’s forecast of an environmental trend. Cross-impact analysis is a more complex technique. Its’ output is summarized in a matrix that shows the favorable or unfavorable interaction of likely developments generated earlier by the Delphi panel. An example is provided later in this review. The output from the Delphi panel, cross-impact analysis and trend-impact analysis was then used to develop a series of probable future scenarios. The details of that process were incomplete. There are many different techniques presented in the literature. Georgantzas and Acar (1995) offer a summary of the GE scenario process used to perform environmental analysis from 1960 to the 1980s in Table 4-2. Schreifer and Mercer (1996) argued it is becoming increasingly difficult to anticipate future conditions for any industry. Favoring scenario planning for its penchant to explore several alternative futures, she describes a scenario construction process that includes the following steps. The step identification is Schriefer’s, the summary description this reviewer’s.

„ Know the now—understand current situation and dynamics. Give maximum exposure to present and seek agreement on any assumptions used.

„ Keep it simple—complex scenarios are useless, people cannot understand them. „ Work up the group carefully. Five to seven members.

4-10



Section 4 Scenario Planning Literature

TABLE 4-2 GENERAL ELECTRIC SCENARIO PROCESS Prepare Background

Select Critical Indicators

Establish Past Behavior for Each Indicator

Verify Potential Future Events

Forecast Each Indicator Write Scenarios

Assess overall environmental factors of the sector under investigation „ Demographic and lifestyle „ General business and economic „ Legislative and regulatory „ Scientific and technological Identify the industry’s key indicators (trends) Undertake literature search to identify potential future events impacting the key trends Nominate Delphi panel participants whose expert opinion is credible in evaluating the industry’s future Indicator „ Potential future events „ Experts on indicator Establish historical performance for each indicator Enter these data into a database Analyze reasons for past behavior of each trend „ Demographic and social „ Economic „ Political „ Technological Construct Delphi panel interview artifact Interrogate Delphi panel „ Evaluate past trends „ Assess potential impact of future events „ Assess probability of future events „ Forecast future values Specify and document assumptions for forecasts Specific and document rationale for projected values Operate trend impact analysis and cross impact analysis on the literature search and Delphi output to establish ranges of future values Analyze forecast results Document scenarios

„ Try to stick to an 8- to 10-year setting. Less time than that and people extend what is going on now. More time than that and people are guessing.

„ Be iterative—go back and summarize, remove contradictions, stay on track. „ Blend drivers (key factors causing change)—make them work together do not let one drive the entire scenario—challenge conventional wisdom.

„ Have a “hang-together“ check at the end—challenge it, look for fatal flaws, try to break the scenario.



4-11

Section 4 Scenario Planning Literature

„ Plan to use a given scenario several times—multiple uses lower cost, fits business better than Corps.

„ Use group again and again—there is skill acquisition here, people get better at it the more they do it. Clemons (1995) identified five simple steps that provide a nice conceptual overview of what a scenario might look like. First, identify key uncertainties facing the partnership (firm). This includes environmental and operational uncertainties. Second, rank the environmental uncertainties. Third, select two or three critical uncertainties as the driving uncertainties. Fourth, combine them for future scenarios. Finally, explore each scenario and develop strategies for each scenario. Mercer (1995) argues that scenarios can be simple and the simpler the better. He identifies three basic groups of activities: environmental analysis, scenario planning and corporate strategy. Environmental analysis is stressed by Mercer because the scenarios are only as good as the information they are based upon. High quality analysis of the environment is essential. This is a strong point of the Corps planning process. The planning team needs to be totally immersed in the facts that define the environment they are studying. Scenario development depends less on the facts available than what is in the team’s heads. Environmental analysis is followed by scenario planning. The sequence Mercer describes is very compatible with the Corps way of doing things. That is a plus for possibly integrating the better of the two models. Mercer’s scenario planning has six steps. Step One—decide on the drivers for change. The results of environmental analysis are examined to determine the most important factors that will decide the nature of the future environment within which the organization, or in the case of Civil Works, the plan operates. Planners1 must carefully decide the broad assumptions upon which the scenario will be based. Only then should the key drivers be specifically defined. The purpose of these two tasks is to free people from the preconceptions they bring to the scenario structuring process. Too short a time frame leads to reliance on extrapolation from the present. This is to be avoided. Brainstorm lists of drivers. Apply the 80:20 rule so managers and others can focus on what is truly important. We’re looking for drivers that are subject to significantly different alternative futures, i.e., things that are important and uncertain. Factors that are important but predictable (e.g., hydrology) should be identified in the introduction to the scenarios. Step Two—bring drivers together into a viable framework. Scenarios are more than simple forecasts of unknown variables. It is necessary to link the critical drivers together in a viable framework. Build “event strings“ that link drivers so as to see the linkages and their progression over time. Group drivers into combinations that are meaningful to participants. 1

4-12

‘Planners’ is intentionally used without careful definition here. There is no effort to prescribe how or if this should be done by the Corps. This reviewer has attempted to generalize the original author’s work to the Corps context to make the ideas more accessible to people who have not read the original work. This, unfortunately, results in some rather arbitrary use of language at times.



Section 4 Scenario Planning Literature

For example, many individual drivers may be aggregated into groups like commerce, navigation and ecosystem. This framework is often the most difficult step conceptually. Step Three—produce mini-scenarios. Step Two often produces seven to nine groups of drivers. Now it is time to work out the connections among these groups. What does each group represent? Develop a mini-scenario around each group of drivers. The scenario architects need to stretch their imaginations while remaining believable. Step Four—reduce scenarios. Reduce the mini scenarios to a few (two or three) larger scenarios. Experience suggests managers cannot cope effectively with too many more than three scenarios. Shell used two complementary scenarios. ”Complementary scenarios” means there is no preferred scenario. There is no most likely condition as the Corps uses. So this would mark a change. The idea is not to produce a good/bad scenario or a high/medium/low scenario so much as to produce balanced and reasonable scenarios. Any scenario produced here should be tested. Does it hang together? Does it make sense? Make sure the assumptions are not unrealistic. Step Five—write the scenarios. Produce the scenarios in the form most suitable for use by planners and managers who are going to use the scenarios to develop plans/strategies. They are usually in word form and qualitative. In the Corps process they would become more quantitative as evaluation proceeds. There is a potential issue here in that planners are not personally involved in the way a business’s employees would be. Planners are going to walk away at some point. When the plan is implemented they go on to the next plan. A corporation must continue to live with the impacts of its strategic decisions. So the personnel doing the planning and making the decisions will differ from the corporate model. This may or may not make a difference in terms of the agency’s actors’ willingness to engage in the scenario construction process. Step Six—identify issues arising. Examine the scenarios to identify the most critical outcomes. These are the branching points related to the issues that will have the greatest impact on the future. The scenarios are then used in planning. They become logical devices, ways to present the most important topics to planners and managers so they can address them. They provide a way to test our consistency. Are they internally consistent? The insight they offer as to the general shape of the future is important. At this point it is no longer a theoretical exercise, it is a genuine framework for dealing with the future. Scenarios have been used for corporate strategy, they are a means to an end. They identify forces and long term consequences that must be addressed by plans. The idea is to search not just for an optimal outcome but the best overall outcome. This is defined as one that protects as far as possible against all future threats and exploits major opportunities. In a planning context, the scenarios would be used for formulation, evaluation, comparison and, as a result, for plan selection. The search for a best overall outcome could present a conceptual problem for the Corps planning process, given its current emphasis on the National Economic Development (NED) plan, an optimal outcome. Becker’s (1983) general approach allows a place for the “genius scenario.“ This is a scenario created by one person based on his or her knowledge and experience. Becker offers several common sense thoughts that should not be taken for granted, as common sense is not all that common. Scenarios must avoid containing mutually exclusive events. They need to be



4-13

Section 4 Scenario Planning Literature

internally consistent. Ideally people with different viewpoints participate in their formulation (this of course rules out the genius scenario). The future’s history is evolved in a sequential fashion. Events leading to a branch point are described; it is cause and effect oriented. Becker’s basic scenario characteristics include the following:

„ Select basic characteristics: the few conditions most important to shaping the system or marketplace being studied. These are the drivers.

„ Set the possible range of values that will be studied for the basic characteristics, quantified if possible. Decide the extremes that will be used. Bound the analysis. This needs to be more than variations on a theme.

„ Select the number of scenarios that will be studied. This may be based on combinations of basic characteristics that are internally consistent and sufficiently plausible. Use a middle ground, or most likely, and two other scenarios that represent demanding situations in the marketplace. Recognize the most likely may have a low probability of being realized.

„ Designate the indicator and trends that will be treated in each scenario. Drivers were identified early on, now identify other details of scenarios that are important and useful.

„ List important events. These are developments necessary for the conditions of each scenario to come about and those important to shaping the indicators and trends. This is getting to the more detailed effects.

„ Estimate probabilities of each event in each scenario and the impacts of each on the indicators. This includes estimates of the likelihood of occurrence and influence on each indicator.

„ Project the indicators; quantified values vs. time. „ Prepare narratives. Describe the evolution of conditions in each scenario spotlighting key events/developments, important trends, implications for the system or market place studied and, where possible, implications for strategies, policies and actions. The resulting scenarios are then used to nominate strategies for implementation. Schoemaker in his book, “When and How to Use Scenario Planning: A Heuristic Approach With Illustration“ (1991), offers a list of ten steps in scenario construction.

„ Define the issues you wish to understand better in terms of time frame, scope and decision variables. Review the past to get a feel for degrees of uncertainty and volatility.

„ Identify the major stakeholders or actors who would have an interest in these issues, both those who may be affected by them and those who could influence matters appreciably. Identify their current roles, interests and power positions.

4-14



Section 4 Scenario Planning Literature

„ Make a list of current trends, or predetermined elements, that will affect the variable(s) of interest. Constructing a diagram may be helpful to show interlinkages and causal relationships.

„ Identify key uncertainties, whose resolution will significantly affect the variables of interest to you. Briefly explain why and how these uncertain events matter and examine how they interrelate.

„ Construct two forced scenarios by placing all positive outcomes of key uncertainties in one scenario and all negative outcomes in the other. Add selected trends and predetermined elements to these extreme scenarios.

„ Assess the internal consistency and plausibility of these artificial scenarios. Identify where and why these forced scenarios may be internally inconsistent (in terms of trends and outcome combinations).

„ Eliminate combinations that are not credible or impossible and create new scenarios (two or more) until you have achieved internal consistency. Make sure these new scenarios cover a reasonable range of uncertainty.

„ Assess the revised scenarios in terms of how the key stakeholders would behave in them. Where appropriate, identify topics for further study that might provide stronger support for your scenarios or might lead to revisions of these learning scenarios.

„ After completing additional research, reexamine the internal consistencies of the learning scenarios and assess whether certain interactions should be formalized via a quantitative model. If so, use this model to run some Monte Carlo simulations after obtaining subjective uncertainty ranges (or entire distributions) for key independent variables.

„ Finally, reassess the ranges of uncertainty of the dependent (i.e., target) variables of interest and retrace Steps One through Nine to arrive at decision scenarios that might be given to others to enhance their decision making. Wollenberg (2000) offers a four step process for creating scenarios. The four elements common to scenario analysis are:

„ „ „ „

Definition of the purpose of the scenarios. Information about a system’s structure and major drivers of change. Generation of the scenarios. Implications of the scenarios and use by decision makers.

Ringland’s Checklist (Ringland, 2003b) for developing scenarios comprises the following twelve steps:

„ Identify focal issue or discussion—the key question. „ Key forces in the local environment—factors that will influence success or failure of decisions.



4-15

Section 4 Scenario Planning Literature

„ Driving forces—forces in macro environment that drive key forces in local environment. „ Rank by importance and uncertainty—identify two or three factors most important and most uncertain.

„ Selecting the scenario logics—the two or three factors used to create a visual map of scenarios.

„ Fleshing out the scenarios—How would we get from here to there? What events have to happen for that to come true?

„ Implications for strategy—how does question to be decided look against the scenarios? „ Selection of leading indicators and signposts—how will the history of scenario be tracked?

„ Feed the scenarios back to those consulted—get feedback on created scenarios. „ Discuss the strategic options—generate complete set of options against the scenarios. „ Agree on the implementation plan—owner of process. „ Publicize the scenarios—this is akin to an evaluation step. Some of the literature has suggested that companies sometimes do not know what to do with the scenarios once constructed and so they do not achieve desired results. The key is to ask what the scenarios really mean for the company. In the case of the Corps planning, one might put each plan up against each scenario. For example, if we implement Plan A and Scenario 1 is realized we will have this problem or that risk or opportunity. What will we do about it?

4.6 Adapting Scenario Planning to the Corps There is no real literature on this topic, although the Hobbs article in Section 5 comes very close to this. One of the tasks of this research is to develop a process to use scenario-planning techniques in decision- making conducted by the Corps for inland navigation, deep draft navigation and flood damage reduction studies. This section simply takes the opportunity to review some of the concepts reviewed to this point and to place them in a more familiar Corps planning context. There are two useful distinctions to make in how scenarios are used in scenario-driven planning. One involves distinctly different scenarios the other involves variations of a

4-16

Scenario Analysis Does…

ƒ ƒ ƒ ƒ

Acknowledge uncertainty and highlight key, critical source of uncertainty. Develop a range of possible future scenarios for exploration. Develop a range of strategies and future indictors of strategies that may become most critical. Acknowledge future uncertainties may create discontinuities, rendering current data useless. (Clemons, 1995)



Section 4 Scenario Planning Literature

single scenario. Translating these two notions into the language of the Corps planning process is helpful. Scenario planning as described in the literature involves the identification of distinctly different alternative future without-project conditions. Consider a hypothetical navigation study for a South Atlantic Port. One without-project condition scenario might be an extension of the present into the future. A distinctly different future will result if the United States normalizes relations with Cuba and begins trade with them. Another future might be described by a world situation in which the U.S. and other nations withdraw from the global economy as a result of terrorist activity. Yet another scenario would result if the world moves to global government. The key here is that each scenario is distinctly different. If we go back to the original meaning of the word, these scenarios offer very different plotlines or scripts for the port’s future. The various scenarios in the hypothetical example can differ in a variety of ways including commodity forecasts, fleet forecasts, origin-destination pairs for imports and exports, commodity mixes, trade agreements and so on. Contrast these kinds of scenarios with the more common without-project condition that forecasts that a port will continue to do what it is now doing but will do more of it. With this kind of scenario there are usually a couple of variables that are subject to considerable uncertainty. Let us suppose a commodity forecast is one of these. All the scenarios follow essentially the same plotline, they only vary in their specific details for a single variable. For example, the port may realize a decrease in tonnage, no change, a low increase, a moderate increase or a high increase in tonnage. This makes a great deal of difference to project benefits and its ultimate feasibility but it is not a different scenario. The two will be distinguished in this report by considering the first example one of different scenarios and the second example one of key uncertainties within a given scenario. That leaves open the question of how different the key uncertainties must be before they constitute different scenarios, but that is of minor concern to the questions of interest in this paper. If the plotlines differ then an uncertain future, i.e., different scenarios, is the principle challenge for planners to address. If the plotline is essentially set while the details differ then variable uncertainty is the principle challenge in considering the scenario. It is this latter situation which rises to the surface most often within the Corps Civil Works Program. It could, however, be validly argued that the former is the real challenge to the Corps planners.

Scenario Analysis Does Not…

ƒ ƒ ƒ ƒ

Hide uncertainty or ambiguity. Develop a single most likely answer. Develop a single strategy to which the firm can commit. Obtain unavailable data or make decisions on available data that may not be relevant to the future planning process. (Clemons, 1995)

In its traditional Shell sense, scenario planning involves the identification of alternative future scenarios and then devising a plan or plans that would perform well against any and all of these scenarios. Translating this notion into the Corps planning process is not difficult. This amounts to an admission that the future without a project is uncertain and unknowable. Any one scenario will be wrong. Instead of a single most likely without-project condition scenario, the Corps would identify multiple without-project condition scenarios. Then



4-17

Section 4 Scenario Planning Literature

instead of identifying the NED plan for the comparison of an effectively deterministic without- and with-project condition, decision makers would seek the plan that performs best against all of the reasonable future scenarios identified. The conceptual issue of reconciling the new notion of a best plan with the notion of a NED plan remains. That is not insurmountable in concept, however, because if the scenarios are assigned probabilities an expected value can be calculated. Pragmatically, it is not yet clear that pursuing the maximum expected value of net NED benefits is desirable. At one level, scenario planning can be characterized as essentially the identification and use of alternative without-project conditions. There are many ways in which this can be done and they will be examined in the development of a procedure. Adapting Becker (1983) multiple without-project condition scenarios can be used for at least three purposes. First, they can be used to estimate if various policies and actions can assist or prevent the undesirable conditions of a scenario from coming about. Second, they can be used to assess how well alternate policies and strategies would perform under the conditions depicted, i.e., to estimate risks in choosing certain courses of action. Third, they can be used to provide a common background for various groups or individuals involved in planning within an organization. Scenario analysis helps determine actions organizations should take now; actions they should stop in the future if they are headed toward a scenario that makes them unnecessary or less attractive; and actions they should take if they are headed toward a scenario that makes them necessary or more attractive. A plan with a series of triggers or thresholds might be in order and as scenario uncertainties are resolved, the strategy or measures for responding are there and ready for use. In this sense, we can see scenario planning melding into adaptive management trains of thought. Planning then expands to include identifying contingent possibilities (triggers and the like) so managers can anticipate and respond quickly. This is fundamentally more difficult when considering large scale public works projects than it is for a business making decisions about a product line, however. In some versions of scenario planning application, the firm makes no attempt to determine which scenario is correct or more importantly there is no attempt to rank the scenarios by probability. The firm simply takes a set of future possibilities and determines the appropriate strategic responses to know where it is headed with sufficient lead time (Clemons, 1995). In other versions, several of which are discussed in Section 5, scenarios are weighted by probability of occurrence and more quantitative data are used in the evaluation and decision making process. Scenarios, however constructed, are used for forecasting alternative futures. When these scenarios are subsequently analyzed and used as the basis for developing corporate strategies they give rise to scenario planning, a planning model used by business that represents an alternative to the six step planning model presented in the P&G. Scenarios can be used for strategic planning as the literature amply illustrates or for project planning. In fact, the Corps has tried to do some scenario planning in the past by coming up with a set of commodity forecasts that would be used by all. The Institute for Water Resources (IWR) Inland Waterway Users Board has been used somewhat like this. Experts on the Board help provide scenarios for inland waterborne commerce that can be used to establish infrastructure investment strategies. These examples, though limited, are

4-18



Section 4 Scenario Planning Literature

suggestive of the notion of the Corps corporate scenario. Such scenarios provide a common point of departure for planning efforts. In addition, Corps guidance has provided some ad hoc scenario guidance, e.g., no net gains in tonnage from other ports. Using guidance to settle scenario issues is, however, an anathema to scenario planning. This is precisely the kind of rigid thinking that behavior scenario planning is trying to overcome. Scenario planning is open to all the possibilities in an uncertain future. Much of the Corps guidance functions to constrain the realities Corps planners can continue. In this regard there is a fundamental clash between scenario-driven planning and the Corps regulationheavy culture. It is the writer’s opinion that for scenario-driven planning to succeed in the Corps Civil Works planning program a culture of uncertainty must be established. Planning must embrace uncertainty. It must provide the framework to support solutions that are radically different. It needs to move from precision and inaccuracy to accuracy and imprecision. Planning needs a culture of uncertainty. This is a topic to be addressed at some length in the development of a planning procedure.

4.7

Selected Issues

4.7.1 The Role of Stakeholders Adaptive co-management (ACM) relies on iterative social learning among stakeholders and the on-going adjustment of management decisions to be acceptable to relevant actors. In this respect it is compatible with the planning process, although the learning in ACM was built on the monitoring of past actions. Wollenberg (2000) shows how scenarios can be used as a tool for ACM to enable groups of forest users to not only respond to changes, but also anticipate them. Wollenberg argues that community-level decision-making will be more effective to the extent that it takes account of social and ecological processes at the scale of landscapes or larger. This context mirrors some Corps studies where multiple stakeholders are involved. The author asserts the need for new methods to facilitate community-level decision-making that can account for risks and opportunities with origins at larger scales. Learning is an important part of this process and that learning is facilitated by the use of scenarios. Scenarios are described as stories or ”snapshots” of what might be. Decision makers use them to evaluate what to do now, based on different possible futures. As has already been seen previously, the term scenario is associated with several distinct approaches for gaining information about the future (Millett, 1988; Fischhoff, 1988; Sapio, 1995). Scenario methods can be said to refer to a general category of techniques associated with creative visioning. Participatory Rapid Appraisal (PRA) techniques have been used to elicit people’s vision about the future. Some PRA techniques include the use of possible futures (Slocum and Klaver, 1995) and guided imagery (Borrini-Feyerabend, 1997) exercises. Scenarios differ from these in part by their focus on the analysis of uncertainties, i.e., what other authors call the drivers of change and causal relationships associated with a potential decision. Scenarios encourage critical thinking about risks and systems relationships and this supports learning about problems and solutions. In corporate cultures scenarios have been used to adapt current mental models to rapidly changing circumstances because the “existing mental models include assumptions that are no longer valid or habits of observation that prevent



4-19

Section 4 Scenario Planning Literature

seeing new relationships“ (Wack, 1985b). The ability to break out of their highly structured and policy-based mode of thinking will present a substantial challenge to the Corps ability to adapt scenario-based planning methods or ACM techniques. Scenarios enable people to overcome cognitive biases to (1) undervalue that which is hard to remember or imagine, (2) better remember and give more weight to recent events, (3) underestimate uncertainties, (4) deny evidence that does not support one’s views, (5) overestimate their ability to influence events beyond their control, (6) be overconfident about their own judgments and (7) overestimate the probability of desirable events (Becker, 1983; Barnes, 1984; Bunn and Salo, 1993; Schoemaker, 1993). Van de Klundert (1995) suggests that the application of scenarios has evolved to reflect the historical context in planning. In the 1960s scenarios emphasized prediction based on existing stable trends. In the 1970s and 1980s scenarios began to focus on uncertainty. In the 1990s scenarios have included ”stakeholders” in the discussions around public and shared decision-making. What is the role of stakeholders in scenario analysis? In the Corps case I think they help with the scenario structuring. Scenarios need to be able to integrate different interest groups in the planning process if they are to be useful for public agencies. This need not mean that all groups participate equally in every stage of scenario construction and analysis. Stewart and Scott (1995) found that differences in sophistication among stakeholders in community forests required designing understandable, transparent methods for each participating stakeholder group. Scenario methods are themselves adaptable and have used various forms of stakeholder input to inform the scenario process and help make it relevant to users. Techniques for stimulating creativity and overcoming biases include: (1) using extreme outcomes, not just predictable ones, (2) creating disruptions to historic trends, (3) selecting scenarios that are distinct, not ones that reflect a gradient such as high, medium and low values, or a positive and negative scenario, (4) including undesirable scenarios and (5) starting the construction of the scenario from an imagined future, rather than from extrapolation of current trends (Schoemaker, 1991, 1993; Bunn and Salo, 1993; Wack, 1985a). Remember the purpose of scenarios is not to predict the future but to improve abilities to adapt to it. Thus, such techniques are not as extreme and discontinuous as they may first appear. The cultural attitudes of organizations may make overcoming the natural barriers to creativity difficult to overcome.

4.7.2 The Corps Culture of Uncertainty The overarching question behind this research project seems, to this reviewer, quite clear. It is, quite simply, how will the Corps cope with the world of uncertainty that envelops their Civil Works Program? In other words, what will the Corps culture of uncertainty look like? An evaluation of the current organizational attitudes and approaches to this pervasive uncertainty is well beyond the scope of this review. However, it is quite clear that the Corps military culture with its deference to higher authority, its reliance on established (synonomously past) policy and practices codified in copious detail in Engineering Circulars, Regulations and Pamphlets and the like operates in a reality that is heavily burdened by the past. That heritage, along with the military command structure, makes the Corps less flexible and able to respond effectively and efficiently to the rapidly changing environment of the

4-20



Section 4 Scenario Planning Literature

present. This, reality will go a long way in determining if and how the Corps planning process is modified to take advantage of any potential benefits offered by scenario analysis. As the scenario literature has developed from the 1970s there seems to be a maturation process in the scenario concept and its use. Initially, scenarios seem to have represented an alternative to deterministic forecasts of the future that were increasingly inaccurate and incapable of identify turning points and significant changes in the environment. I am not sure that all authors understood the process in this way, but some were more sophisticated in their thought processes than others. Over time, however, there has been a decided shift toward the notion that scenarios are not forecasts of the future. Much as the story of the ghost of Christmas future in Dicken’s A Christmas Carol scenarios are more like descriptions of what could be than they are what will be. And as was true for scrooge, changes in present behavior can help one not only adapt to the future but to shape it as well. What has evolved from this review is a clearer vision of the differences between scenario analysis and scenario planning than is possible to discern from the early literature alone. Scenario analysis focuses on the definitions and techniques of developing scenarios. It also includes using scenarios in a structured sensitivity analysis of a management measure of some sort. Scenario planning or scenario-driven planning overlaps the definition and development of scenarios but it takes a different focus on the use of the scenarios. The scenarios are used to help formulate and evaluate and choose future courses of action. And although there are examples of such use in water resources and public investment, the greatest use of these planning techniques has been in the arena of strategic planning. This usage is significantly different from the Corps Civil Works water resources planning. One of the major ways in which this is true, is in the private sector’s emphasis on flexible plans and the Corps emphasis on public works infrastructure investments that are to a great extent irreversible, the antithesis of flexibility. One of the keenest innovations of scenario planning is its ability to get managers to focus on the future by taking a fifteen to twenty year view into the distance. In the 1960s it was commonplace for the Corps to be using a 100year planning horizon, which has been shortened to fifty years. For the Corps, its heritage and these differences in planning applications present a choice with respect to making use of scenario-driven planning techniques. Should the Corps adopt or adapt the common practice of scenario planning? Or perhaps it has nothing of value to offer the Corps. As noted previously, the Corps relies on the use of most likely forecasts of the future without-project and with-project. These scenarios, once defined, are treated as rather deterministic. However, individual elements of the scenarios are often treated as risky or uncertain. Expanding and formalizing the manner in which uncertainties within a scenario are treated is one possible response by the Corps. Another would be to take what is useful and adaptable that provides value added to the Corps planning process in at least some situations and adapt it for use. A third and far less likely option is to adapt the scenario-driven planning process as its own. To make any use of the scenario literature will require that the Corps see the future differently. This is not a subtle change that can be made for this study here or that study there. The organization has to recognize the pressing need to acknowledge and address the uncertainty that pervades their Civil Works function. The first step is an organizational one.



4-21

Section 4 Scenario Planning Literature

The Corps must first acknowledge its inability to control the uncertainty in the world around them and then resolve to change its behavior by creating a culture of uncertainty.

4-22



Section 5 Scenarios in Water and Other Resource Contexts The literature pertaining to water and natural resources is voluminous. An Ingenta search for key words “scenario water Upper Yellowstone River Flow and resources“ for example, produced Teleconnections With Pacific Basin Climate fifty-four articles from 1997 to 2003. Variability During the Past Three Centuries All the abstracts were read and the most promising articles were read in their entirety. The majority of this literature took a much narrower view of scenario planning and analysis than is generally helpful to the Corps planning process. A sample abstract, chosen because it is representative of the general lack of germaneness of most of the literature, is provided in the textbox. Relatively few articles provide examples of the primary thread (multiple future conditions against all of which a plan is evaluated) described in Section Three. Most articles that address scenario analysis tend to be of the second thread (techniques for dealing with specific sources of significant uncertainty). Consequently, the format of this section will differ from that of the rest of this review. This section will provide a brief review of the most relevant articles found in this research. One of the more promising areas of the resource literature for scenario planning is climate change. There have been a great number of articles that make some use of, or reference to, scenario analysis. It is quite common to see articles that make use of different scenarios for climate change. Doubling the CO2 levels is an example of such a scenario. The foci of articles are often on the model used to generate these scenarios or the



Climatic Change, July 2003, vol. 59, no. 1-2, pp. 245-262 (18) Graumlich L.J. [1]; Pisaric M.F.J. [2]; Waggoner L.A. [2]; Littell J.S. [2]; King J.C. [2] [1] The Big Sky Institute, Montana State University, Bozeman, MO 59717-3490, U.S.A.; E-mail: [email protected] [2] The Big Sky Institute, Montana State University, Bozeman, MO 597173490, U.S.A. Abstract: Climate variability, coupled with increasing demand is raising concerns about the sustainability of water resources in the western United States. Tree-ring reconstructions of stream flow that extend the observational record by several centuries provide critical information on the short-term variability and multi-decadal trends in water resources. In this study, precipitation sensitive Douglas-fir (Pseudotsuga menzeisii) tree ring records are used to reconstruct annual flow of the Yellowstone River back to A.D. 1706. Linkages between precipitation in the Greater Yellowstone Region and climate variability in the Pacific basin were incorporated into our model by including indices Pacific Ocean interannual and decadal-scale climatic variability, namely the Pacific Decadal Oscillation and the Southern Oscillation. The reconstruction indicates that twentieth century streamflow is not representative of flow during the previous two centuries. With the exception of the 1930s, streamflow during the twentieth century exceeded average flows during the previous 200 years. The drought of the 1930s resulted in the lowest flows during the last three centuries, however, this probably does not represent a worst-case scenario for the Yellowstone as other climate reconstructions indicate more extreme droughts prior to the eigthteenth century.

5-1

Section 5 Scenarios in Water and Other Resources Context

techniques needed to scale these scenarios from a large spatial scale to a smaller scale. Climatic Change is a journal dedicated to the topic that contains numerous articles that make some use of scenarios. The September 1997, Volume 37, Issue 1 edition of the journal contained several useful articles, three of which are summarized as follows. Climate Change and Water Resources, K.D. Frederick and D.C. Major Climatic Change 37 (1): 7-23, September 1997. The authors introduce some current perspectives on global climate change based on then-recent reports of the Intergovernmental Panel on Climate Change (IPCC). Changes in precipitation and runoff patterns, sea level rise, land use and population shifts following from these effects and changes in water demands are presented as potential impacts of a greenhouse effect that would affect water planning and evaluation. Irrigation water demands were identified as particularly sensitive to some of these climate changes. A key emphasis of the article was the substantial uncertainty that remains as to how and when climate will change and how these changes will affect the supply and demand for water at the river basin and watershed levels. In an effort to bound the uncertainty of climate factors the authors explored the influence of non-climate factors such as population, technology, economic conditions, social and political factors, and the values society places on alternative water uses are considered on the supply and demand for water. The authors conclude that our track record anticipating these uncertainties has not been especially good and neither should we expect our ability to anticipate climate change uncertainties to be. Assessing Urban Water Use and the Role of Water Conservation Measures under Climate Uncertainty, J.J. Boland Climatic Change 37 (1): 157-176, September 1997 This article was unique for its focus on the effects of climate change on urban water use. The paper takes a look at the suitability of various water use forecasting models for predicting climate impacts or of the best procedures for assessing this issue. The paper argues that a scenario approach to describing possible changes in climate is useful. It uses six climate change scenarios. None of them is intended as a prediction of climate change but taken together they capture the likely range of uncertainty about climate. This range permits an assessment of the sensitivity of different water management alternatives to climate change. The author also evaluates the IWR-MAIN model as a source of plausible water use forecasts given uncertain future climate. This article used scenarios to describe a portion of the range of uncertainty and as such it represents a slightly different application of scenarios to a problem evaluation. Using Decision Analysis to Include Climate Change in Water Resources Decision Making, B.F. Hobbs, P.T. Chao and B.N. Venkatesh, Climatic Change, 37 (1): 177-202, September 1997 The authors’ take off point for this article is that a necessary condition for uncertainty to be important to managers is that it could affect their decisions. That means the manager’s beliefs about the uncertainty can affect the choice

5-2



Section 5 Scenarios in Water and Other Resources Context

of the most desirable alternative. Uncertainty matters. The sufficient condition is that there must be a significant economic or other loss associated with that change in the decision. This is an important and useful finding for Corps planning. Using the expected cost of ignoring uncertainty (ECIU) the authors offer a framework for approaching uncertainty using a climate change example. Briefly, the ECIU is found by evaluating an expected value like net NED benefits when the probability of the alternative scenario is zero and subtracting from it the expected value of net benefits for the plan that is optimal under the alternative scenario. So, for example (using this reviewer’s numbers) if the best plan with a no climate change scenario has benefits of $100 and the best plan with a climate change scenario has benefits of $60 the ECIU is $40. The article provides a good example of a well-articulated climate change scenario (p. 182) that was applied to two test cases. It may be worth emulating if an example is needed at some future point. A no climate change scenario and one climate change scenario are compared. Of greatest value to the Corps planning process is a five step process using a scenario tree that holds considerable promise for addressing alternative scenarios. The steps, in brief, follow:

„ Determine whether the decision has characteristics that suggest that climate change could be relevant.

„ Evaluate the options under a climate change scenario. „ If net benefits are significantly affected, assess the “regret“ that would occur if a decision was made assuming no climate change but global warming occurs anyway.

„ If the amount of regret is important, construct a decision tree with two or more climate scenarios for evaluating the options. Then evaluate the expected performance of the options under a range of subjective probabilities for the scenarios.

„ For larger projects the benefits of waiting a decade or longer for better information on climate change could be assessed. Assessing Climate Change Implications for Water Resources Planning, A.W. Wood, D.P. Lettenmaeir and R.N. Palmer, Climatic Change, 37 (1): 203-228, September 1997 Noting that many studies show water supply systems are sensitive to climate change, the authors ask if planning methods should be modified accordingly. This paper identifies three principal sources of climate change uncertainty in water resources planning. They are: (a) climate modeling; (b) scaling model results from global to regional levels and (c) water demands. In the process of going from GCM-based studies to water resource management strategies there can be a cascade of uncertainty. The resulting uncertainty makes it difficult to prescribe a set of steps for addressing the unknown pragmatically.



5-3

Section 5 Scenarios in Water and Other Resources Context

Of interest to this review is a simple approach proposed and used in a case study to demonstrate how the planning process might be modified to address these uncertainties. The methodology is generalized from its climate change application to a more traditional planning approach. For ease of explication assume two alternative scenarios A and B. They could be alternative without-project conditions or alternative with-project conditions for the same plan. One could be with climate change the other without; or one with Cuban trade the other without; or one with 5 percent commodity growth the other with 1 percent. The possibilities are endless and there is no need to restrict the approach to two scenarios. First, a plan is selected using traditional planning techniques, i.e., most likely alternative futures without- and with- the project are identified. Assume Scenario A is the most likely with-project condition. The best plan using Scenario A is identified. Then this plan is evaluated under Scenario B. Second, the best plan is picked assuming Scenario B. This plan is then evaluated under conditions of Scenario A. Finally the results of these two analyses are compared. For example, the net benefits of the two best plans could be compared as could the loss of net benefits for each plan if the alternative scenario is realized instead. The authors suggest that if this process is repeated many times for alternative without- and with-project conditions (if appropriate to the situation) it can result in probability distributions of the relevant evaluation metrics. Thus, if Plan 1 under Scenario A yields $100 and under Scenario B it yields $85 and under Scenario C it yields $46 and so on a distribution of both net benefits and the expected losses can be estimated if the scenarios are assigned probabilities of occurrence. This latter point is this reviewer’s observation. In the case study done by the authors, it was found that the prospect of climate change did not play an important role in the best reallocation plan. Water Resources Planning Principles and Evaluation Criteria for Climate Change: Summary and Conclusions, K.D. Frederick, D.C. Major and E.Z. Stakhiv, Climatic Change, 37 (1): 291-313, September 1997 This paper summarizes the findings of the many articles in this volume of Climatic Change. The prospect of anthropogenically-induced climate change presents water planners with a variety of challenges. With respect to the sixstep planning process detailed in the P&G the methods of sensitivity analysis, scenario planning and decision analysis that are encouraged by the P&G are found to be generally appropriate for planning and project evaluation under the prospect of climate change. The review found here does not summarize Frederick’s et al., summary of the articles. The most useful of the articles are reviewed elsewhere. There were several findings among the conceptual issues raised in this article that are interesting because they can be generalized for consideration of other

5-4



Section 5 Scenarios in Water and Other Resources Context

planning problems with substantial uncertainty. Much of what follows is based on the arguments of Frederick et al. and the opinions of this author. Bearing in mind the climate change orientation of these water resource issues, the authors note it is problematic because the timing and nature of the change are both uncertain. And the scale of the uncertainty is, perhaps, different from what planners have approached in the past. These considerations seem to be easily generalized beyond climate change uncertainty for some planning investigations. Any significant uncertainty can raise issues about intergenerational equity. Generalizing from the authors’ findings it can be argued that when faced with large and significant uncertainties, water resource plans should maintain options and build in dynamic flexibility. The climate change article suggests that it may well be worth waiting and building a project later rather than now if doing so reduces the probability of a bad outcome. Particularly interesting were the authors’ comments on adaptation through infrastructure investments, beginning at p. 300. Generalizing, uncertainties can sometimes cause significant shifts in hydrologic regimes (in the case of climate change) or in economic factors like supply and demand or supply chains, social values and ecosystem performance. When the magnitude, timing and direction of the shifts are uncertain it can be very difficult to plan infrastructure investments. Matalas, N.C. (1997) Stochastic Hydrology in the Context of Climate Change Climatic Change, September 1997, vol. 37, no. 1, pp. 89-101 (13) Kluwer Academic Publishers Matalas suggests using what-if analysis to assess the robustness of alternative designs. Frederick et al. suggest that perhaps a robustness criterion can be introduced to the planning process. This would presumably join the completeness, effectiveness, efficiency and acceptability criteria of the P&G. The authors suggest a move away from finding the “right“ design as has traditionally been done in plan evaluation to a new emphasis on finding the robust design. A robust design may not be best under any given scenario but it is fairly good under a wide range of outcomes. Such an idea is intriguing but it represents a sea change in attitude for some planners. This is also an idea that fits well into the scenario planning mentality. Developing resource management systems that are more flexible and responsive to changes in the underlying assumptions about future conditions is a valid approach to addressing uncertainty from any source. The authors assert that institutional flexibility that complements or substitutes for costly infrastructure projects is important for water resources planning. This is a very timely idea. But it marks a change in the way of doing planning that will cause some challenges. For example, the goal of maximizing net NED benefits has, arguably, always been one of hitting a moving target. There have always been uncertainties in project costs and benefits for a variety of reasons, project performance and the future being only two of them.



5-5

Section 5 Scenarios in Water and Other Resources Context

Consequently, identification of a NED plan was an informed guess at best. Creating a culture of uncertainty within the Corps will require flexibility. And in the current instance that flexibility will require making explicit many things that have not been made explicit in the past. For example, the NED plan must be more formally and publicly recognized as a moving target. And the best economic plan may not always be the one with the highest expected value for net NED benefits. The authors here and in several articles in this volume of Climatic Change argue persuasively that the advantages of postponing costly, one-of-a-kind and perhaps irreversible responses to potential future situations with significant uncertainties for as long as possible or at least until the uncertainty can be reduced may often be a viable strategy. The next two articles demonstrated the continuing relevance of both threads of scenario planning to resource management. They demonstrate that multiple and contrasting scenarios can be incorporated into resource management decisions. The first article indicates the importance of sensitivity analysis to explore the residual uncertainty in different scenarios. The scenarios in these instances resulted more from a model-based process than an overtly expert-judgment process such as the Corps is likely to use. They differ from the preceding articles in that they do not focus on water resources. Climate Change and Winter Wheat Management: A Modeling Scenario for Southe-Eastern England, Ghaffari, A., Cook, H.F., Lee, H.C., Climatic Change vol. 55, no. 4, December 2002, pp. 509-533 Weather and climate represent the two major uncertainties in agricultural production. Obtaining a consensus on the direction of climate change is no simple matter as more and more scenarios are created. The authors use the dynamic crop-growth model, CERES-Wheat, to examine crop management responses, including six climate change scenarios for the years 2025 and 2050. It is noted that these scenarios are “probable“ but the probability is not developed. Large uncertainties remain in these scenarios due to a general lack of data. Differences in temperature, CO2 and rainfall were used to create the basic scenarios. The focus of this article is more on the CERES-wheat crop simulation methodology than it is on generally applicable principles that can be adapted by the Corps planners. A sensitivity analysis was conducted using a one-at-a-time investigation of increases in temperature alone. Wheat yields were then constrained to assess crop performance under water-limited production scenarios with different soils. What was interesting was that different management practices like planting dates and nitrogen application were applied to find the best adaptation strategies across all scenarios. This was one of the few examples of the first thread of scenario planning found in the resource literature. There was nothing of specific value to the Corps planning process in this article. Assessing Winter Wheat Responses to Climate Change Scenarios: A Simulation Study in the U.S. Great Plains, A. Weiss, C.J. Hays, J. Won, Climatic Change vol. 58, no. 1-2, May, 2003, pp. 119-147

5-6



Section 5 Scenarios in Water and Other Resources Context

The uncertainty associated with climate change has spawned a large literature. As is common in much of the professional literature a string of related articles often appears on a topic and this article is closely related to the preceding one. The authors consider the effect of climate on hard red winter wheat in the Great Plains region of the U.S. It investigated the effects of two contrasting global climate change projections (one from the UK and one from Canada), on the yield and percent kernel nitrogen content of winter wheat at three locations in Nebraska. These locations represented different moisture conditions. In this article the emphasis was more on the tools used and the implications for agricultural management. The climate scenarios were based on projections using the LARS-WG weather generator along with data from automated weather stations. CERES-Wheat was also used in this study to simulate the responses for two kinds of wheat and two sowing dates. From the resulting analysis proactive steps to meet the challenges of global climate change as represented by these climate scenarios were recommended. A Scenario-Based Stochastic Programming Model for Water Supplies From Highland Lakes, D.W. Watkins, D.C. McKinney, L.S. Lasdon, S.S. Nielsen, Q.W. Martin, International Transaction in Operational Research 7 (2000), pp. 211-230 The authors use a scenario-based, multistage stochastic programming model to explore management of the Highland Lakes by the Lower Colorado River Authority (LCRA) in Central Texas. Thirty scenarios were generated by the model and were solved using both a primal simplex method and Benders decomposition. The results show the amount of water to contract for the coming years is highly dependent on the initial reservoir storage levels. The article tends toward a more technical focus on the modeling technique. Unlike deterministic models that select values of decision variables with ”perfect knowledge of the future,” scenario-based stochastic programming models consider a number of possible futures. This particular model is proposed for use in ”here and now” decision making as well as providing a number of ”wait and see” strategies dependent on which scenario unfolds. The method allows large-scale problems to be decomposed by scenario and solved in a nested manner with inputs represented as a scenario tree. The objective function for the model was a weighted combination of two goals, roughly maximizing water sales revenues and maximizing recreational benefits. A scenario in this model was defined to be a sequence of monthly available flows. Available flows are defined as those in excess of environmental needs and water rights senior to the LCRAs. Monthly inflow data were used to generate scenarios. The model is solved using GAMS and SP/OSL. The authors argue that by explicitly considering a number of inflow scenarios, the stochastic model can determine a contract level that balances interruptible water supply and recreational goals while appropriately hedging against the effects of drought. This article offered little pragmatic information for the Corps planning process.



5-7

Section 5 Scenarios in Water and Other Resources Context

Toward a Scenario Analysis Framework for Energy Footprints, Jiun-Jiun Ferng, Ecological Economics 40 (2002) pp. 53-69 The ecological footprint is an index developed for quantifying humanity’s dependence on ecosystems. It measures land and water areas required to support the resource provision and environmental assimilation necessary to satisfy the consumption needs of a human population. The author proposes a framework that enables scenario analyses of policy instruments that could reduce energy footprints. In other ways he uses scenario analyses to look for ways to reduce energy needs. The purpose of this paper was to demonstrate the feasibility of the framework rather to actually apply it. The author compared the results of one hypothetical scenario to a baseline. The paper uses a six-stage calculation. Final domestic demand is estimated for a new policy instrument using a computable general equilibrium (CGE) model. The use of the CGE model (an equation-based simulation model) is a centerpiece of this research. Next, the sectoral outputs necessary to satisfy the final demand are estimated using an input-output analysis. Stage three estimates the various final energy consumption required to produce the sectoral outputs. Then the primary energies of various kinds needed are calculated using the Input-Output analysis. Energy footprints are calculated for the various primary energy requirements estimated and the final stage is to estimate the deficits of energy lands. Energy footprints and deficits can be compared for different scenarios in this manner. An Adaptive Approach to Planning and Decision-Making, G. Lessard, Landscape and Urban Planning 40 (1998) 81-87 The conclusion of this article is a good place to begin. It says: “So why an adaptive approach? Clearly, there are critical uncertainties in our knowledge base which will provide a continuous supply of surprise events. Since we will never have perfect information, we will continually learn from the response of ecosystems to implementation of our decisions. Planning for and adapting to surprise will provide an actionary rather than a reactionary basis for more informed decisions.“ This article proved quite helpful in stimulating potentially useful thoughts on the Corps planning process. This review departs from the standard approach found here by presenting the abstract verbatim. The article, although geared toward adaptive management, lends itself well to the more traditional planning processes like the Corps through simple analogical thinking, which follows the abstract in the next paragraph. Article Abstract A formal process of adaptive management will be required to maximize the benefits of any option for land and natural resource management and to achieve long-term objectives through implementation of ecosystem management. The process itself is straightforward and simple: new information is identified, evaluated and a determination is made whether to

5-8



Section 5 Scenarios in Water and Other Resources Context

adjust strategy or goals. While relatively straightforward, applying the concept of adaptive management to complex management strategies requires answers to several critical questions. What new information should compel an adjustment to the management strategy? What threshold should trigger this adjustment? Who decides when and how to make adjustments? What are the definitions and thresholds of acceptable results? Adaptive ecosystem management depends on a continually evolving understanding of causeand-effect relationships in both biological and social systems. Planning for and adapting to surprise will provide an actionary rather than a reactionary basis for more informed decisions. This article stimulated some ideas that are captured here. To adapt the author’s ideas, the initial hypothesis of interest in the context of the Corps planning process is the performance of a plan using the traditional evaluation process that compares the without- and withproject scenarios. New information becomes alternative scenarios based on a sensitivity analysis of key uncertainties in the process. This new information (an alternative without- or with-project condition) is evaluated to provide feedback on the formulated plan for the purpose of reaffirming, refining or reformulating the plan. With each scenario created by the planning team, planners can experiment with the performance of their project. In the process they can learn effectively about the potential performance of their project under scenarios other than the one most anticipated. Each scenario provides new information about the plan. Planners can use that information to decide whether to modify it in some way to improve its performance. Reformulation could effectively become a new planning step. It is not warranted in all or perhaps even most cases. But reformulation may be an effective strategy when: (1) uncertainties are large or numerous; (2) plans are irreversible or (3) there is controversy and disagreement. This step would include examination of uncertainties through the use of alternative without- or with-condition scenarios, as the case may warrant. Unacceptable project performance2 under reasonable alternative scenarios would result in a “back to the drawing board“ effort to reformulate the plan specifically to improve its performance under the alternative scenarios as well as the most likely scenario. Environmental Management Scenarios: Ecological Implications, J.C. Ogden, J.A. Browder, J.H. Gentile, L.H. Gunderson, R. Fennema, J. Wang, Urban Ecosystems 3 (1999) pp. 279-303 The authors assert that prevailing scientific consensus is that the current spatial extent and patterns of ecology and hydrology do not support a sustainable Everglades or South Florida ecosystem. As a part of the U.S. Man and the Biosphere (US MAB) Human-Dominated Systems Directorate (HDS) project, five plausible, regional-scale environmental management scenarios were developed to illustrate the potential for recovery of the physical defining characteristics of the South Florida system. In essence the authors describe the management of a range (five scenarios) of possible land additions to core and buffer areas intended to meet spatial-scale requirements and to achieve different degrees of hydrological improvement 2



Examples of unacceptable performance might be negative net NED benefits, unacceptable ecosystem losses or significant stakeholder opposition.

5-9

Section 5 Scenarios in Water and Other Resources Context

to examine their potential for ecological sustainability. Their evaluation was based on specific hydrologic characteristics for each scenario. The measure of ecological sustainability is the degree to which a scenario recovers the defining ecological characteristics of the regional landscape mosaic. A set of eight “risk hypotheses“ was used to show the relationship between the human-caused alterations in the defining physical characteristics and the resulting losses of ecological sustainability in these wetlands. The hypotheses identify the physical parameters of interest and enable each scenario to be evaluated on the basis of its capacity to recover the predrainage conditions for each parameter. The authors’ assessment of the five scenarios suggests they would all improve the problems addressed by the eight hypotheses. A transferable point for the Corps planning process would be to explicitly identify the hypotheses that underlie a formulated plan. These hypotheses would serve as a criteria for plan evaluation and comparison, albeit more complex criteria than are normally encountered. Stating hypotheses would be a trivial exercise in busy work for some studies. For example, to develop a hypothesis for how a floodwall would reduce flood damages is silly, as would be a hypothesis to suggest that a larger lock can pass more cargo. But the Everglades project is also a Corps activity and it is evident from this article and much other literature that hypotheses can be valuable management and decision aids for complex or large projects with significant complexity and uncertainty. The art then is in ascertaining when the planning process might best be aided by the definition of specific hypotheses that capture and embody the essence of the critical uncertainties in a planning investigation. Component Ecological Footprint: Developing Sustainable Scenarios, J. Barrett, Impact Assessment and Project Appraisal, 1 June 2001, vol. 19, no. 2, pp. 107-118 (12) The focus of this article is more on the concept of a “component ecological footprint“ than it is on scenario planning, although the footprint analysis is proposed for use as a regional planning tool. The author calculated the ecological footprint of waste, transport, energy, water and bio-resources for Guernsey (Channel Islands) using an integrated resource accounting framework, an apparent innovation of this approach. By altering the inputs to the model one is able to identify a footprint that is considered sustainable. The inputs leading to this desired result then provide guidance for regional planners. The ecological footprint was used to offer new insights into regional sustainability and what a sustainable society might look like. This research is more valuable for its ecological footprint, but it does demonstrate that scenarios of idealized benchmark futures can be imagined or created and plans can be developed to achieve these scenarios.

5-10



Section 5 Scenarios in Water and Other Resources Context

WARSYP: A Robust Modeling Approach For Water Resources System Planning Under Uncertainty, L.F. Escudero, Annals of Operations Research, 2000, vol. 95, no. 1/4, pp. 313339 (27) This article was of interest because it explicitly addressed decision making under uncertainty. The main elements of this problem were water resource sources (surface and groundwater), water demands (hydropower generation, irrigation, industrial, domestic, recreation and ecology) and infrastructure (reservoirs and distribution systems). A multi-period optimization model (WARSYP) was used to address the uncertainty in these main elements. Instead of using a mathematical programming model to capture the uncertainty the authors used scenarios to capture the uncertainty caused by random parameters in the model. The basic problem was to balance water resource supplies and demands over a multiperiod planning horizon. A multistage scenario tree was used along with “full recourse“ techniques to solve the model. The scenario model is described in considerable detail and it has value in its thoroughness. The paper is a good demonstration of the feasibility of addressing uncertainty in a scenario tree model. The paper is not without application to the Corps but its true value would be limited to problems of a water balance. It is not a generally applicable approach. Development of an Environmental Flows Decision Support System, W.J. Young; D.C.L. Lam; V. Ressel; I.W. Wong, Environmental Modeling and Software with Environment Data News, March 2000, vol. 15, no. 3, pp. 257-265 (9) This article provides a good example of a reasonably common use of the scenario terminology. The authors examine the desirability of different flow management scenarios. This is much more closely aligned to the Corps planning process than other uses of the terminology. Scenarios as used in this article are like alternative plans in the Corps jargon. The Murray–Darling Basin in Australia is severely environmentally degraded as a result of a range of anthropogenic changes. Withdrawal of irrigation water is the principle stressor. The resulting environmental problems include eutrophication of rivers and storages, elevated salinity levels, widespread blooms of toxic blue–green algae, decline of native fish and bird populations, and reduction of area of riverine wetlands. To facilitate the on-going trade-off process between competing water uses, the authors are developing a DSS that will enable explicit prediction of the likely response of key features of the riverine environment to proposed flow management scenarios. The DSS, which is the true focus of this article, will integrate a range of simple qualitative and quantitative models of riverine ecology that are being developed.



5-11

Section 5 Scenarios in Water and Other Resources Context

Water Resources Implications of Global Warming: A U.S. Regional Perspective, D.P. Lettenmaier; A.W. Wood; R.N. Palmer; E.F. Wood; E.Z. Stakhiv, Climatic Change, November 1999, vol. 43, no. 3, pp. 537-579 (43) The original formulation of scenario planning created alternative scenarios and analyzed the performance of strategies or plans across all the scenarios. This study reversed that sequence and examined the implications of global warming for the performance of six existing U.S. water resource systems. The six case study sites represent a range of geographic and hydrologic, as well as institutional and social settings. The studies essentially examined the sensitivity of six water resources systems to changes in precipitation, temperature and solar radiation. Thus alternative scenarios were “run“ against a variety of existing projects, not for the purpose of choosing the best project but to examine the robustness with which the projects would perform. A standard experimental design, consisting of specific base case and hypothetical altered climate simulations and similar evaluation measures for all sites, was used to maintain consistency between the six parts of the study. A sequence of models was used to infer the water resources effects of each of the climate change scenarios.

The climate change scenarios used in this study are based on results from transient climate change experiments performed with coupled oceanatmosphere GCMs for the 1995 IPCC assessment. The effects of climate change on system performance varied from system to system, from climate change model to climate change model and for each system operating objective, such as hydropower production, municipal and industrial supply, flood control, recreation, navigation and instream flow protection. Where possible, the effects of climate change were compared with the effects of non-climate related changes that could plausibly take place over the same period as the climate changes measured in these assessments (1990–2050). The studies showed, among other things, that these non-climate changes, such as demand growth and operational changes, had effects as large or larger than climate change. This study represents a useful data point in the investigation of climate change impacts on water resource infrastructure and systems. It did not contribute appreciably to scenario planning. Although it did use a variation of the original scenario-planning concept in its research design, this was more incidental to the study purpose. Water Resources Planning Principles and Evaluation Criteria for Climate Change: Summary and Conclusions, K.D. Frederick; D.C. Major; E.Z. Stakhiv, Climatic Change, September 1997, vol. 37, no. 1, pp. 291-313 (23) The prospect of anthropogenically-induced climate change presents water planners with a variety of challenges. Drawing on work presented in this volume, these challenges are summarized and conceptual issues surrounding strategies for adapting water planning and project evaluation practices to this prospect are examined. The six-step planning process detailed in the P&G is described; its ability to incorporate consideration of, and responses to,

5-12



Section 5 Scenarios in Water and Other Resources Context

possible climate impacts is assessed. The methods of sensitivity analysis, scenario planning and decision analysis that are encouraged by the P&G are found to be generally appropriate for planning and project evaluation under the prospect of climate change. However, some important planning and evaluation criteria require review and possible adaptation. The IPCC impact assessment procedures are found to be particularly useful as a framework for climate change impact and sensitivity analyses and would fulfill the requirements for future environmental impact statements. The ideas and principles are compatible with those found in the P&G. The water resources guidelines in the P&G deal explicitly with the specific comparison, appraisal and selection of project alternatives based on normative decision rules associated with benefit cost analysis and maximizing national welfare. These basic rules and normative decision criteria for evaluating alternative adaptation measures were validated to a large degree by the IPCC Working Group III report (Bruce, et al. 1996) on economic and social dimensions of climate change. Neither IPCC guidelines nor general environmental impact procedures possess comparable prescriptive decision criteria. The paper concludes with guidance to planners as to: (1) climate-related factors that are of concern and should be monitored; (2) conditions under which climate change should receive particular attention; and (3) adaptation opportunities.



5-13

Section 5 Scenarios in Water and Other Resources Context

5-14



Section 6 Expert Opinion, Subjective Probability and Sensitivity Analysis in Related Literatures This section provides a brief discussion of a limited amount of literature on two topics of interest to the Corps for the current research topic. These are expert opinion and subjective probability. Both of these topics appear in the scenario planning literature. Each has spawned its own literature. It is beyond the scope of this task order to consider these bodies of work at length but it is important to consider them, however briefly. This section considers some early lessons in the Aerospace and Intelligence fields. These were chosen because as government agencies involved in high stakes outcomes in the presence of great uncertainties they seemed to have something in common with some of the Corps own decision making situations. Indeed there seem to be some valuable lessons to be learned from these two fields. The section next turns to the use of subjective probability in PRA in government applied policy or planning contexts. The section concludes with the briefest consideration of the use of subjective probabilities in policy making. The main points there being that probabilistic forecasting is better than deterministic forecasting when there are substantial uncertainties and decision makers need to be aware of the significance of the uncertainties behind the deterministic estimates that are often treated as certain.

6.1 Aerospace Sector The aerospace sector has an extensive and now very public history of using expert opinion to assess safety. The history of this agency contains salient lessons for the Corps or any public agency that is required to make significant decisions under uncertainty. The basic lesson is that good techniques in estimating the probabilities of unobserved events is important and deserves careful attention from the agency’s highest management down throughout the agency. The National Aeronautics and Space Administration (NASA) managers needed to assess the risks associated with rare or unobserved catastrophic events to protect its astronauts and to respond to politicians’ concerns. Estimating the likelihoods of events that could occur but that have never been observed via traditional statistical methods was clearly not possible. Problems associated with estimating such likelihoods were dramatically brought out on January 28, 1986, with the tragic loss of the Challenger space shuttle and its crew. A review of a NASA sponsored estimate of shuttle failure modes and failure probabilities (Colglazier and Weatherwax, 1986) indicated an estimate of the solid rocket booster (SRB) failure probability per launch, based on subjective probabilities and operating experience, was roughly 1 in 35. This estimate was rejected by the NASA management, which relied on its own “engineering judgment,“ and used a figure of 1 in 100,000. This, in retrospect, clearly indicates a significant danger in ignoring good technique in making decisions under uncertainty. An excerpt from the Colglazier and Weatherwax article explained: ”We estimated in 1983 that the probability of a SRB failure destroying the shuttle was roughly 1 in 35 based on prior experience with this technology. . .



6-1

Section 6 Expert Opinion, Subjective Probability and Sensitivity Analysis in Related Literature

Our estimates of SRB failure were based on a Bayesian analysis utilizing the prior experience of 32 confirmed failures from 1902 launches of various solid rocket motors. We also found that failure probabilities for other accident modes were likely to have been underestimated by as much as a factor of 1000. . . NASA had decided to rely upon its engineering judgment and to use 1 in 100,000 as the SRB failure probability estimate for nuclear risk assessments. We have recently reviewed the critiques and stand by our original conclusions. . . We believe that in formulating space policy, as well as in assessing the risk of carrying RTGs on the shuttle, the prudent approach is to rely upon conservative failure estimates based upon prior experience and probabilistic analysis.” Formal risk assessment is a useful decision support tool for dealing with uncertain situations. NASA developed an interest in the risk assessment methodology after the fire on Apollo flight AS-204 on January 27, 1967, killed three astronauts. Before the Apollo fire NASA relied on its contractors to apply “good engineering practices“ to provide quality assurance and quality control. This was an agency that needed something that would both serve their decision making needs and stand up to the withering scrutiny of politics and the public better than good engineering practice did. The Space Shuttle Task Group, formed April 5, 1969, developed “suggested criteria“ for evaluating the safety policy of the shuttle program. The probability of mission completion was to be at least 95 percent and the probability of death or injury per mission was not to exceed 1 percent. These numerical safety goals were not adopted in the subsequent shuttle program (Wiggins, 1985). This perhaps marked another unfortunate instance of ignoring good scientific technique to quantify key uncertainties. Following a lead from the military, NASA adopted what they called risk assessment matrix tables to “quantify and prioritize“ risks. An example follows. Risks are presumed here to have a consequence and a likelihood. Table 6-1A shows the consequences considered by NASA. Table 6-1B adds a frequency of occurrence dimension to the consequence to produce a matrix in which the frequency-consequence pairs receive a numerical index. Higher numbers are lesser risks. Table 6-1C “judges“ the risks. The published reason for using the index was that the low numerical assessments of accident probability produced by quantitative risk estimates do not guarantee safety. A report describing the NASA safety program described the problem like this: “. . . the problem with quantifying risk assessment is that when managers are given numbers, the numbers are treated as absolute judgments, regardless of warnings against doing so. These numbers are

TABLE 6-1A NASA RISK ASSESSMENT MATRIX Description Catastrophic Critical Marginal Negligible

6-2

Category I II III IV

Hazard Severity Categories Mishap Definition Death or system loss Severe injury, severe occupational illness, or major system damage Minor injury, minor occupational illness, or minor system damage Less than minor injury, occupational illness or system damage



Section 6 Expert Opinion, Subjective Probability and Sensitivity Analysis in Related Literature

then taken as fact, instead of what they really are: subjective evaluations of hazard level and probability“ (Wiggins, 1985).

TABLE 6-1B NASA RISK ASSESSMENT MATRIX Hazard Risk Management Matrix Hazard Categories Frequency of I II III Occurrence Catastrophic Critical Marginal (A) Frequent 1 3 7 (B) Probable 2 5 9 (C) Occasional 4 6 11 (D) Remote 8 10 14 (E) Improbable 12 15 17

IV Negligible 13 16 18 19 20

TABLE 6-1C NASA RISK ASSESSMENT MATRIX Hazard Risk Index 1-5 6-9 10-17 18-20

Suggested Criteria Unacceptable Undesirable (project management decision required) Acceptable with review by project management Acceptable without review

This published explanation seems at odds with the practice according to Colglazier and Weatherwax who suggest NASA managers did not treat quantitative risk assessments as absolute numbers, instead, they ignored them and relied on their own judgment. In hindsight this self-serving decision met with tragedy. There are persistent rumors in the aerospace world that the primary motive for abandoning quantitative risk assessment was not distrust in overoptimistic reliability assessments so much as the fact that the estimates of catastrophic failure probabilities were so high they would have threatened the political viability of the entire space program. For example, a risk assessment on the likelihood of a successful manned moon landing done by GE estimated the chance of success was “less than 5 percent.“ When the NASA administrator was presented with the results, he “felt that the numbers could do irreparable harm and disbanded the effort“ (Bell and Esch, 1989).

6.2 Military Intelligence There is insufficient evidence in the available literature to make any truly informed commentary on the way the intelligence community deals with uncertainty. However, the Defense Intelligence Agency/Directorate of Estimates (DIA/DE), did make the results of a report available in 1980 (Morris and D’Amore, 1980). Morris and D’Amore suggest an intelligence service actively concerned with problems of dealing with uncertainty at a high level of mathematical sophistication.



6-3

Section 6 Expert Opinion, Subjective Probability and Sensitivity Analysis in Related Literature

A major focus of this work was predicting future force levels for the USSR. The failure to have predicted the fall of the Shah of Iran also appears as a motive for this report. The strategic planning of the intelligence community depends on the intentions and capabilities of an enemy. This differs some from the kinds of strategic planning done by the Corps. However, the highly uncertain motives and intentions of an enemy have easy corollaries in highly uncertain motives and corollaries of the markets and global politics that can affect navigation projects as well as the highly uncertain structure and function of ecosystems.

Lessons From Aerospace Experience

„ Use good science and good technique to estimate the probability of non-observed events.

„ Quantitative estimates of likelihoods can be misused, especially if the limitations on their use are not understood and appreciated.

„ Qualitative estimates of risks and their likelihoods are more useful than no estimates.

„ Self-serving estimates of likelihoods The data upon which intelligence assessments are the worst of all options. were based were often, if not usually, of dubious reliability. The so-called expert sources of these data included the testimony of defectors, reports from informants and interpretation of high-altitude photographs. A significant issue for the analysts in these situations was how to convey estimator’s uncertainty to the “consumers“ of intelligence reports so they could take this into account in making decisions. This problem remains critically important in the Corps planning process. How can planners convey their degree of confidence in the uncertain forecasts and judgments they must rely upon in the planning process? How can the uncertainty in a without- or with-project condition best be portrayed? How can the uncertainty in plan effects derived from the comparison of uncertain withoutand with-project conditions be conveyed? There may be a lesson to be learned from the intelligence community. Intelligence estimates in the 1960s contained phrases like “it is likely that,“ “it is unlikely that,“ and “it is probable that.“ The Directorate of Estimates introduced more precise forms for expressing uncertainty in 1976. For example, terms like “there is a 60 percent probability that,“ replaced the text phrases of the sixties. These estimates were sometimes accompanied by a colored sheet of paper on which was printed: Numeric forms are used to convey to the reader this degree of probability more precisely than is possible in the traditional verbal form. Our confidence in the supporting evidence is taken into account in making these quantifications. . . . All efforts at quantifying estimates are highly subjective, however and should be treated with reserve.“ (Morris and D’Amore, 1980, pp. 2-3) Qualitative expressions were still given alongside these numeric estimates. Some years later, the practice of using a “high,“ “low“ and “best“ estimate was introduced. The high and low values were chosen such that there would be a 75 percent probability the true value falls between the high and low values. This practice was proposed for use in the interval estimation cost estimating approach (Yoe, 2000). Intelligence analysts were not satisfied with this system for two principal reasons. First, the numbers were indeed “highly subjective,“

6-4



Section 6 Expert Opinion, Subjective Probability and Sensitivity Analysis in Related Literature

and there were no clearly defined method for obtaining them. Second, the numbers were generally ignored by the intelligence consumers. Consumers tend to take the best estimates (or sometimes the highest values) as if it were certain and disregard the uncertainty attached to them. The Directorate of Estimates’ response to these weaknesses in their then existing methods was to sponsor research for improving the quality and usefulness of numerical assessments of uncertainty. The resulting program was, at the time, considered better than any other method described in the public literature in both scale and sophistication. The communication and evaluation of uncertainty assessments, as envisioned in this program, are summarized as follows for their potential interest to the Corps planning process. Communicating uncertainty to “consumers“ so they will make proper use of it, was and remains a formidable challenge. Several systems were implemented and subsequently discarded. An early system, combining reliability and accuracy ratings, is shown in Table 6-2. “Information“ would be assigned a rating such as C2 or A5. The problem with this technique was that intelligence officers placed too much emphasis on the assessed accuracy of reports and not enough on the reliability of the source. In addition, there was a wide disparity in interpreting the meanings of the qualitative ratings (Samet, 1975).

TABLE 6-2 RELIABILITY AND ACCURACY RATINGS Source Reliability A: Completely reliable B: Usually reliable C: Fairly reliable D: Not usually reliable E: Unreliable F: Reliability cannot be judged

Information Accuracy 1: Confirmed 2: Probably true 3: Possibly true 4: Doubtfully true 5: Improbable 6: Accuracy cannot be judged

The Directorate of Estimates also made use of Kent charts. These charts (not all of them agree on the assignments), provide a quantitative interpretation of natural language expressions of uncertainty. Kent charts define the terms most frequently used to describe the range of likelihood in the key judgment of intelligence, as shown in Table 6-3 (Morris and D’Amore, 1980, p. 5-21). The Kent charts were abandoned in favor of direct numeric estimations of probability. Most of these numeric estimates were provided by experts using subjective probability estimates. There remains a substantial issue with the validity of numerical or calibration of these subjective probabilities. A subjective probability expert is said to be “well calibrated“ if statements with an assessed probability of X percent turn out to be true X percent. In other words, if an expert says it rains 60 percent of the time and it does rain 60 percent of the time, the expert is well calibrated. If it rains 30 percent of the time he overestimates and if it rains 80 percent of the time he underestimates. Calibration of expert opinion has produced its own voluminous literature. The Directorate of Estimates proposed a three-pronged effort to improve calibration. It included: debiasing; use of proper scoring rules; and use of feedback and systematic evaluation. The psychometric literature identifies a number of biases that tend to impair



6-5

Section 6 Expert Opinion, Subjective Probability and Sensitivity Analysis in Related Literature

TABLE 6-3 A KENT CHART FOR ESTIMATING TERMS AND DEGREES OF PROBABILITY Order of Likelihood Near Certainty Probable

Even Chance Improbable

Near Impossibility

Synonyms Virtually (almost) certain, we are convinced, highly probable, highly likely Likely We believe We estimate Chances are good It is probable that Chances are slightly better than even Chances are about even Chances are slightly less than even Probably not Unlikely We believe . . . not Almost impossible Only a slight chance Highly doubtful

Chances in 10 9

Percent

8 7 6

99 90 60

5 4

40

3 2

10

1

1

experts’ opinions. And procedures for eliciting and calibrating subjective probability estimates have been developed (Morgan and Henrion, 1988). Lessons From Intelligence These procedures are all quite standard and are well described in the literature. Many universities and some consultants offer decision support centers that rely on computersupported interactive elicitation processes that incorporate one or more of techniques described in the literature. Debiasing efforts are directed toward the process of eliciting subjective probabilities. The intelligence community has made use of scoring rules for their probability assessors. A scoring rule scores or rates a probability assessor on the basis of whether a “predicted“ outcome is later observed. The most intuitively appealing scoring rules are not truly useful. The “hit or miss“ rule simply counts the number of projections that have proven true and divides this by the total number of projections. This rule leads to evaluations like “70 percent of the projections from estimator A have come out.“ A rule like this encourages estimators to “hedge,“ that is, to give projections that are more cautious than they really think appropriate. A

6-6

„ The intelligence community was/is confronted with uncertainty on a large scale.

„ They have been forced to confront virtually all the problems known from the literature.

„ The intelligence community literature on this topic is understandably not very open.

„ The response of the intelligence community has been to try to improve the numerical estimations of uncertainty and to support the consumers in making the best possible use of these numerical assessments.

„ It would be very useful to have access to more recent reports on the success of such initiatives.



Section 6 Expert Opinion, Subjective Probability and Sensitivity Analysis in Related Literature

second flawed but appealing rule is the “direct rule.“ It rewards a probability assessment like “A will happen with probability 60 percent“ by giving the score 60 if A happens and the score 40 otherwise. This rule encourages overconfident assessments. Scoring rules are difficult to apply in intelligence estimates because many estimates concern events 10 or 20 years into the future, much as some Corps forecasts might. The last initiative described by Morris and D’Amore for improving the quality of probability assessments concerned an elaborate computerized data bank of intelligence estimates called the “institutional memory“ (IM). The purposes of IM were: to support the elicitation process; to provide feedback on past performance; and to inform the consumer of the quality of the Directorate of Estimates’ estimates. In 1980 the IM may have been the most advanced system of its kind for handling uncertainty. It is not possible to know without a better understanding of the techniques in general usage at that time. It is not likely that such sophisticated analysis would well suit the Corps planning process, however.

6.3 Probabilistic Risk Analysis Probabilistic risk analysis is a science-based decision support framework that first introduced subjective probabilities on a large scale. Many of the advances in expert elicitation, especially in the area of probability estimation have owed some debt to the practice of risk analysis. The notion of probabilistic risk assessment is not new. Its practice as a formal decision making paradigm is a rather recent development. More or less explicit risk assessment can be found in the writings of Arnobius the Elder in the fourth century (Covello and Mumpower, 1985). In the United States radiation biology was the birthplace of modern PRA. The Atomic Energy Commission, now the Nuclear Regulatory Commission (NRC), applied risk analysis concepts to the assessment of the “maximum credible accident.“ The best known example of such a study is WASH-740 released in 1957. It focused on three scenarios of radioactive releases from a 200 Megawatt-electric power nuclear power plant operating 30 miles from a large population center (Cooke, 1992). The report essentially concluded that the probability of such an event cannot be known. As larger reactors were being proposed, the desire to quantify and evaluate the associated risks led to the introduction of the PRA. The first full-scale PRA was undertaken in the Reactor Safety Study (RSS) called the Reactor Safety Study: An Assessment of Accident Risks in U.S. Commercial Nuclear Power Plants, NRC, NUREG 75/014, Washington, D.C., October 1975. With respect to the use of subjective probabilities in risk assessment a review of WASH-1400 said in part: The RSS had to use subjective probabilities in many places. Without these, the RSS could draw no quantitative conclusions regarding failure probabilities at all. The question is raised whether, since subjective probabilities are just someone’s opinion, this has a substantial impact on the validity of the RSS conclusions.



6-7

Section 6 Expert Opinion, Subjective Probability and Sensitivity Analysis in Related Literature

It is our view that the use of subjective probabilities is necessary and appropriate and provides a reasonable input to the RSS probability calculations. But their use must be clearly identified and their limits of validity must be defined (Lewis et al., 1979, p. 8). The NRC distanced itself from the results of the RSS in January 1979 when it said: In particular, in light of the Review Group conclusions on accident probabilities, the NRC does not regard as reliable the RSSs numerical estimate of the overall risk of reactor accident.“ (U.S. NRC, 1979) The future of PRA improved significantly after this initial foray into the field as the Food and Drug Administration, the U.S. Department of Agriculture, the Occupational Safety and Hazard Administration (OSHA) and the U.S. Environmental Protection Agency began to make regular and continuous use of quantitative and qualitative risk assessment techniques to support their decision-making processes. During the Carter Administration the so-called “Benzene case“ made it to the Supreme Court. Justice Stevens, arguing for the majority, said risk assessment is feasible and OSHA must do one before taking rule-making action to reduce or eliminate the risk associated with benzene. Shortly after, the “Cotton-dust“ case gave the Supreme Court the opportunity to reaffirm the Stevens decision. As a common practice among U.S. Federal Government agencies that has been legitimized by the Supreme Court of the United States, probabilistic risk assessment is a well-established tool for public decision making. Returning to the matter of subjective probabilities, the methodology of the Zion Probabilistic Safety Study has been well documented in the Journal of Risk Analysis (Kaplan and Garrick, 1981). The first issue of this journal defines “probability“ as follows: “. . . ‘probability’ as we shall use it is a numerical measure of a state of knowledge, a degree of belief, a state of confidence,“ (Kaplan and Garrick, 1981, p. 17). Expert elicitation of subjective probabilities and less structured approaches have since been used in several large studies. These include, for example, the risk study of the fast breeder reactor at Kalkar, Germany (Hofer, Javeri and Loffler; 1985); studies of seismic risk by Okrent (1975) and Bernreuter, et al., (1984); a study of fire hazards in nuclear power plants (Sui and Apostolakis, 1985); and numerous food safety risk assessments including but not limited to the more than 100 examples that are found at the Food Safety Risk Analysis Clearinghouse (http://www.foodriskclearinghouse.umd.edu/risk_assessments.cfm). Two sources that are most useful to anyone who wants to access the early literature on this topic include the Handbook of Human Reliability (Swain and Guttmann, 1983) and (Humphreys, 1988). Together they give a good review of the literature and methods. Volume 93 (1986) of Nuclear Engineering and Design is entirely devoted to the role of data and judgment in risk analysis. Four practical issues emerged early in the subjective probability literature and they remain germane to this day. They include the spread of expert opinion, the dependency between experts, the reproducibility of the results and finally the calibration of the results. Cooke’s

6-8



Section 6 Expert Opinion, Subjective Probability and Sensitivity Analysis in Related Literature

(1992) excellent review of the topic, which provides the structure for much of this section, is used to introduce these issues.

6.3.1 Subjective Data: Spread A nuclear reactor core meltdown is a good example of a quantity that can not be computed without substantial use of subjective probabilities. The RSS estimated the value to be 4.7 x 10-7. The Okrent study (1975) estimated the same value to be probability as 8 x 10-5. Another study (Lee, Okrent and Apostolakis, 1979) estimated a probability of 1.77 x 10-4 per reactor year for reactors with no design errors and 2.32 x 10-3 per reactor year for systems having a “maximal number of design errors and reduced original safety factors.“ These subjective probability estimates span four orders of magnitude, illustrating a common concern about subjective probability estimates. Expert probability assessments in risk analysis typically show extremely wide spreads (Cooke, 1992).

6.3.2 Subjective Data: Dependence A study of the RSS (Cooke, 1992) shows that the estimates experts offered for probabilities of various events were not independent. Experts pessimistic with respect to one component of a reactor tended to be pessimistic about other components as well. For example, an expert whose estimate for a given component lies above the median estimate often tends to be above the median for other components as well. An expert is “rank independent“ if his responses show no tendency to cluster either toward optimism nor pessimism. Other studies (Shooman and Sinkar, 1977) have confirmed the high degree of clustering in the probability estimates of experts.

6.3.3 Subjective Data: Reproducibility There have been some studies that attempt to gage the extent to which risk assessment results are reproducible. And although they do not focus solely on subjective probability estimates, they are primary quantities of interest in explaining the different results experts obtained in these studies. These studies are called bench mark studies. The Joint Research Centre at Ispra, Italy (Amendola, 1986) undertook such a study on the new Paluel Auxiliary feedwater system. Ten teams from different European countries were formed to independently estimate the probability that this system would not fulfill its design requirements. The study design was quite unique for this field and is worth some description. In stage one, the teams carried out their first probabilistic analysis. This was the “blind evaluation.“ Without discussing the results beforehand, the teams were then brought together to compare their analyses qualitatively. Stage two followed this comparison. In stage two the teams performed an intermediate “fault tree analysis.“ The teams were unable to agree on a common fault tree model and the spread in results after this stage was rather large. In stage three the researchers separated the effects of different fault tree models from the effects of different failure data by providing the teams with a common fault tree. The goal of stage four was to determine whether different methods of calculation played a significant role in the results observed. The results show a broad dispersion of outcomes. Brune, et al. (1983) conducted another bench mark study concerning human reliability. Human error is implicated in anywhere from 30 percent to 80 percent of serious accidents with sophisticated technical systems (Levine and Rasmussen, 1984). A study was designed to determine to what extent techniques for quantifying human error probabilities in risk analysis developed at the Sandia National Laboratories would lead to similar results when



6-9

Section 6 Expert Opinion, Subjective Probability and Sensitivity Analysis in Related Literature

applied by different human reliability experts. Qualified persons using the same techniques failed to obtain an answer within the prescribed confidence intervals in about 38 percent of the experts’ responses. The differences in the experts’ answers were not caused by different values for individual failure probabilities. These were given in the exercise. The differences were caused by the fact that the experts analyzed the human reliability problems differently. For example, they identified different tasks that might be performed incorrectly. The evidence suggests that reproducibility of risk assessment results is not easy to achieve. The relevance of this for the Corps planning process would seem to be that subjective processes are fundamentally prone to human error. And planning is always going to rely on some subjective processes.

6.3.4 Subjective Data: Calibration Having experts agree is important but it is less important than having experts who are accurate and whose assessments are good. This is the calibration issues, which is concerned with the extent to which the assessed probabilities agree with observed relative frequencies. It is always difficult and often not even possible to calibrate subjective probability estimates when the events whose probabilities are assessed are rare. Ordinary events are not often the subject of risk analysis. It is the rare events that most often and most controversially capture the risk assessors’ attentions. Because rare events can often be deconstructed to a series of component or ingredient events it is sometimes possible to calibrate assessments for ingredient events. The literature on the calibration of these rare event probabilities is sparse. There has been some effort to calibrate some of the subjective probability assessments in the RSS (Minarick and Kukielka, 1982; Cottrell and Minarick, 1984). These studies suggested two sorts of biases in the RSS estimates. First, there is a “location“ or “first moment“ bias. This caused the estimate to be too low in the RSS case, but it could as easily be caused by an overestimate. Second, there is a “scale“ or “overconfidence“ bias, causing the confidence bounds to be too narrow. To compound this entire issue there is little documented evidence to suggest how confident the experts were of their flawed estimates. Mosleh, Bier and Apostolakis (1988) propose the use of a degree of confidence indicated by “range factors.“ To obtain the range factor for a quantity you first find the ratio of the 95th and 5th percentiles of the distribution for this quantity and then take the square root of that quantity. Thus, if a range factor of 3.2 is reported the 95th percentile value is a factor of 10 larger than the 5th percentile value. Cooke (1992) applied this concept in a study and concluded the experts’ assessments reflect significant overconfidence. There is a great deal of “gray literature“ on the matter of subjective probability assessment. This field is dominated by a great deal of applied work and only a fraction of it ever finds its way into print. This reviewer has participated in the conduct of over a dozen such assessments all of which resulted in proprietary reports. A review of more than 3,500 abstracts on judgment and reasoning by Christensen-Szalanski and Beach (1984) says that poor performances with expert opinion were reported more than good performances. It is unclear whether that is because poor performances outnumber good performances or simply whether “bad news“ makes for better articles than good news.

6-10



Section 6 Expert Opinion, Subjective Probability and Sensitivity Analysis in Related Literature

The conclusions from this limited review of the literature related to the use of subjective probabilities in risk analysis shows that expert opinions in PRA have exhibited extreme spreads, have shown clustering and have led to results with low reproducibility and poor calibration. There have been significant gains and advances in methodological approaches to the use of expert opinion. A review of that literature is beyond the scope of this task. But, if the Corps relies on methodologies dependent on subjective probability estimates in particular, or expert opinion in general, it is best to make use of one or more of the more recent protocols and methodologies. The Internet is one of the best sources of information about the most recent advances in expert knowledge elicitation, which is a much broader and more modern field of inquiry. Many university courses make notes and unpublished papers available to their students and any other members of the public with an interest in this area.

6.4 Policy Analysis Continuing with Cooke’s useful structure, “policy analysis“ is a catchall for those things that do not fit into the areas discussed previously. It would seem that the Corps planning process as currently practiced would fall more comfortably into this category than into aerospace, intelligence or PRA. Morgan et al. (1984), Morgan and Henrion (1988); and Merkhofer and Keeney (1987) provide good examples of policy analysis that makes explicit use of subjective probability assessments. In general, policy tends to rely far more on the use of deterministic methods of expert opinion. This brief review highlights some of the differences in probabilistic and deterministic methods of using expert opinion. The Corps has tended more toward the deterministic methods than the probabilistic methods when using experts in its planning process, hence some review of the differences may be useful. Many planning uncertainties revolve around the planners’ ability to forecast the future (scenarios) or the future value of some variable. Planners must predict values, e.g., fleet composition, commodities, flood damages and flows that will eventually become known. Once known they may lend themselves very well to objective probabilistic analysis. Before they are known, however, subjective probabilistic assessment is often the best that can be done. And although the Corps is quite on the leading edge in the use of probabilistic methods for some of these quantities, in others there has been very little use of probabilistic assessment. The Corps early efforts to make commodity forecasts for specific harbors employed various mathematical models in economic forecasting, sometimes in conjunction with expert forecasts. At times the experts have had impressive credential and at other times the subjective judgments were being offered by Corps analysts with more planning experience than specific expertise in the field. Experts’ forecasts combined with model outputs have often been used to produce estimates of critical variables based on past performance and other factors. These methods are deterministic when they make no attempt to assess or communicate the uncertainty attending the estimate of this critical variable. Decision makers are given no information to help them gage the extent to which their decisions should or should not hedge for the uncertainty inherent in the planning studies upon which they base their decisions. These naively deterministic estimates of future values of critical variables are even more primitive than the qualitative attempts, like the Kent chart, shown in Table 6-3, to represent



6-11

Section 6 Expert Opinion, Subjective Probability and Sensitivity Analysis in Related Literature

uncertainty. In practice this often means that the decision maker treats the forecast values as certain. This can be problematic when others, especially opponents of a specific course of action, recognize the uncertainty in these forecasts. Granger (1980) said there was no practical way to evaluate a forecasting technique because some values are more easily forecast than others, so the technique is not independent of the nature of the task. For example, it is easy to predict how old someone will be in ten years. It is not as easy to predict the number of heads in ten tosses of a coin. How would one compare and evaluate the techniques used for such different tasks? Cooke (1992) suggests this is perhaps the single most important difference between probabilistic and deterministic methods, i.e., probabilistic forecasters can be evaluated independently of the “forecastability“ of the things they are forecasting, while deterministic forecasters cannot. Granger (1980) and Beaver (1981) were two early studies to suggest that the consensus of several experts is generally a better forecaster than any of the experts individually. Beaver illustrated this with an example of forecasting winners of football games. He reviewed the published predictions of the sports staff of the Chicago Daily News from 1966 to 1968. The study found the consensus prediction outperformed all the other predictors, for the entire period. Building on Beaver’s idea, the advice of several experts is to be preferred to the advice of any one expert. A potential problem with Beaver’s argument is that in the real world, forecasters are not always unbiased. Experience shows they are not “as likely to overestimate as to underestimate“ a value. Worse, it is possible to have all experts with identical biases if the selection pool itself is biased. Thus, a team of Corps experts, or a team of industry experts, or a team of environmental interest group experts may be biased. And in this case a team of biased experts may not perform as well as the single best expert. This suggests that a diverse group of bona fide experts may be desirable. When such biases are absent there are techniques that may work very well. A much more recent example of this principle in action was advanced by the Defense Advanced Research Projects Agency (DARPA) in a program, called the Futures Markets Applied to Prediction (FutureMAP). This program would have involved many investors betting small amounts of money that a particular event—a terrorist attack or assassination— would happen. A large number of independent and unbiased “experts“ would produce a better composite forecast than any one or two experts could have produced on their own. A story posted by CNN on Wednesday, July 30, 2003 (http://www.cnn.com/2003/ ALLPOLITICS/07/29/terror.market/) reported that the FutureMAP had part of the Total Information Awareness program. The DARPA acknowledged, under considerable political pressure, that the program faced “a number of major technical challenges and uncertainties. Chief among these are: Can the market survive and will people continue to participate when the U.S. authorities use it to prevent terrorist attacks? Can futures markets be manipulated by adversaries?“ So the debate over who is an expert and how many experts does it take to make a good consensus forecast continues to the present moment.

6-12



Section 6 Expert Opinion, Subjective Probability and Sensitivity Analysis in Related Literature

6.5 Sensitivity Analysis Scenario analysis is not sensitivity analysis. Sensitivity analysis is changing variables, often one at a time, all other things equal. There is no one at a time variation in scenario analysis and there is no attempt to identify an average case. Sensitivity analysis often runs off the average case to see how it is affected by a change in a variable here or there. Nonetheless, sensitivity analysis is a common way of addressing uncertainty. This section briefly reviews a few articles dealing with sensitivity analysis. ‘‘In a sensitivity analysis, one systematically and comprehensively tests to see how changes in the parameters of the model affect the model’s output,’’ (Starfield and Bleloch, 1991). The goal is to learn which of the model parameters exert significant influence on the output variables and which are inconsequential. In order to understand the decision support models in use, analysts need to know, for example, if small increments in any parameters produce unexpectedly large alterations in results. Most importantly, recommendations based upon a model without explicit sensitivity analysis lack foundation (Beres and Hawkins, 2001). Despite the growing interest and acknowledged importance of sensitivity analysis, there is ‘‘a dearth of information’’ (Henderson-Sellers and Henderson-Sellers, 1996) on how to do sensitivity analysis. ‘‘There is no single, universally accepted procedure for the sensitivity analysis of stochastic models’’ (McCarthy et al., 1995). Perhaps the most common approach to sensitivity analysis is to explore the effects of changing parameters, one at a time, on a target output variable (Henderson-Sellers and Henderson-Sellers, 1996; Swartzman and Kaluzny, 1987). This ceteris paribus procedure holds all variables constant at a mean or representative value while each parameter is allowed to change to determine its effect in isolation from the possible effects of other variables. In an interdependent system this approach frequently can result in serious errors. Daniel (1973) suggested that even when the sensitivity of all individual parameters is investigated, the one-at-a-time (OAAT) method cannot uncover potentially important interactions between two parameters. Experimental designs for sensitivity analysis of more than one parameter at a time are needed. Analysts are often interested in the effects of individual parameters, the 2-way and multi-way ‘‘interactions’’ of pairs and more of parameters (Swartzman and Kaluzny, 1987). Some such methods have been devised, but a review of that literature is beyond the scope of this review. One such design includes the complete factorial design, which consists of all possible combinations of selected high and low values for the parameters. Beres and Hawkins describe the Plackett–Burman Sensitivity Analysis (PBSA) design that provides an alternative that is both convenient and informative. They offer a rationale for using this approach based on the following points:

„ „ „ „ „ „



PBSA is not an OAAT method PBSA finds 2-way interactions PBSA is not restricted to any particular type of model PBSA is prescriptive, using pre-determined designs PBSA is efficient in terms of number of scenarios needed PBSA designs for up to 100 parameters are readily available

6-13

Section 6 Expert Opinion, Subjective Probability and Sensitivity Analysis in Related Literature

„ „ „ „

PBSA rankings are easy to compute PBSA works with categorical as well as numerical parameters PBSA does not require parameters to be considered over identical intervals PBSA is statistically sound

Having noted these limitations of sensitivity analysis, there is some potential for using a sensitivity analysis where the effects of different scenarios rather than different parameters are examined for their impact on planning outcomes of interest.

6-14



Section 7 Bibliography Ackoff, R.L. Creating the Corporate Future. New York: Wiley, 1981. Adya, M., et al. “An Application of Rule-Based Forecasting To A Situation Lacking Domain Knowledge,” International Journal of Forecasting, October 2000, vol. 16, no. 4, pp. 477484 (8). Al-Jayyousi, O.R. “Scenarios for Public-Private Partnerships in Water Management: a Case Study from Jordan,“ International Journal of Water Resources Development, June 2003, vol. 19, no. 2, pp. 185-201 (17) Amendola, A. “Systems Reliability Benchmark Exercise Parts I and II,“ EUR-10696, EN/I, 1986. Anderson, J., J. Clement and L.V. Crowder. 1999. “Pluralism in Sustainable Forestry and Rural Development-An Overview of Concepts, Approaches and Future Steps,“ in Food and Agriculture Organization, Pluralism and Sustainable Forestry and Rural Development. In: Proceedings of an International Workshop, December 1997, pp. 9± 12. Ataie-Ashtiani, B., et al. “Numerical Modeling of Two-Phase Flow in a Geocentrifuge,“ Environmental Modeling and Software, April 2003, vol. 18, no. 3, pp. 231-241 (11). Athanassopoulos, A.D., N. Lambroukos and L. Seiford. “Data Envelopment Scenario Analysis for Setting Targets to Electricity Generating Plants,“ European Journal of Operational Research, 6 June 1999, vol. 115, no. 3, pp. 413-428 (16). Bailenson, J.N., et al. A Bird’s Eye View: Biological Categorization and Reasoning Within and Across Cultures. Cognition, May 2002, vol. 84, no. 1, pp. 1-53 (53). Baker, S., D. Ponniah and S. Smith. “Techniques for the Analysis of Risks in Major Projects,“ Journal of the Operational Research Society, June 1998, vol. 49, no. 6, pp. 567-572 (6). Bardram, J. “Scenario-Based Design of Cooperative Systems,“ Group Decision and Negotiation, May 2000, vol. 9, no. 3, pp. 237-250 (14). Barnes, Jr., J.H. “Cognitive Biases and Their Impact On Strategic Planning,“ Strategic Management Journal, 1984, vol. 5, no. 2, pp. 129±137. Barrett, J. “Component Ecological Footprint: Developing Sustainable Scenarios,“ Impact Assessment and Project Appraisal, 1 June 2001, vol. 19, no. 2, pp. 107-118 (12). Baumert, L., S.W. Golomb and M. Hall. “Discovery of an Hadamard Matrix of Order 92,“ American Mathematical Society Bulletin 1962, no. 68, pp. 237–238. Baumont, G., et al. “Quantifying Human and Organizational Factors in Accident Management Using Decision Trees: the Horaam Method.” Reliability Engineering and System Safety, November 2000, vol. 70, no. 2, pp. 113-124 (12) Bazerman, M.H. and M.A. Neale. “Negotiating Rationally,“ the Free Press, 1992



7-1

Section 7 Bibliography

Beaver, W.M. Financial Reporting: An Accounting Revolution. Prentice Hall, Englewood Cliffs, NJ, 1981. Becker, H.S., “Scenarios: a Tool of Growing Importance to Policy Analysts in Government and Industry.” Technology Forecasting and Social Change, 1983, vol. 23, no. 2, pp. 95±120. Beliakov G. and J. Warren. “Fuzzy Logic For Decision Support in Chronic Care.“ Artificial Intelligence in Medicine, January 2001, vol. 21, no. 1, pp. 209-213 (5). Bell, T.E. and K. Esch. “The Space Shuttle: A Case of Subjective Engineering.“ IEEE Spectrum, pp. 42-46, June 1989. Belzer, R.B. “Getting Beyond ‘Grin and Bear It’ in the Practice of Risk Management“ Reliability Engineering and System Safety, May 2001, vol. 72, no. 2, pp. 137-148 (12). Bennett, R.L., et al. “Fabry Disease in Genetic Counseling Practice: Recommendations of the National Society of Genetic Counselors.” Journal of Genetic Counseling, April 2002, vol. 11, no. 2, pp. 121-146 (26). Beres, D.L. “A Methodological Study of Modeling for California Condors.“ Ph.D. Dissertation, University of Minnesota. Beres, D.L. and D.M. Hawkins. (2001), “Plackett-Burman technique for sensitivity analysis of many-parametered models.” Ecological Modeling, 141, 171-183. Beres, D.L., C. Clark, G. Swartzman and A.M. Starfield. “Truth in Modeling.“ Natural Resource Modeling 2000, vol. 14, no. 3, in Press. Bernreuter, D.L., J.B. Savy, R.W. Mensing and D.H. Chung. “Seismic Hazard Characterization of the Eastern United States: Methodology and Interim Results for Ten Sites,“ NUREG/CR3756, 1984. Bialasiewicz, L. “Another Europe: Remembering Habsburg Galicja.” Cultural Geographies, 1 January 2003, vol. 10, no. 1, pp. 21-44 (24). Bilkle O. and Knolker, U. “The Installation of a Psychiatric Out-Patient Drug and Addiction Service for Adolescents—First Results on Clinical and Institutional Problems,” European Psychiatry, 1998, vol. 13, no. 1004, pp. 229s-229s (1). Blair, A.R., et al. “Forecasting the Resurgence of the US Economy in 2001: An Expert Judgment Approach Socio-Economic Planning Sciences,“ June 2002, vol. 36, no. 2, pp. 77-91 (15). Blythe, M.J., and R. Young. “Scenario Analysis: a Tool for Making Better Decisions for the Future,” Evaluat. J. Australasia, 1994, vol. 6, no. 1, pp. 1±17. Bocco, G., and V.M. Toledo. “Integrating Peasant Knowledge and Geographic Information Systems: a Spatial Approach to Sustainable Agriculture,“ Indig. Know. Develop. Monitor 1997, vol. 5, no. 2, pp. 10-13.

7-2



Section 7 Bibliography

Bommer, J.J., S.G. Scott and S.K. Sarma. “Hazard-Consistent Earthquake Scenarios Soil Dynamics and Earthquake Engineering,“ June 2000, vol. 19, no. 4, pp. 219-231 (13). Bood R., and T. Postma. “Strategic Learning with Scenarios,” European Management Journal, December 1997, vol. 15, no. 6, pp. 633-647 (15). Borrini-Feyerabend, G. “Beyond Fences: Seeking Social Sustainability in Conservation, a Resource Book,“ 1997, vol. 2. Bower, B.T., and T.R. Kerry. “Characterizing and Analyzing Benefits From Integrated Coastal Management (ICM) - Market and Intervention Failures,“ Ocean and Coastal Management, 1 January 1998, vol. 38, no. 1, pp. 41-66 (26). Box, G.E.P. and N.R. Draper. “Empirical Model-Building and Response Surfaces,“ 1987, p. 669. Brauers, J. and M. Weber. 1988. “A New Method of Scenario Analysis for Strategic Planning,“ Journal of Forecasting, 1988, vol. 7, no. 1, pp. 31-47. Briand, L.C. and J. Wust. “Integrating Scenario-Based and Measurement-Based Software Product Assessment,“ Journal of Systems and Software, 15 October 2001, vol. 59, no. 1, pp. 3-22 (20). Brown, R.A., et al. “Integrated Assessment of Hadley Centre (Hadcm2) Climate Change Projections On Agricultural Productivity and Irrigation Water Supply in the Conterminous United States - I. Climate Change Scenarios and Impacts On Irrigation Water Supply Simulated With the Humus Model,“ Agricultural and Forest Meteorology, June 30 2003, Vol. 117, No. 1, Pp. 73-96 (24). Brown, S., Middleton D., Lightfoot G. “Performing the Past in Electronic Archives: Interdependencies in the Discursive and Non-Discursive Ordering of Institutional Rememberings.” Culture and Psychology, June 2001, vol. 7, no. 2, pp. 123-144 (22). Bruce, J.P., H. Lee and E.F. Haites (Eds). 1996. Climate Change 1995: Economic and Social Dimensions of Climate Change. Contribution of Working Group III to the Second Assessment of the Intergovernmental Panel on Climate Change. Cambridge University Press, United Kingdom. Brune, R., Weinstein, M. and Fitzwater, M., “Peer Review Study of the Draft Handbook for Human Reliability Analysis with Emphasis on Nuclear Power Plant Applications,“ NUREG/CR-1278, Human Performance Technologies, Inc., Thousand Oaks, Calif., 1983. Bunn, D.W., and Salo, A.A. “Forecasting With Scenarios,“ European J. Oper. Res. 1993, Vol. 68, No. 3, Pp. 291-303. Burgman M.A., et al. “A Method For Setting the Size of Plant Conservation Target Areas“ A Method For Setting the Size of Plant Conservation Target Areas. Conservation Biology, June 2001, Vol. 15, No. 3, Pp. 603-616 (14).



7-3

Section 7 Bibliography

Burgman, M.A., Ferson, S., and Akcakaya, H.R. “Risk Assessment in Conservation Biology,“ 1993, P. 314. Carroll J. M. “Becoming Social: Expanding Scenario-Based Approaches in Hci,“ Behaviour and Information Technology, 1 July 1996, Vol. 15, No. 4, Pp. 266-275 (10). Carroll, J.M. “Five Reasons for Scenario-Based Design,“ Interacting With Computers, September 2000, Vol. 13, No. 1, Pp. 43-60 (18). Chandler J. and Cockle P. (1982). Techniques of Scenario Planning. London: McGraw-Hill. Chandler, A.M. and Lam, N.T.K. “Scenario Predictions for Potential Near-Field and FarField Earthquakes Affecting Hong Kong“ Soil Dynamics and Earthquake Engineering, January 2002, Vol. 22, No. 1, Pp. 29-46 (18). Chao, P.T., B.F. Hobbs and B.N. Venkatesh. “Using Decision Analysis to Include Climate Change in Water Resources Decision Making,“ Climatic Change, September 1997, Vol. 37, No. 1, Pp. 177-202 (26). Chau, K.W. and F. Albermani. “Expert System Application on Preliminary Design of Water Retaining Structures,” Expert Systems With Applications, February 2002, Vol. 22, No. 2, Pp. 169-178 (10). Chen, F., H.-S Chu and C. Yeh. “Effects of Porosity Change of Gas Diffuser On Performance of Proton Exchange Membrane Fuel Cell,“ Journal of Power Sources, 15 September 2003, Vol. 123, No. 1, Pp. 1-9 (9). Choi, Y.S., S.H. Lee and B.W. Lee. “Expert Judgment For Nuclear Energy,” Annals of Nuclear Energy, May 2000, Vol. 27, No. 7, Pp. 575-588 (14). Christensen-Szalanski, J.J.J., and Beach, L.R., “The Citation Bias: Fad and Fashion in the Judgment and Decision Literature,“ American Psychologist, vol. 39, pp. 75-78, 1984. Cionco, R.M., and Ellefsen, R. “High Resolution Urban Morphology Data for Urban Wind Flow Modeling,“ Atmospheric Environment, January 1998, Vol. 32, No. 1, Pp. 7-17 (11). Clemen, R.T., and Winkler R.L. “Combining Probability Distributions from Experts in Risk Analysis“ Risk Analysis, April 1999, Vol. 19, No. 2, Pp. 187-203 (17). Clemons E.K. “Using Scenario Analysis to Manage the Strategic Risks of Reengineering,“ Long Range Planning, December 1995, Vol. 28, No. 6, Pp. 123-123 (1) Coen F., et al. “Nitrogen Removal Upgrade of a Wastewater Treatment Plant Within Existing Reactor Volumes: a Simulation Supported Scenario Analysis,“ Water Science and Technology, 1996, Vol. 34, No. 3, Pp. 339-346 (8). Coffey J.W. and Hoffman R.R. “Knowledge Modeling for the Preservation of Institutional Memory“ Journal of Knowledge Management, 9 July 2003, vol. 7, no. 3, pp. 38-52 (15).

7-4



Section 7 Bibliography

Cojazzi, G., et al. “Benchmark Exercise on Expert Judgment Techniques in Psa Level 2,“ Nuclear Engineering and Design, November 2001, Vol. 209, No. 1, Pp. 211-221 (11). Coles S. and J.Rowley. “Spreadsheet Modeling for Management Decision Making,“ Industrial Management and Data Systems, 1 July 1996, Vol. 96, No. 7, Pp. 17-23 (7). Colfer, C.J.P. “Who Counts Most in Sustainable Forest Management?“ Cifor Working Paper, 1995, No.7 Colglazier, E. W., and Weatherwax, R. K., “Failure Estimates for the Space Shuttle,“ Abstracts for Society for Risk Analysis, Annual Meeting 1986, Boston, Mass., p. 80, Nov. 9-12, 1986. Cooke, R. 1992. Experts in Uncertainty: Opinion and Subjective Probability in Science. New York: Oxford University Press. Cosier R. A. (1981a). “Dialectical inquiry in strategic planning: A case of premature acceptance?“ Academy of Management Review, 6: 643-648. Cosier R. A. (1981b). “Further thoughts on dialectical inquiry: A rejoiner to Mitroff and Mason.” Academy of Management Review, 6: 653-654. Cottrell W., and Minarick, C., “Precursors to Potential Severe Core Damage Accidents: 19801982, a Status Report,“ NUREG/CR-3591, 1984. Courtney, H. “Decision-Driven Scenarios for Assessing Four Levels of Uncertainty,“ Strategy and Leadership, 2003 Vol. 31, No. 1, Pp. 14 – 22 Covello, V. T., and Mumpower, J. “Risk Analysis and Risk Management: An Historical Perspective,“ Risk Analysis, vol. 5, no. 2, pp. 103-120, 1985. Crane, S.A. “Memory, Distortion, and History in the Museum.” History and Theory, December 1997, vol. 36, no. 4, pp. 44-63 (20) Croise, J., and Kaleris, V. “Estimation of Cleanup Time in Layered Soils by Vapor Extraction,“ Journal of Contaminant Hydrology, 15 February 1999, Vol. 36, No. 1, Pp. 105-129 (25) Croizet, J.-C., and Fiske, S.T. “Moderation of Priming By Goals: Feeling Entitled To Judge Increases Judged Usability of Evaluative Primes.” Journal of Experimental Social Psychology, March 2000, Vol. 36, No. 2, Pp. 155-181 (27) Daniel, C. “Applications of Statistics to Industrial Experimentation,“ 1976, P. 294. Daniel, C. “One-At-A-Time Plans,“ Journal of the American Statistical Association 1973, Vol. 68, Pp.353–360 Das, A., and Kandpal, T.C.“ Energy-Environment Implications of Cement Manufacturing in India: a Scenario Analysis,“ Fuel and Energy Abstracts, November 1997, Vol. 38, No. 6, Pp. 430-430 (1)



7-5

Section 7 Bibliography

Davis, J.E. “Accounts of False Memory Syndrome: Parents, “Retractors,“ and the Role of Institutions in Account Making.” Qualitative Sociology, 2000, vol. 23, no. 1, pp. 29-56 (28) De Wit, S., Augenbro,E G. “Analysis of Uncertainty in Building Design Evaluations and Its Implications.” Energy and Buildings, October 2002, Vol. 34, No. 9, Pp. 951-958 (8) Delic, J.I., et al. “The Utility of Pbpk in the Safety Assessment of Chloroform and Carbon Tetrachloride.” Regulatory Toxicology and Pharmacology, October 2000, Vol. 32, No. 2, Pp. 144-155 (12) Deshler, D. “Techniques for Generating Futures Perspectives,“ In: Brockett, Ralph G. (Ed.), Continuing Education in the Year 2000. New Directions for Continuing Education, 1987, No. 36, Pp. 79-82 Diaper, D. “Task Scenarios and Thought,“ Interacting With Computers, October 2002, Vol. 14, No. 5, Pp. 629-638 (10) Diaz, Iii J., and Hansz J.A. “The Use of Reference Points in Valuation Judgment.” Journal of Property Research, 1 June 2001, Vol. 18, No. 2, Pp. 141-148 (8) Doran C. M., et al. “General Practitioners’ Role in Preventive Medicine: Scenario Analysis Using Smoking As a Case Study,“ Addiction, 1 July 1998, Vol. 93, No. 7, Pp. 10131022. Dorp, J.R.V, and Kotz, S. “A Novel Extension of the Triangular Distribution and Its Parameter Estimation.” Journal of the Royal Statistical Society Series D (The Statistician), 2002, Vol. 51, No. 1, Pp. 63-79 (17) Dreybrodt, W., Gabrovsek, F., and Romanov, D. “Dam Sites in Soluble Rocks: a Model of Increasing Leakage By Dissolutional Widening of Fractures Beneath a Dam,“ Engineering Geology, October 2003, Vol. 70, No. 1, Pp. 17-35 (19) Driscoll, D.-M., and M. Hoffman W. “Gaining the ethical edge: procedures for delivering values-driven management.” Long Range Planning, April 1999, vol. 32, no. 2, pp. 179-189 (11) Ducey M.J. “Representing Uncertainty in Silvicultural Decisions: An Application of the Dempster-Shafer Theory of Evidence,.” Forest Ecology and Management, 15 September 2001, Vol. 150, No. 3, Pp. 199-211 (13) Ducot, C., and Lubben, G.J. “A Typology for Scenarios,“ Futures, 1980 Vol. 12, No.1, 51±57 Duval, A., Fontela, E., and Gabus, A. “Cross-Impact Analysis: a Handbook on Concepts and Applications,“ In: Baldwin, M.M. (Ed.), Portraits of Complexity: Applications of Systems Methodologies to Societal Problems, 1975, Pp. 202±222 Edmond G. “Legal Engineering: Contested Representations of Law, Science (And NonScience) and Society.” Social Studies of Science, June 2002, Vol. 32, No. 3, Pp. 371-412 (42). Sage Publications Inc.

7-6



Section 7 Bibliography

Elander J., Hardman D. “An Application of Judgment Analysis To Examination Marking in Psychology.” British Journal of Psychology, 1 August 2002, Vol. 93, No. 3, Pp. 303328 (26) ER 1105-2-100. Planning Guidance Notebook. Department of the Army, U.S. Army Corps of Engineers, Washington, DC, April 2000. Escudero, L.F. “Warsyp: a Robust Modeling Approach for Water Resources System Planning Under Uncertainty,“ Annals of Operations Research, 2000, Vol. 95, No. 1/4, Pp. 313339 (27) Evans J.S.T.B.T., et al. “Explicit and Implicit Processes in Multicue Judgment“ Memory and Cognition, 1 June 2003, Vol. 31, No. 4, Pp. 608-618 (11) Evans, R.J. “History, Memory, and the Law: the Historian As Expert Witness“ History and Theory, October 2002, Vol. 41, No. 3, Pp. 326-345 (20) Fahey L. Strategy and Leadership, 15 January 2003, Vol. 31, No. 1, Pp. 32-44 (13) Fahey, L., and Randall, R.M. “Learning From the Future: Competitive Foresight Scenarios,“ 1998 Farrington, J. “Socioeconomic Methods in Natural Resources Research,“ Natural Resource Perspectives, 1996, no. 9 Fayerweather W.E., et al. “Quantifying Uncertainty in a Risk Assessment Using Human Data,“ Risk Analysis, December 1999, vol. 19, no. 6, pp. 1077-1090 (14) Ferraro, D.O., C.M. Ghersa and G.A. Sznaider. “Evaluation of Environmental Impact Indicators Using Fuzzy Logic To Assess the Mixed Cropping Systems of the Inland Pampa, Argentina.” Agriculture, Ecosystems and Environment, June 2003, vol. 96, no. 1, pp. 1-18 (18) Fisher, E. “Drowning By Numbers: Standard Setting in Risk Regulation and the Pursuit of Accountable Public Administration.“ Oxford Journal of Legal Studies, 2000, vol. 20, no. 1, pp. 109-130 (22) Fischhoff, B. “Judgmental Aspects of Forecasting: Needs and Possible Trends,“ Int. J. Forecasting, 1988, vol. 7, pp. 421±433 Floris, F.J.T. and M.R.H.E. Peersmann. “Integrated Scenario and Probabilistic Analysis for Asset Decision Support,“ Petroleum Geoscience, 2002, vol. 8, no. 1, pp. 1-6 (6) Foran, B. and K. Wardle. “Transitions in Land Use and the Problems of Planning: a Case Study From the Mountain Lands of New Zealand,“ J. Environ. Manage. 1995, vol. 43, pp. 97±127 Ford, P.D. “Examining Child Sexual Abuse Evaluations: the Types of Information Affecting Expert Judgment*1.“ Child Abuse and Neglect, January 2001, vol. 25, no. 1, pp. 149-178 (30)



7-7

Section 7 Bibliography

Foxen, P. “Cacophony of Voices: A K’iche’ Mayan Narrative of Remembrance and Forgetting,“ Transcultural Psychiatry, September 2000, vol. 37, no. 3, pp. 355-381 (27) Frank, M.V. “Treatment of Uncertainties in Space Nuclear Risk Assessment with Examples From Cassini Mission Applications,“ Safety Factor Associates, Inc., Received 25 April 1998; Accepted 11 December 1998. Available Online 13 October 1999. Frederick, K.D., D.C. Major and E.Z. Stakhiv. “Water Resources Planning Principles and Evaluation Criteria for Climate Change: Summary and Conclusions,“Climatic Change, September 1997, vol. 37, no. 1, pp. 291-313 (23) Fuller, D., A. Akinwande and C. Sodini. “Leading, Following or Cooked Goose? Innovation Successes and Failures in Taiwan’s Electronics Industry.“ Industry and Innovation, June 2003, vol. 10, no. 2, pp. 179-196 (18) Ganzach, Y. “Nonlinear Models of Clinical Judgment: Communal Nonlinearity and Nonlinear Accuracy.“ Psychological Science, September 2001, vol. 12, no. 5, pp. 403-407 (5) Garthwaite, P.H. and A. O’hagan. “Quantifying Expert Opinion in the Uk Water Industry: An Experimental Study.“ The Statistician, Journal of the Royal Statistical Society - Series D, December 2000, vol. 49, no. 4, pp. 455-477 (23) Georgantzas, N.C. and W. Acar. Scenario-Driven Planning: Learning to Manage Strategic Uncertainty. Westport, CT: Quorum Books, 1995. Ghaffari, A., H.F. Cook and H.C. Lee. “Climate Change and Winter Wheat Management: a Modelling Scenario for South-Eastern England,“ Climatic Change, December 2002, vol. 55, no. 4, pp. 509-533 (25) Ghyym, S.H. “A Semi-Linguistic Fuzzy Approach To Multi-Actor Decision-Making: Application To Aggregation of Experts’ Judgments.” Annals of Nuclear Energy, August 1999, vol. 26, no. 12, pp. 1097-1112 (16) Giraud, G., J.C. Impara and C. Buckendahl. “Making the Cut in School Districts: Alternative Methods For Setting Cutscores.“ Educational Assessment, 1 August 2000, vol. 6, no. 4, pp. 291-304 (14) Godet, M. “Scenarios and Strategic Management: Prospective ET Plantification Stratégique,“ 1987, (D. Green and A. Rodney Translators). Gomez-Limon, J.A., M. Arriaza and J. Berbel. “Conflicting Implementation of Agricultural and Water Policies in Irrigated Areas in the Eu,“ Journal of Agricultural Economics, 1 July 2002, vol. 53, no. 2, pp. 259-281 (23) Granger, C.W.J., Forecasting in Business and Economics, Academic Press, New York, 1980. Gray V. “Developing the corporate memory: the potential of business archives.” Business Information Review, March 2002, vol. 19, no. 1, pp. 32-37 (6)

7-8



Section 7 Bibliography

Grimble, R. and Chan, Man-Kwun, “Stakeholder Analysis for Natural Resource Management in Developing Countries,“ Natural Resources Forum 1995, vol. 19, no.2, pp.113-124 Gross A. D. (1984). “How Golf Oil deals with uncertainty.“ Planning Review, 12 (2): 8-13. Gudjonsson, G. H. “The Ira Funeral Murders: the Confession of Pk and the Expert Psychological Testimony.” Legal and Criminological Psychology, 1 February 1999, vol. 4, no. 1, pp. 45-50 (6) Gyldenkaerne, S., et al. “Pesticide Leaching Assessment Method for Ranking Both Single Substances and Scenarios of Multiple Substance Use,“ Chemosphere, April 1998, vol. 36, no. 10, pp. 2251-2276 (26) Gumbo, B., et al. “Coupling of Digital Elevation Model and Rainfall-Runoff Model in Storm Drainage Network Design.“ Physics and Chemistry of the Earth, Parts A/B/C, 2002, vol. 27, no. 11, pp. 755-764 (10) Gumerman E., J.G. Koomey and M.A. Brown. “Strategies for Cost-Effective Carbon Reductions: a Sensitivity Analysis of Alternative Scenarios,“ Energy Policy, November 2001, vol. 29, no. 14, pp. 1313-1323 (11) Hampton, M. and J. Sperling. “Positive Negative Identity in the Euro-Atlantic Communities: Germanys Past, Europe’s Future?.” Journal of European Integration, 1 January 2002, vol. 24, no. 4, pp. 281-301 (21) Hanso, M. “Institutional Theory and Educational Change.” Educational Administration Quarterly, December 2001, vol. 37, no. 5, pp. 637-661 (25) Harrell, A.T. “New Methods in Social Science Research: Policy Sciences and Future Research,“ Praeger, 1978 Harris K.M. “Can High Quality Overcome Consumer Resistance To Restricted Provider Access? Evidence From A Health Plan Choice Experiment,“ Health Services Research, June 2002, vol. 37, no. 3, pp. 551-571 (21) Harvey D.C. Jones R. “Custom and Habit (U.S.): The Meaning of Traditions and Legends in Early Medieval Western Britain.“ Geografiska Annaler: Series B, Human Geography, 1999, vol. 81B, no. 4, pp. 223-233 (11) Harwell, C.C., et al. “Use of a Conceptual Model of Societal Drivers of Ecological Change in South Florida: Implications of an Ecosystem Management Scenario,“ Urban Ecosystems, 11 October 1999, vol. 3, no. 3/4, pp. 345-368 (24) Helgason T. and S.W. Wallace. (1991). “Approximate scenario solutions in the progressive hedging algorithm: A numerical study with an application to fisheries management.” Annals of Operations Research, 31 (2): 425-444. Henderson-Sellers, B. and A. Henderson-Sellers. “Sensitivity Evaluation of Environmental Models Using Fractional Factorial Experimentation,“ Ecological Modeling, 1996, vol. 86, pp. 291–295



7-9

Section 7 Bibliography

Hofer, E., V. Javeri and H. Loffler. “A Survey of Expert Opinion and Its Probabilistic Evaluation for Specific Aspects of the SNR-300 Risk Study,“ Nuclear Technology, vol. 68, pp. 180-225, 1985. Holling, C.S. “Adaptive Environmental Assessment and Management,“ Wiley International Series On Applied Systems Analysis, 1978, Vol. 3 Hsia P., D. Kung and C. Sell. “Software Requirements and Acceptance Testing,“ Annals of Software Engineering, 1997, vol. 3, no. 1, pp. 291-317 (27) Humphreys, P., Human Reliability Assessors Guide, Safety and Reliability Directorate, United Kingdom Atomic Energy Authority, 1988. Hurley, W.J. and D.U. Lior. “Combining Expert Judgment: on the Performance of Trimmed Mean Vote Aggregation Procedures in the Presence of Strategic Voting.“ European Journal of Operational Research, 1 July 2002, vol. 140, no. 1, pp. 142-147 (6) Huss, W.R. and E.J. Honton. “Scenario Planning: What Style Should You Use?“ Long Range Planning, 1987, vol. 20, no.4, pp.21-29 Hunt, E.R., S.C. Piper and J.C. Winslow. “The Influence of Seasonal Water Availability On Global C3 Versus C4 Grassland Biomass and Its Implications for Climate Change Research,“ Ecological Modeling, 1 May 2003, vol. 163, no. 1, pp. 153-173 (21). Ichii, K., et al. “A Simple Global Carbon and Energy Coupled Cycle Model for Global Warming Simulation: Sensitivity to the Light Saturation Effect,“ Tellus B, April 2003, vol. 55, no. 2, pp. 676-691 (16). Imundo A. (1986). Are you optimizing returns on your check processing dollar? Bank Systems and Equipment, 23 (9): 76-78. Jackson, D.A., et al. “Developing A Generalized Flaw Distribution For Reactor Pressure Vessels“ Nuclear Engineering and Design, 1 September 2001, vol. 208, no. 2, pp. 123-131 (9) Jauch and Glueck. Business Policy and Strategic Management, 1st Edition, Jauch and Glueck, McGraw-Hill, 1988. Jinno, K. “Performance Risk Analysis for Fukuoka Water Supply System,“ Water Resources Management, February 1998, vol. 12, no. 1, pp. 13-30 (18). Jones, S.T. (1985). Multiple scenario planning: Atlantic Richfield’s experience. L Journal of Business Forecasting, 4 (3): 19-23. Jorgensen, S.E. Fundamentals of Ecological Modeling, 1994, 2nd Edition, p. 628. Kahane, A. “Scenarios for Energy: Sustainable World vs. Global Mercantilism,“ Long Range Planning, 1992, vol. 25, no. 4, pp. 38-46. Kahn, H., 1965. On Escalation: Metaphors and Scenarios. Praeger, New York.

7-10



Section 7 Bibliography

Kaluarachchi, J.J. and I. Khadam. “Applicability of Risk-Based Management and the Need for RiskBased Economic Decision Analysis At Hazardous Waste Contaminated Sites,“ Environment International, July 2003, vol. 29, no. 4, pp. 503-519 (17). Kaplan, S. and B. Garrick. “On the Quantitative Definition of Risk,“ Risk Analysis, vol. 1, pp. 11-27, 1981. Kaplan, S., Y.Y. Haimes and B.J. Garrick. “Fitting Hierarchical Holographic Modeling into the Theory of Scenario Structuring and a Resulting Refinement to the Quantitative Definition of Risk,“ Risk Analysis, October 2001, vol. 21, no. 5, pp. 807-807 (1). Kayse, R.K., et al. “Recent Developments and Present Status of Telepathology.” Analytical Cellular Pathology, 2000, vol. 21, no. 3-4, pp. 101-106 (6). Kazman, R., S.J. Carrière and S.G. Woods. “Toward a Discipline of Scenario-Based Architectural Engineering,“ Annals of Software Engineering, 2000, vol. 9, no. 1/4, pp. 5-33 (29). Kemp, W. “Who is Taking Notes? Institutional Memory and the OSCE.“ Helsinki Monitor, 2001, vol. 12, no. 4, pp. 243-244 (2). Kennedy, P., C. Perrottet and C. Thomas. “Scenario Planning After 9/11: Managing the Impact of a Catastrophic Event,“ Strategy and Leadership, 2003, vol. 31 no. 1 pps. 4 – 13. Kessler, J.H. and R.K. Mcguire. “Total System Performance Assessment for Waste Disposal Using A Logic Tree Approach.“ Risk Analysis, October 1999, Vol. 19, No. 5, Pp. 915-931 (17). Khan, Faisal, I. Husain, Tahir and Abbasi, S. A. “Design and Evaluation of Safety Measures Using a Newly Proposed Methodology “SCAP““ Journal of Loss Prevention in the Process Industries, March 2002, vol. 15, no. 2, pp. 129-146 (18). Kim M-S. “Cloning and Deliberation: Korean Consensus Conference.” Developing World Bioethics, November 2002, vol. 2, no. 2, pp. 159-172 (14). Klein. H.E. and R.E. Linneman. (1984). “Environmental assessment: An international study of corporate practices.” Journal of Business Strategy, 5: 66-84. Kraan B.C.P. and R.M. Cooke. “Uncertainty in Compartmental Models For Hazardous Materials A Case Study.” Journal of Hazardous Materials, 7 January 2000, vol. 71, no. 1, pp. 253268 (16). Kumar, M.J. “The White House As City Hall: A Tough Place to Organize.” Presidential Studies Quarterly, March 2001, vol. 31, no. 1, pp. 44-55 (12). Lad, F. and M. Di Bacco. “Assessing the Value of A Second Opinion: the Role and Structure of Exchangeability.” Annals of Mathematics and Artificial Intelligence, May 2002, vol. 35, no. 1-4, pp. 227-252 (26). Leden, L., P. Garder and U. Pulkkinen. “An Expert Judgment Model Applied To Estimating the Safety Effect of A Bicycle Facility.” Accident Analysis and Prevention, July 2000, vol. 32, no. 4, pp. 589-599 (11).



7-11

Section 7 Bibliography

Lee, Y.T., D. Okrent and G. Apostolakis. “A Comparison of Background Seismic Risks and the Incremental Seismic Risk Due to Nuclear Power Plants,“ Nuclear Engineering and Design, vol. 53, pp. 141-154, 1979. Lee, H., C. Lee and C. Yoo. “A Scenario-Based Object-Oriented Hypermedia Design Methodology,“ Information and Management, September 1999, vol. 36, no. 3, pp. 121138 (18). Legesse, D., C. Vallet-Coulomb and F. Gasse. “Hydrological Response of a Catchment to Climate and Land Use Changes in Tropical Africa: Case Study South Central Ethiopia,“ Journal of Hydrology, 25 April 2003, vol. 275, no. 1, pp. 67-85 (19). Lelong, S. and C. Santos. “The child that shames dead as indication of a psychotic potentiality.“ Annales medico-psychologiques, May 2001, vol. 159, no. 4, pp. 336-340 (5). Lessard, G. “An Adaptive Approach to Planning and Decision Making,“ Landscape and Urban Planning 1998, vol.40, no. 1-3, pp.81-87 Lettenmaier, D.P., et al. “Water Resources Implications of Global Warming: a U.S. Regional Perspective,“ Climatic Change, November 1999, vol. 43, no. 3, pp. 537-579 (43) Leufkens H., Y. Hekster and S. Hudson. “Scenario Analysis of the Future of Clinical Pharmacy,“ Pharmacy World and Science, August 1997, vol. 19, no. 4, pp. 182-185 (4) Levine, S. and N. Rasmussen. “Nuclear Plant PRA: How Far Has It Come,“ Risk Analysis, vol. 4, 247-255, 1984. Lewis, H.W., R.J. Budnitz, H.J.C. Kouts, W.B. Lowenstein, W.D. Rowe, Von F. Hippel, and F. Zachariasen. Risk Assessment Review Group Report to the U.S. Nuclear Regulatory Commission, NUREG/CR-0400, 1979. Liken, M.A. “Managing Transitions and Placement of Caring for a Relative With Alzheimer’s Disease.” Home Health Care Management and Practice, December 2001, vol. 14, no. 1, pp. 31-39 (9) Linneman R.E. and H.E. Klein. (1983). “The use of multiple scenarios by US industrial companies: A comparison study, 1977-1981.” Long Range Planning, 16 (6): 94-101. Lind, M.R. and J.M. Sulek. “A Methodology for Forecasting Knowledge Work Projects.” Computers and Operations Research, September 2000, vol. 27, no. 11, pp. 1153-1169 (17) Lloyd, D. “Colonial Trauma/Postcolonial Recovery?“ Interventions: International Journal of Postcolonial Studies, 1 July 2000, vol. 2, no. 2, pp. 212-228 (17) Locke, A. and D. Edwards. “Bill and Monica: Memory, emotion and normativity in Clintons Grand Jury testimony.” British Journal of Social Psychology, 1 June 2003, vol. 42, no. 2, pp. 239-256 (18)

7-12



Section 7 Bibliography

Lucius-Hoene, G. and A. Deppermann. “Narrative Identity Empiricized: A Dialogical and Positioning Approach to Autobiographical Research Interviews.” Narrative Inquiry, December 2000, vol. 10, no. 1, pp. 199-222 (24) Lundberg, C.G. and B.M. Nagle. “Post-Decision Inference Editing of Supportive and Counterindicative Signals Among External Auditors in A Going Concern Judgment.” European Journal of Operational Research, 16 January 2002, vol. 136, no. 2, pp. 264281 (18) Lupton D. and J. Tulloch. “Theorizing Fear of Crime: Beyond the Rational/Irrational Opposition.“ British Journal of Sociology, 1 September 1999, vol. 50, no. 3, pp. 507-523 (17) Maarleveld, M. and C. Dangbegnon. 1998. “Managing Natural Resources in Face of Evolving Conditions: a Social Learning Perspective,“ 76 E. Wollenberg et al. / Landscape and Urban Planning 47 (2000) 65±77 Paper Presented At the Conference Crossing Boundaries, 7th Conference of the International Association for the Study of Common Property, Vancouver, Canada, 10±14, 1998, June. Macdonell S.G. and M.J. Shepperd. Combining Techniques to Optimize Effort Predictions in Software Project Management.” Journal of Systems and Software, 15 May 2003, vol. 66, no. 2, pp. 91-98 (8). Malafant, K.W.J. and D.P. Fordham. “GIS, DSS and Integrated Scenario Modeling Frameworks for Exploring Alternative Futures,“ In: J.L. USO, C.A. Brebbia and H. Power. (Eds.), Advance in Ecological Sciences: Ecosystems and Sustainable Development, 1997, vol. 1, Proceedings of a Conference, Peniscola, Spain, 14-16 October 1997, pp. 669678. Mantovani, G. “Social Context in HCL: A New Framework for Mental Models, Cooperation and Communication,“ Cognitive Science, 6 April 1996, vol. 20, no. 2, pp. 237-269 (33). Marcot, B.G., et al. “Using Bayesian Belief Networks to Evaluate Fish and Wildlife Population Viability Under Land Management Alternatives From an Environmental Impact Statement,“ Forest Ecology and Management, 1 October 2001, vol. 153, no. 1, pp. 2942 (14). Marti P., F. Pucci and A. Rizzo. “External Aids for Social Memory,“ Information, Communication and Society, 1 June 2001, vol. 4, no. 2, pp. 261-273 (13). Mathevet, R., et al. “Agent-Based Simulations of Interactions Between Duck Population, Farming Decisions and Leasing of Hunting Rights in the Camargue (Southern France),“ Ecological Modelling, 15 July 2003, vol. 165, no. 2, pp. 107-126 (20). Matsuoka U. “Fuel and Energy Abstracts,“ November 1995, vol. 36, no. 6, pp. 467-467 (1). McCarthy, M.A., M.A. Burgman and S. Ferson. “Sensitivity Analysis for Models of Population Viability,“ Biological Conservation 1995, pp. 73, 93–100.



7-13

Section 7 Bibliography

McGovern, T.J. and H.W. Samuels. “Our Institutional Memory at Risk: Collaborators to the Rescue.“ Campus-Wide Information Systems, 14 August 1998, vol. 15, no. 3, pp. 103107 (5). Mcintosh, N., et al. “Genetic Counseling For Fragile X Syndrome: Recommendations of the National Society of Genetic Counselors.“ Journal of Genetic Counseling, August 2000, vol. 9, no. 4, pp. 303-325 (23). Mclain, R.J. and R.G. Lee. “Adaptive Management: Promises and Pitfalls,“ Environ. Manage. 1996, vol. 20, no.4, pp. 437-448. Mercer, D. “Scenarios Made Easy.“ Long Range Planning, August 1995, vol. 28, no. 4, pp. 78+81 (2), Elsevier Science. Merkhofer, M. and R. Keeney. “A Multiattribute Utility Analysis of Alternative Sites for the Disposal of Nuclear Waste,“ Risk Analysis, vol. 7, no. 2, pp. 173-194, 1987. Millett, S.M. “How Scenarios Trigger Strategic Thinking,“ Long Range Planning, 1988, vol. 21, no. 5, pp. 61-68. Millett S.M. and F. Randles. (1986). “Scenarios for strategic business planning: A case history for aerospace and defence companies.” Interfaces, 16 (6): 64-72. Minarick, J. and C. Kukielka. “Precursors to Potential Severe Core Damage Accidents: 19691979,“ NUREG/CR-2497, 1982. Mitr, S.K. “Language and federalism: the multi-ethnic challenge.” International Social Science Journal, March 2001, vol. 53, no. 167, pp. 51-60 (10). Mitter, R. “Old Ghosts, New Memories: Chinas Changing War History in the Era of Post-Mao Politics,“ Journal of Contemporary History, 1 January 2003, vol. 38, no. 1, pp. 117-131 (15). Mizina, S.V., et al. “An Evaluation of Adaptation Options For Climate Change Impacts on Agriculture in Kazakhstan.“ Mitigation and Adaptation Strategies For Global Change, 1999, vol. 4, no. 1, pp. 25-41 (17). Montgomery, D.C. “Design and Analysis of Experiments,“ 1997, Wiley, New York, p. 704. “Movements.“ Qualitative Sociology, 2002, vol. 25, no. 3, pp. 443-458 (16). Morgan, G.M. , D. Amaral, D., M. Henrion and S. Morris. “Technological Uncertainty in Quantitative Policy Analysis—A Wulfur Pollution Example,“ Risk Analysis, vol. 3, pp. 201-220, 1984. Morgan, G.M. and M. Henrion, Uncertainty; a Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis, Department of Engineering and Public Policy, Carnegie Mellon University, 1988.

7-14



Section 7 Bibliography

Morris, J. M. and R.J. D’Amore. “Aggregating and Communicating Uncertainty,“ Pattern Analysis and Recognition Corp., 228 Liberty Plaza, Rome, New York, 1980. Mosleh, A., V. Bier and G. Apostolakis. “Critique of Current Practice for the Use of Expert Opinions in Probabilistic Risk Assessment,“ Reliability Engineering and System Safety, vol. 20, pp. 63-85, 1988. Mozina M., “Can we remember differently? A case study of the new culture of memory in voluntary organizations,.” International Journal of Social Welfare, October 2002, vol. 11, no. 4, pp. 310-320 (11). Mulvey J.M., D.P. Rosenbaum and B. Shetty. “Parameter Estimation in Stochastic Scenario Generation Systems,“ European Journal of Operational Research, 1 November 1999, vol. 118, no. 3, pp. 563-577 (15). Nobel, C.E. and D.T. Allen. “Using Geographic Information Systems (Gis) in Industrial Water Reuse Modelling,“ Process Safety and Environmental Protection, 1 July 2000, vol. 78, no. 4, pp. 295-303 (9). Nordmann, L.H. and J.T. Luxhoj. “Application of A Performance Measure Reduction Technique To Categorical Safety Data,“ Reliability Engineering and System Safety, January 2002, vol. 75, no. 1, pp. 59-71 (13). Nuclear Engineering and Design Volume 93 (1986). Nyhart, L.K. “The Importance of the “Gegenbaur School“ for German Morphology“Theory in Biosciences, 1 May 2003, vol. 122, no. 2-3, pp. 162-173 (12). O’Connor A. “WHOS EMMA AND THE LIMITS OF CULTURAL STUDIES.” Cultural Studies, 1 October 1999, vol. 13, no. 4, pp. 691-702 (12). Ohe, T. “Simple Assignment of Partitioning and Transmutation Objectives to Reduce the Overall Repository Impacts Due to Thermal Load and Nuclide Migration,“ Progress in Nuclear Energy, April 2002, vol. 40, no. 3, pp. 423-430 (8). Okrent, D., “A Survey of Expert Opinion on Row Probability Earthquakes,“ in Annals of Nuclear Energy, Pergamon Press, pp. 601-614, 1975. Ozbekahn, H. “The Future of Paris: a Systems Study in Strategic Urban Planning,“ Philosophical Transactions of the Royal Society of London, 1977, pp. 387: 523-544. Phelps, R., C. Chan and S.C. Kapsalis. “Does Scenario Planning Affect Performance? Two Exploratory Studies,“ Journal of Business Research, March 2001, vol. 51, no. 3, pp. 223232 (10). Plackett, R.L. and J.P. Burman. “The Design of Optimum Multifactorial Experiments,“ Biometrika, 1946, Vol. 33, Pp305–325. “Political Studies Books.” Political Studies, August 2002, vol. 50, no. 3, pp. 587-658 (3).



7-15

Section 7 Bibliography

Pollitt, C. “Institutional Amnesia: A Paradox of the ‘Information Age’?“ Prometheus, 1 March 2000, vol. 18, no. 1, pp. 5-16 (12). Ports R. E.E. (1985). “A strategic planning system in a Canadian gas utility.” Public Utilities Fortnightly, 116 (8): 22-30. Prebble J. F. and A.Reichel. (1988). “Scanning the future environment for banking.” Mid. American Journal of Business, 3 (2): 23-31. Ramachandran, G. “Retrospective Exposure Assessment Using Bayesian Methods“ The Annals of Occupational Hygiene, November 2001, vol. 45, no. 8, pp. 651-667 (17). Ramachandran G., S. Banerjee and J.H. Vincent. “Expert Judgment and Occupational Hygiene: Application To Aerosol Speciation in the Nickel Primary Production Industry.” Annals of Occupational Hygiene, August 2003, vol. 47, no. 6, pp. 461-475 (15). Ramachandran, G. and J.H. Vincent. “A Bayesian Approach To Retrospective Exposure Assessment.” Applied Occupational and Environmental Hygiene, 1 August 1999, vol. 14, no. 8, pp. 547-557 (11). Regnell, B., P. Runeson and T. Thelin. “Are the Perspectives Really Different? – Further Experimentation On Scenario-Based Reading of Requirements,“ Empirical Software Engineering, December 2000, vol. 5, no. 4, pp. 331-356 (26). Renn, O. Precautionary Principle: Risk Uncertainty and Rational Action. Prof. Ortwin Renn, University of Stuttgart, Germany http://www.allchemeseminars.org/downloads/03-10-01/20031001report.pdf. Ribera, I., et al. “Discovery of Aspidytidae, a New Family of Aquatic Coleoptera.” Proceedings: Biological Sciences, 22 November 2002, vol. 269, no. 1507, pp. 2351-2356 (6). Rienstra S.A. and P. Nijkamp. “The Role of Electric Cars in Amsterdam’s Transport System in the Year 2015, a Scenario Approach,“ Transportation Research Part D: Transport and Environment, January 1998, vol. 3, no. 1, pp. 29-40 (12). Ringland, G. “Using Scenarios to Focus R&D,“ Strategy and Leadership, 2003a, vol. 31, no.1, pp.45-55. Ringland, G. (2003b) Scenario planning: persuading operating managers to take ownership Strategy and Leadership 1 June 2003, vol. 31, no. 6, pp. 22-28 (7). Risbey, J.S. “Sensitivities of Water Supply Planning Decisions to Stream Flow and Climate Scenario Uncertainties,“ Water Policy, June 1998, vol. 1, no. 3, pp. 321-340 (20). Rothman, H.K. “Institutional Memory and Management Needs: History in the Park Service’s Northwest“ The Public Historian, 1 April 2001, vol. 23, no. 2, pp. 87-91 (5). Rothstein, B. Trust, Social Dilemmas and Collective Memories. Journal of Theoretical Politics, October 2000, vol. 12, no. 4, pp. 477-501 (25).

7-16



Section 7 Bibliography

Rowe, G. and G. Wright. “Differences in Expert and Lay Judgments of Risk: Myth Or Reality?“ Risk Analysis, April 2001, vol. 21, no. 2, pp. 341-356 (16). Rushton, M. “Political Oversight of Arts Councils: A Comparison of Canada and the United States.” International Journal of Cultural Policy, 1 January 2002, vol. 8, no. 2, pp. 153-165 (13). Saccomanno, F. and P. Haastrup. “Influence of Safety Measures on the Risks of Transporting Dangerous Goods Through Road Tunnels.” Risk Analysis, December 2002, vol. 22, no. 6, pp. 1059-1069 (11). Sakellaropoulos, G.C. and G.C. Nikiforidis. “Prognostic Performance of Two Expert Systems Based on Bayesian Belief Networks.” Decision Support Systems, January 2000, vol. 27, no. 4, pp. 431-442 (12). Samet, M.G., “Quantitative Interpretation of Two Qualitative Scales Used to Rate Military Intelligence,“ Human Factors, vol. 17, no. 2, pp. 192-202, 1975. Sapio, B. “Search (Scenario Evaluation and Analysis Through Repeated Cross Impact Handling): a New Method for Scenario Analysis With an Application to the Videotel Service in Italy,“ Int. J. Forecast, 1995, vol.11, no. 1, pps.113-131. Sarangi, S. and A. Clarke. “Zones of Expertise and the Management of Uncertainty in Genetics Risk Communication.” Research on Language and Social Interaction, 1 April 2002, vol. 35, no. 2, pp. 139-171 (33). Satterfield, T., P. Slovic and R. Gregory. “Narrative Valuation in A Policy Judgment Context.” Ecological Economics, September 2000, vol. 34, no. 3, pp. 315-331 (17). Schoemaker, P.J.H. “Multiple Scenario Development: Its Conceptual and Behavioral Foundation,“ Strategic Manage. J. 1993, vol. 14, no. 3, pp. 193-213. Schoemaker, P.J.H. “When and How to Use Scenario Planning: a Heuristic Approach With Illustration,“ J. Forecast, 1991, vol. 10, pp. 549-564. Schriefer, A. and D. Mercer. “Scenarios Made Easy - Getting the Most Out of Scenarios“ The Journal of Product Innovation Management, March 1996, vol. 13, no. 2, pp. 175-176 (2) Elsevier Science. Schuenemeyer, J.H. “A Framework For Expert Judgment To Assess Oil and Gas Resources.” Natural Resources Research, June 2002, Vol. 11, No. 2, Pp. 97-107 (11). Schultz D.I. (1986). “Strategic information systems planning sharpens competitive edge.” Data Management, 24 (6): 20-24, 38. Schwartz P. (1991). The Art of the Long View. New York: Doubleday Currency. Schwenk C.R. (1984). “Cognitive simplification processes in strategic decisionmaking.” Strategic Management Journal, 5 (2): 111-128.



7-17

Section 7 Bibliography

Seraphine, A.E. “The Performance of Dimtest When Latent Trait and Item Difficulty Distributions Differ.” Applied Psychological Measurement, March 2000, vol. 24, no. 1, pp. 82-94 (13). Shetle, R.K., A. Karlsdottir and V. Froelicher. “Assessing Patients With Possible Heart Disease Using Scores.” Sports Medicine, 1 June 2001, vol. 31, no. 6, pp. 387-408 (22). Shooman, M. and S. Sinkar. “Generation of Reliability and Safety Data by Analysis of Expert Opinion.“ Proc. 1977 Annual Reliability and Maintainability Symposium, pp. 186-193, 1977. Shu, Y., K. Furuta and S. Kondo. “Team Performance Modeling for Hra in Dynamic Situations.“ Reliability Engineering and System Safety, November 2002, vol. 78, no. 2, pp. 111-121 (11). Silberfeld, M. and D. Checkland. “Faulty Judgment, Expert Opinion and Decision-Making Capacity.” Theoretical Medicine and Bioethics, August 1999, vol. 20, no. 4, pp. 377-393 (17). Siu T.K. and H. Yang. “Subjective Risk Measures: Bayesian Predictive Scenarios Analysis.“ Insurance: Mathematics and Economics, 16 November 1999, vol. 25, no. 2, pp. 157169 (13). Slocum, R. and D. Klaver. “Time Line Variations.“ In: R. Slocum, L. Wichart, D. Rocheleau, B. Thomas-Slayter. (Eds.), Power, Process and Participation -Tools for Change. 1995, pp. 194-197. Squarcini, F. “In Search of Identity Within the Hare Krishna Movement: Memory, Oblivion and Thought Style.” Social Compass, June 2000, vol. 47, no. 2, pp. 253-271 (19). Stamelos, I., et al. “On the Use of Bayesian Belief Networks for the Prediction of Software Productivity.” Information and Software Technology, 1 January 2003, vol. 45, no. 1, pp. 51-60 (10). Starfield, A.M. and A.L. Bleloch. “Building Models for Conservation and Wildlife Management.“ Burgess International Group Inc., 1991, p. 253. Steelman, T.A. and W. Ascher. “Public Involvement Methods in Natural Resource Policy Making: Advantages, Disadvantages and Trade-Offs.“ Policy Sciences, 1997. vol. 30, pp. 71-90. Stern, P.C. and D. Druckman. “Evaluating Interventions in History: The Case of International Conflict Resolution.“ The International Studies Review, Spring 2000, vol. 2, no. 1, pp. 33-63 (31). Stewart, J.T. and L. Scott. “A Scenario-Based Framework for Multi-criteria Decision Analysis in Water Resources Planning.“ Water Resources, 1995, vol. 31, no. 11, pp. 2835-2843. Sui, N. and G. Apostolakis. “Combining Data and Judgment in Fire Risk Analysis.“ 8th International Conference on Structural Mechanics in Reactor Technology, Brussels, Belgium, August 26-27, 1985.

7-18



Section 7 Bibliography

Svartdal, F. “Persistence During Extinction: Are Judgments of Persistence Affected by Contingency Information?“ Scandinavian Journal of Psychology, December 2000, vol. 41, no. 4, pp. 315-328 (14). Swain, A. and H. Guttman. Handbook of Human Reliability, Analysis With Emphasis on Nuclear Power Plant Applications. NUREG/CR-1278, 1983. Swaminathan, S. and C. Smidts. “The Mathematical Formulation for the Event Sequence Diagram Frame Work.“ Reliability Engineering Program, Received 22 June 1998; Accepted 10 October 1998; Available Online 31 March 1999. Swartzman, G.L. and S.P. Kaluzny. “Ecological Simulation Primer.“ 1987, Macmillan, New York, p. 370. Tan-Kim-Yong, U. “Participatory Land-Use Planning for Natural Resource Management in Northern Thailand.“ Network Paper 1992, no. 14b, Rural Development Forestry Network. Tao F., et al. “Future Climate Change, the Agricultural Water Cycle and Agricultural Production in China.“ Agriculture, Ecosystems and Environment, April 2003, vol. 95, no. 1, pp. 203215 (13). Taylor, B., L. Kremsater and R. Ellis. “Adaptive Management of Forests in British Columbia.“ British Columbia Ministry of Forests, Canada, Report, 1997. Taylor, G. and A. Horan. “From Cats, Dogs, Parks and Playgrounds to IPC Licensing: Policy Learning and the Evolution of Environmental Policy in Ireland.” British Journal of Politics and International Relations, October 2001, vol. 3, no. 3, pp. 369-392 (24). Temkin, M., “Avec un certain malaise: The Paxtonian Trauma in France, 1973-74.” Journal of Contemporary History, 1 April 2003, vol. 38, no. 2, pp. 291-306 (16). Todorovska, M.I. and M.D. Trifunac. “Northridge, California, Earthquake of 1994: Density of Pipe Breaks and Surface Strains.“ Soil Dynamics and Earthquake Engineering, 1997, vol. 16, no. 3, pp. 193-207 (15). Tol, R.S.J., et al. “Adapting to Climate: A Case Study on Riverine Flood Risks in the Netherlands.“ Risk Analysis, June 2003, vol. 23, no. 3, pp. 575-583 (9). Trenberth, K. “Science Considers Global Climate Change? No Longer any Reasonable Disagreement Says U.S. Expert.” Refocus, November 2002, vol. 2002, no. 11, pp. 50+52-53 (4). Trepel, M. and W. Kluge. “Ecohydrological Characterization of a Degenerated Valley Peatland in Northern Germany for Use in Restoration.“ Journal for Nature Conservation, November 2002, vol. 10, no. 3, pp. 155-169, (15) and #169. Tuljapurkar, S. and C. Boe. “Validation, Probability-Weighted Priors and Information in Stochastic Forecasts.“ International Journal of Forecasting, July 1999, vol. 15, no. 3, pp. 259-271 (13).



7-19

Section 7 Bibliography

U.S. Nuclear Regulatory Commission. “Nuclear Regulatory Commission Issues Policy Statement on Reactor Safety Study and Review by the Lewis Panel.“ Nuclear Regulatory Commission Press Release, no. 79-19, January 19, 1979. U.S. Water Resources Council. 1983. Economic and Environmental Principles and Guidelines for Water and Related Land Resources Implementation Studies. Washington, D.C: U.S. Government Printing Office (March 10, 1983). Van De Klundert, A.F. 1995. “The Future’s Future: Inherent Tensions Between Research, Policy and the Citizen in the Use of Future Oriented Studies.“ In: Scenario Studies for the Rural Environment. Proceedings of the Symposium Scenario Studies for the Rural Environment, J.F.T. Schoute, P.A. Finke, F.R. Veeneklaas and H.P. Wolfert. (Eds.), 1215 September 1994, pp. 25-32. Van Der Fels-Klerx, I.H.J., et al. “Elicitation of Quantitative Data From a Heterogeneous Expert Panel: Formal Process and Application in Animal Health.” Risk Analysis, February 2002, vol. 22, no. 1, pp. 67-81 (15). Van Der Linden, S., et al. “Sensitivity Analysis of Discharge in the Arctic USA Basin, EastEuropean Russia.“ Climatic Change, March 2003, vol. 57, no. 1, pp. 139-161 (23). Van Dorp, J.R., et al. “A Risk Management Procedure for the Washington State Ferries.” Risk Analysis, February 2001, vol. 21, no. 1, pp. 127-142 (16). Van Huylenbroeck, G. and A. Coppens. “Multi-Criteria Analysis of the Conflicts Between Rural Development Scenarios in the Gordon District Scotland.“ J. Environ. Plann Manage. 1995 vol. 38 no. 3, pp. 393-407. Van Imhoff E. and K. Henkens. “The Budgetary Dilemmas of an Ageing Workforce: A Scenario Study of the Public Sector in the Netherlands.“ European Journal of Population/ Revue European De Dmographie, March 1998, vol. 14, no. 1, pp. 39-59 (21). Van Noortwijk, J.M. “Optimal Maintenance Decisions on the Basis of Uncertain Failure Probabilities.” Journal of Quality in Maintenance Engineering, 1 February 2000, vol. 6, no. 2, pp. 113-123 (11). Veldkamp, A. and L.O. Fresco. “Exploring Land Use Scenarios: An Alternative Approach Based on Actual Land Use.“ Agric. Syst. 1997 vol. 55 no. 1, pp. 1-17. Violato, C., A. Marini and C. Lee. “A Validity Study of Expert Judgment Procedures for Setting Cutoff Scores on High-Stakes Credentialing Examinations Using Cluster Analysis.” Evaluation and the Health Professions, 1 March 2003, vol. 26, no. 1, pp. 59-72 (14). Wack, P. “Scenarios: Uncharted Waters Ahead.“ Harvard Business Review 1985a vol. 63 no. 5, pp. 72-89. Wack, P. “Scenarios: Shooting the Rapids,“ Harvard Business Review 1985b vol. 63 no.6, pp. 139-150.

7-20



Section 7 Bibliography

Wagner, D. “Assessment of the Probability of Extreme Weather Events and Their Potential Effects in Large Conurbations.“ Available Online 23 July 1999. Walters, C. “Adaptive Management of Renewable Resources.“ 1986, Macmillan. Werritty, A. “Living With Uncertainty: Climate Change, River Flows and Water Resource Management in Scotland.“ The Science of the Total Environment, 22 July 2002, vol. 294, no. 1, pp. 29-40 (12). Wiggins, J. “ESA Safety Optimization Study.“ Hernandez Engineering, HEI-685/1026, Houston, Texas, 1985. Wohlin C. and A.A. Andrews. “Prioritizing and Assessing Software Project Success Factors and Project Characteristics Using Subjective Data.” Empirical Software Engineering, September 2003, vol. 8, no. 3, pp. 285-308 (24). Wolf, J. and C.A. Diepen. “Effects of Climate Change on Grain Maize Yield Potential in the European Community.“ Climate Change, 1995, vol. 29, no. 3, pp. 299-331. Wollenberg, E. et al. Landscape and Urban Planning. 47 (2000) 65±77 67. Woods B. “Dementia Care: Progress and Prospects.” Journal of Mental Health, 1 April 1995, vol. 4, no. 2, pp. 115-124 (10). Worsley, H. “The Impact of the Inner-Child on Adult Believing.” Journal of Beliefs and Values, 1 October 2002, vol. 23, no. 2, pp. 191-202 (12). Wright P.A. and T.V. Hill (1986). ”Cost Estimating: Dealing With Uncertainty.” AACE Transactions: E.5. 1-E.5.8. Wright, G., A. Pearman and K. Yardle. “Risk Perception in the U.K. Oil and Gas Production Industry: Are Expert Loss-Prevention Managers Perceptions Different From Those of Members of the Public?“ Risk Analysis, October 2000, vol. 20, no. 5, pp. 681-690 (10). Yin, Y., J.T. Pierce and E. Love. “Designing a Multi-Sector Model for Land Conversion Study.“ J. Environ. Manage 1995, vol. 44, pp. 249-266. Yoe, C.E. Risk Analysis Framework for Cost Estimation. December 2000, IWR Report 00-R-9, U.S. Army Corps of Engineers, Alexandria, Virginia. Yoe, C.E. and K.D. Orth. Planning Manual. U.S. Army Corps of Engineers, Institute for Water Resources, IWR Report 96-R-21, November 1996. Young, W.J., et al. “Development of an Environmental Flows Decision Support System.“ Environmental Modeling and Software With Environment Data News, March 2000, vol. 15, no. 3, pp. 257-265 (9). Yu D. and W.S. Park. “Combination and Evaluation of Expert Opinions Characterized in Terms of Fuzzy Probabilities.” Annals of Nuclear Energy, May 2000, vol. 27, no. 8, pp. 713-726 (14).



7-21

Section 7 Bibliography

Zavirsek, D. “Pictures and Silences: Memories of Sexual Abuse of Disabled People.” International Journal of Social Welfare, October 2002, vol. 11, no. 4, pp. 270-28.

7-22