GeoMET 2013 Carrasco Lecture, 30Sept2013
Uncertainty, Decisions, Models & People Steve Begg Australian School of Petroleum University of Adelaide GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Outline •
•
Uncertainty, Decisions & Modeling –
Uncertainty & Business Perfomance
–
Uncertainty & Decision-making
–
Need for integrated models
Uncertainty, Judgments & People –
The Nature of Uncertainty
–
Uncertainty & Value Maximization
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Industry Performance: Comments & Observations • Super-major: – “Every one of our 10 most important projects failed to generate the desired return.”
• Large independent: – “The actual performance of our key assets wasn’t even within the P1 to P99 range.”
• CEO to manager: – “I want your guarantee that we will not spend more than the P50 on this project!”
• IPA: – “The bigger and more important a project gets, the more likely it ends up in the “disaster” category (1 in 8 major projects)” GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
A fundamental problem Business outcomes not living up to expectations or possibilities!
Uncertainty
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
A fundamental problem Business outcomes not living up to expectations or possibilities!
•
People tend to grossly under-estimate uncertainty – number of uncertain factors and the magnitude of uncertainty and is its consequences (good or bad) – complexity of the relationships between them and therefore unanticipated non-intuitive outcomes)
•
Naive understanding of NPV “rule” – Uncertainty (and/or delay) = Value Loss. Biased to risk mitigation.
•
Better decision-making requires accurate (= unbiased & appropriate range) uncertainty assessment & response – Reduce uncertainty only IF it can change a decision AND expected benefit of reduction is less than its cost – Mitigate downside risk AND capture upside opportunities – Exploit interactions – dependencies, correlations
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Uncertainties matter: they are everywhere in the (Oil & Gas) evaluation “system” …… Export
Taxes
Drilling Seismic
CapEx
Prices
OpEx
Royalties/PSC
Processing Geology OOIP Model
Economics Asset A
Predicted Production
Petro-physics Reservoir Simulation Production Allocation Economics Asset 1
. . .
Production Data
Portfolio
Decline GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Economics Asset n
Steve Begg
Uncertainties matter: they are everywhere in the evaluation “system” …… Export
Taxes
Mining Geophysics
CapEx
Prices
OpEx
Royalties/PSC
Processing Geology Resource Model Petro-physics
Economics Asset A
Predicted Production
Resource Simulation
Portfolio
Production Data
. . .
Economics Asset 1
Depletion GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Economics Asset n
Steve Begg
….. and occur at many levels ……. • Each domain must address nested layers/types of uncertainty, e.g. sub-surface description, …... Interpretation/Model Parameter / TI Spatial/Temporal variability Data Monte Carlo Geostatistics (
or mp)
Experimental Design Scenario Modeling, Discrete Probabilities
…... with appropriate modeling techniques GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
…. and we spend a lot of money without really knowing which ones matter Parameter
Uncertainty
Impact on NPV
Action Buy more seismic
Gross Rock Volume
Hedge with futures
Oil Price
Take more core
Average Porosity
Different rock model Saturation "There is nothing so inefficient as very Cheaper Steel Supply Facilities efficiently doing the wrong things".
Recovery Factor
simulator model PeterBuild Drucker
Rig Cost
Renegotiate contract
Net:Gross
More gamma logs
Continuity
Survey Analogues Fire lawyers
Fiscal Terms
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
…. and we spend a lot of money without really knowing which ones matter Uncertainty
Parameter
Impact on NPV
Action Drill more holes
Gross Ore-body Volume
Hedge with futures
Copper Price
Take more core
Average Porosity Grade
Different rock model
Processing Facilities Cost
Cheaper Steel Supply
Recovery Factor
Build simulator model
Mining Cost
Renegotiate contract
Net:Gross
More gamma logs
Continuity
Survey Analogues Fire lawyers
Fiscal Terms
"There is nothing so inefficient as very efficiently doing the wrong things".
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
A fundamental problem
Flawed Decision-Making Better performance
Better Decisions
Better uncertainty management GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Decision-making is about ranking: Case 1: we don’t even need a decision criterion We only need to predict values precisely enough to determine which is the best alternative Uncertainty in decision Criterion, eg NPV
Here, B is clearly the best choice
A GeoMet 2013, Brisbane
C B Alternatives (choices) Carrasco Lecture: Uncertainty, decisions models & people
D Steve Begg
Decision-making is about ranking: Case 2: need a decision criterion
Uncertainty in decision Criterion, eg NPV
We only need to predict values precisely enough to determine which is the best alternative
A GeoMet 2013, Brisbane
C B Alternatives (choices) Carrasco Lecture: Uncertainty, decisions models & people
D Steve Begg
Decision Criteria • OK, so we’ve got our PDFs of some decision variable (eg NPV, tonnage) for each decision alternative • What number should we use as a decision criterion? – the P10, P50, P90? – the mode? NPV
– the mean? 0
• By “decision criterion”, we mean a number that can be used – to compare the different decision alternatives, so that we can choose the best (if alternatives are mutually exclusive) – or compare against a minimum acceptable hurdle (if there is only one alternative, or multiple non-mutually exclusive alternatives)
• Answer: Expected Value (for a “risk-neutral” decision-maker)
GeoMet 2013, Brisbane
BUT don’t “expect” the Expected Value! Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Decision Criterion under Uncertainty: Choose the alternative with maximum Expected Value • Intuitively – Defines what would happen “on average” if we repeat the situation numerous times
• Mathematically – The probability weighted average of the possible values (discrete PDF) n
Expected Value =
pi x i
1
• BUT, don’t “expect” the Expected Value! • No other metric (Mode, P10/50/90, etc) will give a higher total value over multiple (different) decisions • For a “risk-neutral” decision-maker, it is value they would attribute to the alternative if all uncertainty was removed. GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Avoiding Confusion: Decision Criteria v Predictions • The PDF is the prediction of all possible outcomes (and their associated probabilities) • The mode is the Most Likely outcome
-
So, perhaps, it is the best prediction of outcome
• The Expected Value is a Decision Criterion – ie used to make the best decision – it is not a prediction of the outcome!
-
at least, it is no more of a prediction than any of the other possible outcomes in the PDF are
-
it might not even be a possible outcome!
• For decision making we need unbiased estimates of the EVs of our decision criteria (eg NPV, Reserves)
-
which requires propagating (unbiased) assessments of uncertainty in input variables through to uncertainty of the decision criteria
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Technical Work in the context of Decisions and Uncertainty: Knowing when enough is enough. The main role of a geoscientist, engineer or economist is to support decision-making • Technical work is fundamentally about uncertainty assessment for the purpose of making decisions – First priority: Accurate (=unbiased) uncertainty assessment – Second priority: Uncertainty reduction.
• But if you have a “make the best possible prediction” focus, there is no stopping rule – you can always reduce uncertainty a bit more (more data, more time, more detail, more accurate physics/geology/tax, more analysis, ...)
• A decision-driven (ranking) focus gives a trivially simple stopping rule – Stop when further analysis doesn’t change the decision!! GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Propagating Uncertainty: Flaw of Using Averages (after Savage)
Log, Price, NCF...
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Averages don’t always work
•
Mean = 10
For non-linear processes,
Mean = 5
–
reservoir simulation – volumetrics with cut-offs – development alternatives
3
even if only a single, “best” estimate is required, we still need to use complete range of inputs - cannot use an average input
•
x
21
1
z
8
Model Y = X2 Z2
/
Simulation Result
Also, P10 (P90) results are NOT given by taking P10 (P90) inputs and running them through the model
GeoMet 2013, Brisbane
True Mean Result ~ 7.8 Y=4 0
Carrasco Lecture: Uncertainty, decisions models & people
Y
30 Steve Begg
The Flaw of Averages: implications for decision metrics • If f is a decision metric (eg NPV) computed from a model with uncertain input variables x,y,z,…
[ f ( x , y , z ....) ]
f ( [ x ], [ y ], [ z ], ...) unless f is linear
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
On Models ……
“…. when you can measure what you are speaking about and express it in numbers, you know something about it. But when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind. It may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of science, whatever the matter may be.” Lord Kelvin Taken from Davis, 1986
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
On Models ……
What could be cuter Than to feed a computer With wrong information But naïve expectation To obtain with precision A Napoleonic decision Major Alexander P. de Seversky Taken from Davis, 1986
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Remember: Expected Value Parameter
Uncertainty
Impact on NPV
Action Buy more seismic
Gross Rock Volume
Hedge with futures
Oil Price
Take more core
Average Porosity Saturation
Different rock model
Facilities
Cheaper Steel Supply
Recovery Factor
Build simulator model
Rig Cost
Renegotiate contract
Net:Gross
More gamma logs
Continuity
Survey Analogues Fire lawyers
Fiscal Terms
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
How? "There is nothing so inefficient as very efficiently doing the wrong things". Peter Drucker
The industry has been over-focussed on: 1. Developing and using ever more efficient tools for trying to come up with precise “single number” estimates 2. Modelling approaches based on simply “wrapping” a probabilistic framework around our (very efficient!) classical deterministic tools 3. Modelling information (uncertainty) in the absence of its impact on decisions GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
On Models
“All models are wrong, some models are useful” Box
• Once you accept decisions really are made under uncertainty, and can be optimally so, it impacts the whole way you view “technical” work (geological, engineering, economic, commercial, legal)
“I would rather be vaguely right than precisely wrong” Keynes GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
How? Hi
Two Components Integrated Uncertainty 1. Modelling Evaluation “philosophy”
Vaguely Right Too expensive (opportunity Missing the to create point (ranking) value) High P(wrong)
(No. of“runs” or Scenarios)
Classical models (precisely wrong)
2. Type of Models
Lo Lo
Model “Richness”
Hi
Trade-off some “domain” rigor for ability to perform integrated uncertainty evaluation GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Stochastic Integrated Asset Modelling System (Oil&Gas). Insightful? Scenario and Decision Analysis
Decision, or feed to portfolio
Monte Carlo Simulation
Processing Facility Model
Export Model Costs
G&G Model Production Model
Prices
Economics Taxes & Royalties
Prob
Drilling Model
Sensitivity Analysis
Scheduling of Decisions & Implementation
Uncertainty Estimates, Calibration, Surrogates
Classical Modeling GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
The role of “Classical” Modeling • Uncertainty Estimates of Input Parameters – Simple sensitivity studies
• Calibration of simplified models • Generation of high-fidelity simple surrogates – Experimental Design and Response Surface Modeling Holistic Evaluation System Uncertainty Estimates, Calibration, Surrogates
Classical Modeling GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Application of Holistic Modelling System
Study
Simplified
Middle End Classical Approach
Detailed
Model Detail
Start
SIAM-based Approach Narrow
GeoMet 2013, Brisbane
Model Scope
Broad
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Outline • Uncertainty, Decisions & Modeling – Uncertainty & Business Performance – Uncertainty & Decisionmaking – Need for integrated models
• Uncertainty, Judgments & People – The Nature of Uncertainty – Uncertainty & Value Maximization
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Probability: The Language of Uncertainty • Classical (Theoretical) Number of outcomes representing the occurrence of an event Total number of possible outcomes – e.g. 30 red balls and 70 green balls in a bag. P(Red) = 30%
• Relative Frequency – Proportion of times an event occurs in the long run – Estimated from sample data ASSUMING identical events e.g. 15 out of 20 wells drilled were dry holes. P(Dry) = 75% – More accurate with greater sample size. May not apply to future.
• Subjective “All business
proceeds on beliefs, or – Personal degree of belief of the likelihood of a future event occurring judgments of probabilities, and not on (or of the unknown outcome of a past event) – May be based on some past similar / analogous occurrences certainties". Charles W. Eliot
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
The nature of uncertainty Throw a die and hide top face. What is the probability of a 3? 1/6
Now you get information. Has the top face changed? No Has the probability of a 3 changed? Yes! What is the probability of a 3 now? 1/3
Uncertainty is a function of what you know. There is no “right” uncertainty (or PDF)! GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Uncertainty is in OUR heads – it’s personal, a function of our state of knowledge Different people can, legitimately, hold different views about the uncertainty of an unknown quantity Person A
Person B
0.333 Prob.
Prob.
0.333
1
2
3 4 5 Outcome
6
1
2
3 4 5 Outcome
6
So we should not talk about THE probability of some outcome/statement, but about MY (or OUR) probability GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Uncertainty is in OUR heads – it’s a function of our state of knowledge Probability is not an inherent “parameter” of the “system”! • The systems that we deal with are essentially deterministic, its just our knowledge of them that is probabilistic • The Die:
-
when the die is tossed, there will be one face that comes up in theory, if we knew (precisely) its initial conditions and we could model (precisely) all of the processes involved with tossing it, we could predict how it would land – but that is, practically, impossible
• And so with Project reserves,cost, schedule etc:
-
its our knowledge that is probabilistic for events that have occurred, we could collect information to reduce uncertainty - complete information would resolve it
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Probability is subjective (personal) and depends upon your information
Person or Company A Info
Shared Info
A’s PDF GeoMet 2013, Brisbane
Person or Company B Info
B’s PDF Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Uncertainty vs Variability Variability of all sandbody widths
A distribution that describes the variability Width of a natural phenomenon is Uncertaintyto in individual not usually appropriate sand-body width ? ?describe the uncertainty Sand 1 of a single occurrence ?
?
Sand 2
Width GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Uncertainty vs Risk
Risk
Uncertainty
• A Risk (noun!) is one possible consequence of uncertainty. It has a negative connotation, which is “personal” to the D-M – an event that, if it occurs, has a negative impact on DM objectives – it is specified by defining the event and assessing its probability, GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Risk isn’t the only possible aspect, or consequence, of uncertainty Consequences of Uncertainty
Opportunity
Risk Possibility of loss or injury
Possibility of exceeding expectations
A dangerous element or factor Upside potential The degree of probability of loss
A wonderful element or factor
• Risk is one outcome of uncertainty - but so is Opportunity! – often over-looked - is a source of value creation GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Uncertainty, decisions & people With respect to uncertainty, the main enemy of good decision making is bias, not the uncertainty being too great • Bias in central value (mean): – eg “rose-tinted glasses”
• Bias in width of distribution: – eg assessing the range of uncertainty to be much less than it really is with respect to your true state of knowledge
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Common biases and traps in judgment and probabilistic assessment • •
Illusion of control
Human beings are not endowed with The confirmation trap rational probabilistic thinking and optimal behaviour under uncertainty.
•
Overconfidence
•
•
Availability and Vividness
•
•
Anchoring Bias &
•
Intuition and Repetition Hindsight Foresight and the “curse of knowledge”and decisions
error => poor judgements • Knowing when enough The Law of Small Numbers => undesirable outcomes is enough
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Overconfidence Results: Large O&G Industry Sample Proportion of Participants
0.35 Expected Observed
0.3 0.25 0.2 0.15 0.1 0.05 0 0
1
2
3
4
5
6
7
8
9
10
Questions Correct /10 GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Does it Matter? Economic impact of overconfidence in O&G project • Field investment decision where the development plan depended on the reserve estimate. • Investigate the impact on value (NPV) of overconfidence in reserve parameters, eg Estimated PDF (Biased) “True”, unbiased PDF 12500
15000
17500
20000
22500 Area
Welsh, Begg & Bratvold (2007) SPE 110765 GeoMet 2013, Brisbane
P10 of biased (overconfident) PDF = P20 of unbiased PDF if the estimated one was 20% over confident (20OC) Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Economic Impact of Overconfidence in O&G project
E(NPV), $Million
400
300
Assessed (Overconfident) Value
200 True NPV
100
using EVs of inputs EV
0
-100 0%
5%
10%
15%
20%
25%
30%
Overconfidence GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Value of Flexibility to Exploit Upside Potential: Extra well slots to manage Volume uncertainty NPV High = 30% 60 kbd Platform
Med = 40%
EMV = $220
Low = 30%
$300 $250 $100
High = 30% 100 kbd Platform
Med = 40%
EMV = $260
Low = 30%
$600 $200 $0 Expand
High = 30%
$580
No action $270 Expand Flexible Platform
$180
Med = 40% No action
EMV = $303
Expand Low = 30% No action GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
$270 -$20 $70 Steve Begg
Value of Flexibility to Exploit Upside Potential: Extra space to manage resource uncertainty NPV High = 30% 60 kbd Mill
Med = 40%
EMV = $220
Low = 30%
$300 $250 $100
High = 30% 100 kbd Mill
Med = 40%
EMV = $260
Low = 30%
$600 $200 $0 Expand
High = 30% Flexible Development
No action $270 Expand
$180
Med = 40% No action
EMV = $303
Expand Low = 30% No action GeoMet 2013, Brisbane
$580
Carrasco Lecture: Uncertainty, decisions models & people
$270 -$20 $70 Steve Begg
Summary • The main purpose of technical work is assessment of uncertainty to aid decision-making - the focus should be ranking, not “best prediction”
• Modeling approach should be decision-driven - holistic models that focus on uncertainty & the complexities of the system – not the most accurate physics/geology/…
• Uncertainty is a function of what we know about a situation – its in our heads, not a “system” parameter - there is no single, “right” probability for an uncertain event - variability is not the same thing as uncertainty
• Accurate, unbiassed, uncertainty assessment is required to assess the value of decision alternatives GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Acknowledgements & Further Info • People – Prof. Reidar Bratvold, University of Stavanger, Norway – Dr. Matthew Welsh, University of Adelaide – Prof. Michael Lee, University of California, Irvine
• Papers (downloadable from www.onepetro.org) – SPE 71414: “Improving Investment Decisions with a Stochastic Integrated Asset Model”
– SPE 77509: “Would You Know a Good Decision if You Saw One?” – SPE 77586: “The value of Flexibility in Managing Uncertainty in Oil & Gas Investments” – SPE 96423 “Cognitive Biases in the Petroleum Industry: Impact and Remediation” GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
O&G Evidence of Bias (IPA data)
All 1000+ projects in the study
No projects
0% 0
50% 50
100% 100
150 150%
If the Forecasted production is the “Base Case”, we should have approximately as many projects producing more than expected as less than expected !!
200 200%
250 250%
300 300%
Basis for development sanction
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Gambling (probability = repeated outcomes) vs. “Real World” (probability = degree of belief) Uncertainty Quantification 2. Assign Probabilities to Outcomes
1. Identify Possible Outcomes All Identified
Some missed or unknowable
Known Distribution Type
Unknown Distribution Type
Some denied or ignored Known Parameters Games of Chance, Geostatistics? GeoMet 2013, Brisbane
Unknown Parameters Oil & Gas: Subjective
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Understanding Risk attitudes
50% Tails Yes
Outcome Heads
Play game 1
50%
No GeoMet 2013, Brisbane
(1) (1,000,000) (10,000) (100,000) (1,000) (100) (10) E=1 E=10,000 E=1,000,000 E=100,000 E=1,000 E=100 E=10
3 3,000,000 30,000 300,000 3,000 300 30
0 Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Understanding Risk attitudes
50% Tails Yes
Outcome Heads
Play game 2
50%
No GeoMet 2013, Brisbane
0 E=150,000 E=1.5 E=1,500,000 E=15,000 E=1,500 E=150 E=15
300,000 3 3,000,000 30,000 3,000 300 30
100,000 1 1,000,000 10,000 1,000 100 10 Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Risk Attitudes – Formal Definitions
– No preference between a sure option (P=1) and an uncertain option whose EV is the same (as the sure value)
• Risk-Averse – Prefer a sure option (P=1) whose value is less than or equal to the EV of an uncertain option – we put extra “value” on certainty
EV
Sure option
Sure- prefer this
Uncertain
EV
Risk-Seeking – Prefer an uncertain option whose EV is less than or equal to a sure option (P=1) – we put extra “value” on uncertainty
GeoMet 2013, Brisbane
Uncertain - prefer this
PDF
•
Uncertain Option
PDF
Risk-Neutral
PDF
•
Carrasco Lecture: Uncertainty, decisions models & people
EV
Sure
Steve Begg
Impact of Risk Attitudes
•
In both cases (risk-seeking and risk-aversion) the lower actual value option is preferred (chosen)!
•
Therefore both Risk-Aversion and Risk-Seeking attitudes lead to lower total long-run (multiple-decision) outcomes! – this is consistent with earlier statement that EV maximizes $ value – any other criterion loses $ value
•
E.g. assuming the EV lies between the P10 and P90 then using either of the following decision criteria – invest if P10>0 (very risk-averse), or – invest if P90>0 (very risk-seeking)
are both value-losing compared to using EV – invest if EV>0 (risk-neutral) GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Knowing When Enough is Enough
40%
• In our desire to reduce uncertainty, we often ask for too much information
% correct predictions
30%
Confidence does increase
20%
Accuracy does not increase
10%
5
10
20
Items of information available
GeoMet 2013, Brisbane
40
• We believe – mistakenly – that more information will increase accuracy • More information helps only to the extent we can use it intelligently
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Heuristics, Biases, Uncertainty & Decisions • Heuristics – simple rules of thumb and mental shortcuts
Human • Biases
beings are not endowed with rational probabilistic thinking and optimal – systematic errors that can result from the use of heuristics behaviour under uncertainty.
• With respect to uncertainty, the main enemy of good decision making is bias, not the uncertainty being too great – bias in central value (mean): eg “rose tinted glasses”
– bias in width of distribution: eg assessing the range of uncertainty to be much less than it really is with respect to your true state of knowledge
•
Bias & error => poor decisions and Our “mental wiring” is judgements just not good when it comes to uncertainty => undesirable outcomes – Intuition and “gut feel” often significantly wrong
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
From
http://www.freegifthome.co.uk/blog/best-optical-illusions/ - may not be original source
Estimate the gray % Using a scale of 0% (black) to 100% (white) estimate the % gray of squares A and B
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*Shepard’s Table
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*Shepard’s Table
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Perceptual Limitations – visual illusions are a metaphor for cognitive illusions • Awareness the illusion, by itself, does not Human of beings are not endowed with produce aprobabilistic more accurate thinking perception.and optimal rational
behaviour under uncertainty. • Illusions, therefore, can be extremely difficult to overcome.
Bias & error => poor decisions or judgements => More frequent undesirable outcomes GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Actual Performance
Decision-Making Under Uncertainty Asset/Project Evaluation
Portfolio Mgmt.
Psychology of Judgments & Decision-making Geology
Cognitive Science
Geophysics GeoMet 2013, Brisbane
Reservoir
Carrasco Lecture: Uncertainty, decisions models & people
Drilling Facilities Steve Begg
Main Research Themes 1. Psychological (cognitive science) aspects of judgment and decision-making – Elicitation of un-biased (accurate) expert knowledge for input to decisions – Matching appropriate decision tools and processes to decision types – Overcoming barriers–to-adoption
2. Development of models and processes for improved asset/project/portfolio economic valuation and decision-making under uncertainty – Incorporating “real option” thinking – valuing learning (incl information gathering) and flexibility. – Focus on accurate, rather than ‘precise’, holistic (systems) models GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
CIBP Publications: Phase 3 1Welsh,
M. & Navarro, D. (in press). Seeing is Believing: Priors, Trust and Base Rate Neglect. Organizational Behavior and Human Decision Processes. Accepted February 7th 2012.
2Willigers,
B.J.A., Begg, S., & Bratvold, R.B. (2011). Valuation of Swing Contracts by Least-Squares Monte Carlo Simulation” SPE Economics & Management, Vol 3, No 4, pp 215-225.
1&2Welsh,
M. & Begg, S. (in press). Personal ruin versus corporate profit: why individual risk attitudes lessen economic outcomes. APPEA Journal. Accepted 10th January 2012.
1Sykes,
M., Welsh, M. & Begg, S. (2011). SPE 146230 - Don't Drop the Anchor: Recognizing and Mitigating Human Factors When Making Assessment Judgments Under Uncertainty. Proceedings of the 2010 Society of Petroleum Engineers Annual Technical Conference and Exhibition. Austin, TX: SPE.
1Bruza,
B, Welsh, M, Navarro, D. & Begg, S. (2011). Does anchoring cause overconfidence only in experts? In L. Carlson, C. Hölscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 1947-1952). Austin, TX: Cognitive Science Society.
1Welsh,
M., Delfabbro, P., Burns, N. & Begg, S. (2011). Individual differences in anchoring: numerical ability, education and experience. In L. Carlson, C. Hölscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 3193-3198). Austin, TX: Cognitive Science Society.
1Welsh,
M, Navarro, D. & Begg, S. (2011). Number preference, precision and implicit confidence. In L. Carlson, C. Hölscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 1521-1526). Austin, TX: Cognitive Science Society.
1Welsh,
M., Alhakim, A., Ball, F., Dunstan, J. & Begg, S. (2011). Do personality traits affect decision making ability: can MBTI type predict biases? APPEA Journal, 51(1), pp 359-368.
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
CIBP Publications: Phase 3 2Willigers,
B.J.A., Begg, S., & Bratvold, R.B. (2010). Valuation of Swing Contracts by Least-Squares Monte Carlo Simulation” Paper #133044 in Proc SPE Asia Pacific Oil and Gas Conference, Brisbane.
1Bratvold,
R.B. and Begg, S.H. (2010) Making Good Decisions. Richardson, Texas, USA: Society of Petroleum Engineers.
2Bratvold,
R.B., Begg, S.H. & Rasheva, S. (2010) A New Approach to Uncertainty Quantification for Decision Making, Paper #130157, in Proc. SPE Hydrocarbon Economics and Evaluation Symposium, Dallas, TX, Mar 8-9
Welsh, M. (2010). Of parrots and parsimony: reconsidering Morgan's canon. Proceedings of the Cognitive Science Conference. 1Bruza,
B., Welsh, M., Navarro, D. & Begg, S. (2010). Effect of presentation order and question format on subjecive probability judgments. Proceedings of the Cognitive Science Conference.
1Mackie,
S., Begg, S., Smith, C. & Welsh, M. (2010). Human Decision-Making in the Oil and Gas Industry. Proceedings of the Society of Petroleum Engineers Asia Pacific Oil and Gas Conference.
1Welsh,
M. & Begg, S. (2010). Don't let it weigh you down: how to benefit from anchoring. Proceedings of the 2010 Society of Petroleum Engineers Annual Technical Conference and Exhibition.
1Welsh,
M., Rees, N., Ringwood, H. & Begg, S. (2010). The Planning Fallacy in Oil and Gas Decision Making. APPEA Journal, 50 (1), pp 389-401.
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
CIBP Publications: Phase 1 1Mackie,
S., Welsh, M. & Lee, M. (2006). An oil and gas decision-making taxonomy. Proceedings of the 2006 Asia Pacific Oil and Gas Conference of the Society of Petroleum Engineers. Richardson, TX: SPE.
1Welsh,
M., Begg, S. & Bratvold, R. (2006). Correcting common errors in probabilistic evaluations: efficacy of debiasing. Proceedings of the 82nd Annual Technical Conference and Exhibition of the Society of Petroleum Engineers. Richardson, TX: SPE.
2Al-Harthy,
M., Khurana, A., Begg, S., Bratvold, R., (2006) Sequential and Systems Approaches for Evaluating Investment Decisions: Influence of Functional Dependencies and Interactions. APPEA Journal, Vol 46, part 1, pp511-523
2Chapman,
T., Nettelbeck, T., Welsh, M. & Mills, V. (2006). Investigating the construct validity associated with microworld research: a comparison of performance under different management structures across expert and non-expert naturalistic decision-making groups. Australian Journal of Psychology, 58(1), pp. 40-47. Drop by to look at the
2Yao,
Y., Begg, S.H., Bratvold, R.B., Behrenbruch, P., van der Hoek, J. (2006) "A Case Study for Comparison of Different Real Option Approaches", Paper # 101031 in Proc. SPE Asia Pacific Oil and Gas Conference and Exhibition Adelaide, Australia, September, 2006.
2Bratvold,
R. and Begg, S.H. (2006) "Education for the Real World: Equipping Petroleum Engineers to Manage Uncertainty", Paper #103339, in Proc. SPE ATCE, San Antonio
1Welsh,
M., Bratvold, R. & Begg, S. (2005). Cognitive biases in the petroleum industry: impact and remediation. Proceedings of the 81st Annual Technical Conference and Exhibition of the Society of Petroleum Engineers. Richardson, TX: SPE.
2Bratvold,R.B.,
Laughton,D., Enloe,T., Borison,A., Begg, S.H., (2005) “A Critical Comparison of Real Option Valuation Methods: Assumptions, Applicability, Mechanics, and Recommendations”, Paper #97011, in Proc. SPE ATCE, Dallas
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
CIBP Publications: Phase 2 1Welsh,
M. & Begg, S. (2009). Repeated judgment elicitation: tapping the wisdom of crowds in individuals. Proceedings of the 85th Annual Technical Conference and Exhibition of the Society of Petroleum Engineers
1Welsh,
M., Lee, M & Begg, S. (2009). Repeated judgments in elicitation tasks: efficacy of the MOLE method. In N.A. Taatgen & H. van Rijn (Eds), Proceedings of the 31st Annual Conference of the Cognitive Science Society (pp1529-1534). Austin,TX:CognitiveScience Society.
1Bruza,
B., Welsh, M. & Navarro, D. (2008). Does Memory Mediate Susceptibility to Cognitive Biases? Implications of the Decision-by-Sampling Theory. In V. Sloutsky, B. Love, & K. McRae (Eds.) Proceedings of the 30th Annual Conference of the Cognitive Science Society.
1Heywood-Smith,
A., Welsh, M. & Begg, S. (2008). Cognitive Errors in Estimation: Does Anchoring Cause Overconfidence? Proceedings of the 84th Annual Technical Conference and Exhibition of the Society of Petroleum Engineers.
1&2Welsh,
M. & Begg, S. (2008). Modeling the Economic Impact of Individual and Corporate Risk Attitude. Proceedings of the 84th Annual Technical Conference and Exhibition of the Society of Petroleum Engineers.
1Welsh,
M., Lee, M. & Begg, S. (2008). More-Or-Less Elicitation (MOLE): Testing a heuristic elicitation method. In V. Sloutsky, B. Love, & K. McRae (Eds.) Proceedings of the 30th Annual Conference of the Cognitive Science Society.
2Begg,
S.H., and Bratvold, R.B. (2008) Systematic Prediction Errors in O&G Project and Portfolio Selection., Paper #116525, in Proc. SPE ATCE, Denver.
2Bratvold,
R. and Begg, S.H. (2008) I would rather be Vaguely Right than Precisely Wrong , AAPG Bulletin, Vol 92, No.10 (October 2008), pp. 1373-1392
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
CIBP Publications: Phase 2 2Cunningham,
P. and Begg, S.H., (2008) Using Value of Information to Determine Optimal Well Order in a Sequential Drilling Program , AAPG Bulletin, Vol 92, No.10 (October 2008), pp. 1393-1402
2Smalley,
P.C., Begg, S.H., Naylor, M., Johnsen, S. & Godi, A. (2008) Handling Risk and Uncertainty in Petroleum Exploration and Asset Management: An Overview , AAPG Bulletin, Vol 92, No.10 (October 2008), pp. 1251-1261
1&2Mackie,
S., Begg, S., Smith, C. & Welsh, M. (2008). "Real World" Decision-Making in the Upstream Oil and Gas Industry - Prescriptions for Improvement. APPEA Journal, 48(1), pp 329-343.
1&2Welsh,
M., Begg, S. & Bratvold, R. (2007). Modeling the economic impact of cognitive biases on oil and gas decisions. Proceedings of the 83rd Annual Technical Conference and Exhibition of the Society of Petroleum Engineers. Richardson, TX: SPE.
1Welsh,
M., Begg, S. & Bratvold, R. (2007). Efficacy of bias awareness in debiasing oil and gas judgments. Proceedings of the 29th Annual Conference of the Cognitive Science Society.
1Welsh,
M., & Navarro, D. (2007). Seeing is believing: priors, trust and base rate neglect. Proceedings of the 29th Annual Conference of the Cognitive Science Society.
1&2Mackie,
S., Begg, S., Smith, C. & Welsh, M. (2007). Decision type - a key to realizing the potential of decision-making under uncertainty. APPEA Journal, 22(1), pp. 307-317.
2Al-Harthy,
M., Begg, S., Bratvold, R., (2007) “Copulas: A New Technique to Model Dependence in Petroleum Decision Making”. Journal of Petroleum Science and Engineering, 57, pp195-208
2Begg,
S.H., and Smit, N. (2007) “Sensitivity of Project Economics to Uncertainty in Type and Parameters of Oil Price Models“, Paper #110812, in Proc. SPE ATCE, Anaheim
1Elliot,
T., Welsh, M., Nettelbeck, T. & Mills, V. (2007). Investigating Naturalistic Decision Making in a simulated micro-world: What questions should we ask? Behavior Research Methods, 39(4), pp 901-910.
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
CIBP Publications: Phase 1 Lee, M., Pincombe, B. & Welsh, M. (2005). An empirical evaluation of models of text document similarity. Proceedings of the 27th Annual Conference of the Cognitive Science Society. Mahwah, NJ: Erlbaum 1Welsh,
M., Begg, S., Bratvold, R. & Lee, M. (2004). Problems with the elicitation of uncertainty. Proceedings of the 80th Annual Technical Conference and Exhibition of the Society of Petroleum Engineers. Richardson, TX: SPE.
2Begg,
S.H. and Bratvold, R., (2004) “The value of flexibility”, Southern China Oil & Gas Journal, vol17, no.4, p59.
2Begg,
S.H., Bratvold, R., Campbell, J.C, (2004) “Abandonment Decisions and The Value of Flexibility “, Paper #91131, in Proc. SPE ATCE, Houston
2Laughton,
D., Bratvold, R., Begg, S.H., Campbell, J.C. (2004) “Development as the continuation of appraisal by other means “, Paper #90155, in Proc. SPE ATCE, Houston
2Lee,
M., O'Connor, T. & Welsh, M. (2004). Decision making on the full information secretary problem. Proceedings of the 26th Annual Conference of the Cognitive Science Society. Mahwah, NJ: Erlbaum.
1&2Begg,
S.H., Bratvold, R., Campbell, J.C. (2003) “Shrinks or Quants: Who will improve decisionmaking? “, Paper #84238, in Proc. SPE ATCE, Denver
2Bratvold,
R., Begg, S.H., Campbell, J.C. (2003) “Even Optimists should optimize“, Paper #84329, in Proc. SPE ATCE, Denver
2Begg,
S.H., Bratvold, R., Campbell, J.C, (2003) “Decision-making Under Uncertainty”, In Proc. 7th International Symposium on Reservoir Simulation, Baden-Baden
2Campbell,
J.C, Begg, S.H., Bratvold, R.B. (2003) “Portfolio Optimization: Living up to Expectations?“, Paper #82005, in Proc. SPE Hydrocarbon Economics and Evaluation Symposium, Dallas
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Outline •
Introductions, Questionnaire
•
Overview of Performance of O&G Industry Capital Investment Decisions
•
Underlying Concepts
•
Major Psychological Factors
•
Limits of Intuition in Complex & Uncertain Situations
•
What can we do… RISC Process
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Uncertainty Assessment Exercise 10%
Lower Limit (P10)
80% Chance
Upper Limit (P90)
10%
Question 1 Question 2 Question 3 etc
E.g. What is my height in cm?
•
172 cm Your goal is to set ranges such that they are
-
Narrow enough not to contain the actual value more than 8 out 10 times (80%) on average, or
-
Wide enough to contain the actual value 8 out of 10 times
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Anchoring Question Large Industry Sample • Question given to two groups. One group had a High anchor, other group had Low anchor • High “Were world proved oil reserves in 2003 greater or less than 1721 Billion Barrels?” Yes [ ] No [ ]
• Low “Were world proved oil reserves in 2003 greater or less than 574 Billion Barrels?” Yes [ ] No [ ]
• Both versions then asked “What is your best estimate of the world proved oil reserves in 2003?” ( GeoMet 2013, Brisbane
) Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Anchoring Results Large Industry Sample Mean Estimated World Proved Reserves 2003 +/- 1sd
3500 3000
Anchor
2500
Estimate
Common approach in E&P project 2000 1932 evaluation: 1722 1500 1000 “Let’s start with a base case and 682 574 500 build some scenarios around it.” then 0 Low
High Anchor Group
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Major Psychological Issues Relevant to Project Cost and Time Estimates and Outcomes •
Overconfidence, Optimism and Superiority Biases
•
Anchoring
•
Unpacking & the Planning Fallacy
•
Availability, Recency and Vividness
•
Hindsight Bias
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
REMOVE THIS SLIDE • I am not convinced about giving these definitions (next 3 slides) up front – liable to lead to unfruitful discussion to distinguish them before they have really learnt what they are • I also really struggled to make / modify the illustrative “quotes” to keep them pithy and descriptive of the bias/factor
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
RISC view of primary Psychological Factors affecting E&P projects Overconfidence: “There is an 90% chance the cost will be less than $10MM”; placing higher probabilities on events (or tighter probability distributions) than is warranted by our true state of knowledge. Due to this bias, actual outcomes will lie outside our ranges more frequently than expected.
Optimism: “There is only a 5% chance of a delay of more than 3 months”; assigning lower chances to the attainment of undesirable outcomes (and/or higher chance to desirable outcomes) than objective criteria, experience or logical analysis warrants. Due to this bias, outcomes will be systematically worse than expected.
Positive Illusions (Superiority Bias): “We can do it better than anyone else” causes people to overestimate, relative to others, their positive qualities, skills and abilities and to underestimate their negative qualities - leading them to believe that THEY are less at risk of experiencing a negative event compared to others who are doing the same thing. Also leads to the Illusion of Control - under attributing the role of chance, in decision outcomes GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
RISC view of primary Psychological Factors affecting E&P projects Anchoring: “I wish the time to build XXX was less than 4 months”; an initial piece of information, typically a number (perhaps irrelevant) causes people (often sub-consciously) to “centre” on that information and to not adjust sufficiently far away when considering other possibilities
Planning Fallacy: “If everything goes to plan we’ll be on-line in 2 years” a tendency for people and organizations to rely on best-case scenarios, leading to optimistic estimates of how long it will take to complete complex tasks, even when they have experience of similar tasks over-running (their own or others).
Packing/Unpacking Effect: “lets not make this too complex and go with a high-level breakdown of tasks” When broad tasks, or causes, are broken down (unpacked) into explicitlyidentified sub-components, the estimates of times, costs, %-contribution for each task, and thus the overall total for all tasks, is more accurate. Failure to unpack adequately is thought to be a contributor to the planning fallacy – “out of sight is out of mind” GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
RISC view of primary Psychological Factors affecting E&P projects Availability, Recency & Vividness: “I have an excel spreadsheet handy with cost estimates from our last project .....” causes individuals to over-weight the most easily accessed, remembered or recent information when assessing the likelihoods of the possible outcomes of future events
Hindsight bias: “I knew that was likely to happen, ...”; the inclination, in retrospect, to assign higher chances to outcomes that have already occurred than were assigned before the event took place – can apply to one’s own, or other peoples, estimates of the chance of the outcome occurring
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Major Psychological Issues Relevant to Project Cost and Time Estimates and Outcomes •
Overconfidence, Optimism and Superiority Biases
•
Anchoring
•
Unpacking & the Planning Fallacy
•
Availability, Recency and Vividness
•
Hindsight Bias
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Refresher: Interpreting PDFs & Percentiles The area under the PDF between any two points is the probability of X lying between those two points 0.10 10% of area
80% of area 10% of area
P10 GeoMet 2013, Brisbane
P90 Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Uncertainty Assessment Exercise 10%
Lower Limit (P10)
80% Chance
Upper Limit (P90)
10%
Actual
Question 1 Question 2 Question 3 Question 4 Question 5 Question 6 Question 7 Question 8 Question 9 Question 10 GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Probability Estimation Exercise
• Excel - Interactive
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*Assessing Uncertainty • Some conclusions from Capen study (3000+ sample): – People who are uncertain about answers to a question have almost no idea of the degree of their uncertainty. They cannot differentiate between a 30- and a 98-percent probability interval – The more people know about a subject, the more likely they are to construct a large probability interval (one that has a high chance of catching the truth), regardless of what kind of interval they have been asked to use. The converse seems to hold as well; the less known, the smaller the chance that the interval will surround the truth – People tend to be a lot prouder of their answers than they should be – Even when people have been told that probability ranges tend to be too small, they cannot bring themselves to get their ranges wide enough
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
* Experts & Over-confidence Concept
Design
Actual uncertainty
Construction
Commission
Perceived uncertainty
Max Concept select
Practical completion
Min Discovery
Detailed design
Procurement
Time GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
We’re not alone: Experts & Over-confidence
• Heavier-than-air flying machines are impossible –
Lord Kelvin, British mathematician, physicist, and president of the British Royal Society, spoken in 1895
• A severe depression like that of 1920-21 is outside the range of probability –
GeoMet 2013, Brisbane
Harvard Economic Society, Weekly Letter, November 16, 1929 Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
We’re not alone: Experts & Over-confidence
• That idea is so damned nonsensical and impossible that I'm willing to stand on the bridge of a battleship while that nitwit tries to hit it from the air –
GeoMet 2013, Brisbane
Newton Baker, U.S. secretary of war in 1921, reacting to the claim of Billy Mitchell (later Brigadier General Mitchell) that airplanes could sink battleships by dropping bombs on them
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
We’re not alone: Experts & Over-confidence
• I think there is a world market for about five computers –
Thomas J. Watson, chairman of IBM, 1943
• They couldn't hit an elephant at this dist… –
GeoMet 2013, Brisbane
General John B. Sedgwick, Union Army Civil War officer's last words, uttered during the Battle of Spotsylvania, 1864 Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
We’re not alone: US DoE Price Forecasts Trends Predicted Beginning From the Actual Price of Year Listed
Dollars per Barrel
120
1982
100
1981
80
1984 1983
60
1985 1986
1987
40 1991 20 0 1975
Actual 1995 1980
1985
1990
1995
2000
2005
Year after U.S. Department of Energy, 1998 GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
We’re not alone: Yr 2000 Price Forecasts
35
$/BBL (1996 Dollars)
30
Widely Divergent Forecasts Make Planning Difficult
IEA DOE High Mobil
25
DRI 20
DOE Base Nat. Res. Canada
15
Nat. West Sec.
10 2000
2005
2010
2015
2020
Pet. Econ. Ltd.
Year DOE Low U.S. Department of Energy, 1998 GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Does it matter? Overconfidence in the Stock Market • Based on a study of monthly positions and trading records of 88000 investors over a 10 year period and $2 million common stock trades (Barber & Odean 2000).
• In contrast to a buy-and-hold strategy, the average investor traded 75% of their investment in any given year. • The investors traded frequently because they thought they could beat the market
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Overconfidence in the Stock Market: You Are What You Trade • The average investor earned a return of 16.4% during the booming market. • The overall market return for this same period was 17.9%. • The 20% of accounts (more than 12000) that had the highest turnover rates earned a return of just 11.4%. • Overconfident investors under diversify.
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Overconfidence in the Stock Market Boys will be Boys • The data (Barber & Odean 2001) show that women performed better than men – Not because they were better at picking stocks but because …
• Men tend to be more overconfident than women. – Men trade 45 percent more actively than women. – For single men and women, the difference is 67%.
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Calibration • Calibration: the degree to which estimated accuracy (Confidence) matches actual accuracy 1.0 0.9
Actual Accuracy
0.8
Perfect Calibration
Under Confidence
0.7 0.6 0.5
Typical O&G result at 80% confidence
0.4 0.3
Over Confidence
0.2 0.1 0.0 0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Estimated Accuracy (Confidence) GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Expert Calibration: Physicians Data: Physicians, after completing history and physical examination, estimated the probability that patients had pneumonia (Source: Christensen-Szalanski & Bushyhead, 1981) 1.0
% Radiographically assigned pneumonia
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Subjective Probability of Pneumonia GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
What Can We Do? Expert Calibration: US Weather Forecasters 1.0
38
9
Source: Russo and Schoemaker
0.9 0.8
Actual Fraction
159
82
0.7 0.6
147 203
0.5 0.4
172
0.3
257
0.2 589
146 0.1
575
161
282
0.0 0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Forecast Probability of Rain
• Why are physicians lousy and weather forecasters great?
GeoMet 2013, Brisbane
A significant reason is because they get frequent, immediate, and accurate feedback Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
In the resources industries pressures towards overconfidence are rife • Experts may feel pressure (motivational or cognitive) to demonstrate their expertise relative to peers or competitors and therefore place overconfident bounds on uncertain quantities – Narrower bounds are used to imply “I(we) know more than you”
• Managers may create a climate that discourages a true assessment of uncertainty: – “you are paid to know, not to not know …” – “you are just covering your …” – “why can’t you just tell me the answer” – “I need a single number to make a decision”
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Optimism • A bias towards – assigning higher chances to the attainment of desirable outcomes – assigning lower chances to the attainment of undesirable outcomes
than objective criteria, experience or logical analysis warrants. • Optimism can be the result of – Dispositional: a personality trait – Situational: e.g. a motivational bias, caused by an incentive to be optimistic rather than realistic
• This bias raises the odds that the projects chosen for investment will be those with the most optimistic forecasts – and hence the highest probability of disappointment – outcomes will be systematically worse than expected (worse being context dependent – ie higher costs, lower production) GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Overconfidence v Optimism: Impact on Assessed Probabilities, Pictorially
“True” Uncertainty
Over-Confidence
- assigning too-narrow a PDF -
compared to your true state of knowledge driven by mis-assessing our level of knowledge
GeoMet 2013, Brisbane
Optimism
- assigning higher chance to -
more desirable outcomes (eg “lower” valued outcomes, such as costs, are better) driven by our personality or by preferences for some of the outcomes
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Overconfidence & Optimism!
Over Confident and Optimistic “True” Uncertainty
-
GeoMet 2013, Brisbane
in this case “higher” outcomes are more desirable (eg production rates)
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Optimism and Organizational Pressure: Motivational perspective • Every company has limited investment funds and time to devote to new projects. – Competition for this time and money is intense, as individuals and units present their own proposals as being the most attractive for investment.
• The selection process often – favours the most communicative and articulate – not necessarily the most knowledgeable – discourages “realism” (unbiasedness), encourages exaggeration – causes bearers of bad news, or people who point out problems, to be labelled as “not a team player” - we do “shoot the messenger”!
GeoMet 2013, Brisbane
Big incentives to accentuate the positive in project forecasts. Ability of the organization to think critically is under-minded Optimistic individual views are self-reinforcing and unrealistic views of the future are validated by the group Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Using outcomes of multiple estimates to check for unbiasedness Unbiased 20% % outcomes lying within estimated range
P0
P20
P40
P60
P100
Biased
% outcomes lying within 20% estimated range !!
!! P0
GeoMet 2013, Brisbane
P80
P20
P40
P60
Carrasco Lecture: Uncertainty, decisions models & people
P80
P100 Steve Begg
Questions • Compared to your peers, on a scale of 1 to 10, where - 1 is poor - 5 is average - 10 is very good
how good a driver are you? how good a decision-maker are you? how good a probability estimator are you? how good is your intuition?
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Positive Illusions (Superiority Bias)
•
People/organizations overestimate their positive qualities, skills and abilities (and underestimate their negative qualities) relative to other people/organizations
•
This can lead them to believe that THEY are less at risk of experiencing a negative event (e.g. cost/time overrun) compared to OTHERS in the same situation.
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*Lake Wobegon effect: “… and all the children are above average” – Garrison Keillor
•
82% of people say they are in the top 30% of safe drivers;
•
86% of MBA students say they are better looking than their classmates;
•
68% of lawyers in civil cases believe that their side will prevail;
•
Doctors consistently overestimate their ability to detect certain diseases;
•
81% of new business owners think their business has at least 70% chance of success, but only 39% think that any business like theirs would be likely to succeed.
Source: Russo & Schoemaker GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Lake Wobegon effect: “… and all the children are above average” – Garrison Keillor
• • • • •
82% of people say they are in the top 30% of safe drivers;
•
86% of MBA students say they are better looking than When assessing their position in a their classmates;
distribution of peers on almost any
68% of lawyers in civil cases believe that their side will positive trait, 90% of people say they prevail;
are inconsistently the top half. Doctors overestimate their ability to detect certain diseases; 81% of new business owners think their business has at least 70% chance of success, but only 39% think that any business like theirs would be likely to succeed.
Source: Russo & Schoemaker GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*Illusion of Control
Doing
Skill
• implementation and other factors under your control
Presumed Cause Success
Deciding • the thinking and decision process
Outcome Failure Presumed Cause Chance • uncontrollable factors, luck
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Illusion of Control
Doing
•
• implementation other factors causes people to Illusion of controland frequently Presumed Cause under your control repeat actions that in the past were followed by
Skill
Success
success.
•• the thinking is true even if there’s no reason to believe
Deciding This
the actions did anything to cause the Outcome success.
•
and decision process
Only by realistically assessing the role of chance in successes can you learn which of your actionsFailure Cause you should repeat and which couldPresumed be improved. Chance
• uncontrollable factors, luck GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Illusion of Control: • People often (knowingly and unknowingly) take credit for positive outcomes and attribute negative outcomes to external factors, no matter what their true cause. • Study of letters to shareholders: – Executives tend to attribute favourable outcomes to factors under their control, and – Unfavourable outcomes were more likely to be attributed to uncontrollable external events such as weather or inflation.
“Victory has a thousand fathers; defeat is an orphan.” -the Duke of Wellington GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Thinking about “Luck” • By definition, luck (or unluck) is something over which you do not have control, therefore: – People are not inherently lucky or unlucky. – Lucky or unlucky things happen to people.
• People cannot “create their own luck” – but they can plan to exploit good luck when it happens, and minimize the impact of bad luck when that happens – “Plan”: by creating an environment, contingencies or opportunities to respond to events, rather than living with fate.
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Summary of Overconfidence, Optimism and Positive Illusions (Superiority) Overconfidence: “There is an 90% chance the cost will be less than $10MM”; placing higher probabilities on events (or tighter probability distributions) than is warranted by our true state of knowledge. Due to this bias, actual outcomes will lie outside our ranges more frequently than expected.
Optimism: “There is only a 5% chance of a delay of more than 3 months”; assigning lower chances to the attainment of undesirable outcomes (and/or higher chance to desirable outcomes) than objective criteria, experience or logical analysis warrants. Due to this bias, outcomes will be systematically worse than expected.
Positive Illusions (Superiority Bias): “We can do it better than anyone else” causes people to overestimate, relative to others, their positive qualities, skills and abilities and to underestimate their negative qualities - leading them to believe that THEY are less at risk of experiencing a negative event compared to others who are doing the same thing. Also leads to the Illusion of Control - under attributing the role of chance, in decision outcomes GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Effect of Overconfidence, Optimism and Positive Illusions (Superiority) •
Collectively these bias’s affect the estimates and uncertainty ranges that are generated and used as part of Project cost and schedule workflows
•
Ultimately the effect is to provides estimates that are too low and uncertainty ranges that are too narrow
•
No-one is immune to these psychological issues and unless people are made aware of them (and even then) they will always skew our work
•
The culture and incentive systems in organisations (and Project teams) actively drives overconfidence and optimism in the people within
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Mitigating Overconfidence, Optimism and Positive Illusions (Superiority) • Be aware of them (not just now, continually remind yourself) • Ask yourself why you might be wrong (as opposed to right) – actively think of all the reasons why the P10 might be smaller and the P90 larger. • Get Feedback – Privately record lots of predictions (probability for discrete events; P10P90 ranges for continuous variables) of things you will soon get to know the answer to – and record the outcomes as well. – Enhance your motivation to be well-calibrated by making small bets with friends, colleagues and family on the outcomes
• Don’t limit your assessment of uncertainty to finding data (of the event in question) and fitting a distribution to them – Unless the data are very numerous and truly repeat outcomes of the same “event” of interest GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Mitigating Overconfidence, Optimism and Positive Illusions (Superiority) • Take an “outside view” – look at the performance of other people/teams/companies, or how you think others would perform if doing your projects
• Managers/Supervisors – Create a climate where honesty about uncertainty is encouraged – Reward for quality of predictive process – not actual outcome – Ask people to justify why their ranges are so narrow (half the P10, double the P90)
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Major Psychological Issues Relevant to Project Cost and Time Estimates and Outcomes •
Overconfidence, Optimism and Superiority Biases
•
Anchoring
•
Unpacking & the Planning Fallacy
•
Availability, Recency and Vividness
•
Hindsight Bias
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Anchoring - Subtle changes in wording of a question can significantly impact responses Question Do you get headaches frequently? How often do you get them? Do you get headaches occasionally? How often do you get them?
Question
Outcome 2.2 / week 0.7 / week
Outcome
How long was the movie?
130 min
How short was the movie?
100 min
Question
Outcome
How tall was the basketball player?
79 inches
How short was the basketball player?
69 inches
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Anchoring - Subtle changes in wording of a question can significantly impact responses Question
Outcome
How wide are the channels? How narrow are the channels?
Question
Outcome
How big is the fault throw? How small is the fault throw? GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Anchoring and Adjustment
•
Participants watched a one-minute film that include a foursecond multiple car crash.
•
The participants were then divided into 5 groups who were asked: – “About how fast were the cars going when they smashed into each other?” – “About how fast were the cars going when they collided with each other?” – “About how fast were the cars going when they bumped into each other?” – “About how fast were the cars going when they hit each other?” – “About how fast were the cars going when they contacted each other?”
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Anchoring and Adjustment “About how fast were the cars going when they ??? into each other?”
Ref.: Loftus and Palmer
GeoMet 2013, Brisbane
Descriptor
Mean Speed
Smashed
40.8
Collided
39.3
Bumped
38.1
Hit
34.0
Contacted
31.8 Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Anchoring Question • Group A?
• Group B?
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Anchoring & Adjustment • Describes a heuristic commonly used by people when estimating values. • Use any given number/statistic/fact as a starting point (anchor) that from then on dominates the thinking process. – adjust away from there to reach estimate – generally people adjust too little so their estimates cluster near the anchor
• Random anchors can have just as large effects as credible anchors. – Quattrone et al (1984) asked whether the average temperature in San Francisco was greater or less than 558º and still found people anchoring on this value
• Subtle changes in wording of a question can have significant impact on how people respond. GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Anchoring - Implications • Referendums and opinion polls. • Jurors: – “Strike that comment”
• Independence of second opinions. • Skilled negotiators often start by setting a suitable anchor. • Used car salesmen and real estate brokers. • Resource industry managers? • Common approach in project evaluation: – “Let’s start with a base case and then build some scenarios around it.” GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Summary of Anchoring Anchoring: “I wish the time to build XXX was less than 4 months”; an initial piece of information, typically a number (perhaps irrelevant) causes people (often sub-consciously) to “centre” on that information and to not adjust sufficiently far away when considering other possibilities
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Anchoring: Effect and Mitigation • Effect
-
This bias can affect the initial estimates (and then uncertainty ranges) that are generated and used as part of Project cost and schedule workflows
-
A bias that is used extensively when negotiating with someone who is aware of the effect
-
Difficult to overcome even when aware of the effect Framing of questions is extremely important to reduce the effect
• Mitigation
-
Be aware of it – not just now, remind yourself Deliberately use multiple Anchors NEED MORE
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Major Psychological Issues Relevant to Project Cost and Time Estimates and Outcomes •
Overconfidence, Optimism and Superiority Biases
•
Anchoring
•
Unpacking & the Planning Fallacy
•
Availability, Recency and Vividness
•
Hindsight Bias
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
The Planning Fallacy • The tendency to underestimate completion times of projects because planners construct a single mental scenario, comprised of broad stages, in which most things go according to plan, – DESPITE knowing that similar projects (whether our own! or others) have gone over time & budget
• The “because” above is the strict/narrow definition of the Planning Fallacy and is thought to be due to – Failing to distinguish between “best guess” and “best case” scenarios – Failure to unpack tasks, especially “other problems”, or ignoring them completely The whole workshop could be considered to be about a broader definition of the “planning fallacy” (lower case) and thus a result of all the psychological factors we are considering – Overconfidence; Optimism (as a personality trait or motivational bias); Illusion of Control and Superiority bias; …… GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
The Planning Fallacy & Unpacking Effect • When asked how long a 1-hour lecture takes to prepare, lecturers estimate around 3 hours
• When asked to estimate how long each task involved in preparing the same lecture takes, the sum of these task times is significantly greater than 3 hours
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Unpacking Questions • 1st Unpacking Question (Packed) – “What % of world proved oil reserves are in the following areas: Saudi Arabia, Iraq, Oman and All Others?”
• 1st Unpacking Question (Unpacked) – “What % of world proved oil reserves are in the following areas: Saudi Arabia, Iraq, Oman Venezuela, USA, China, Russia, Nigeria and All Others?”
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Unpacking Question 1 Results
70 60 (2003) +/- 1SD
Estimated %of World Proved Reserves
80
50 True: Packed
40
Unpacked 30 20 10 0 Saudi Arabia
Iraq
Oman
All Others
Re gion
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Unpacking example: estimating time to drill a well • Four groups were asked to estimate completion times, in hours, for a real-world drilling scenario – 3rd yr ASP Pet. Eng. undergraduates (no decision-making training) – 4th yr ASP Pet. Eng. undergraduates (some decision-making training) – “Conversion” Masters of Pet. Eng. (little Pet. Eng. Knowledge) – Industry petroleum engineers (with average 10 yrs experience)
• Approximately half were given a Packed version of the scenario which consisted of 4 components – Drilling, Tripping, Rigging and All associated problems
and the rest given an version where “All associated problems” was Unpacked into 6 explicit possibilities – Mud conditioning; Well-control operation; Fishing operations; Severe weather; Rig repair; Logistics delays GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Unpacking example: estimating time to drill a well ‘An oilfield is located offshore Louisiana in about 350 ft of water. The field contains four wells that are currently producing 1,000 bbl of oil and 18 MMSCF of gas per day. Production is declining and it has been decided to drill an infill well in order to improve production. The well is required to be drilled to 10,000 ft drilling overbalance with mud. For this example, we will assume the well can be drilled straight to 10,000 ft without the need for staged casing. The rig is already in place and ready to commence drilling. From other well data, the stratigraphy can be assumed to be reasonably homogeneous and consolidated. The well needs to be ready up to the point where casing could then be run if required.’ Bourgoyne et al (1986)
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Estimated problem hours, mean and 95% CI
Unpacking Results: Number of hours of drilling problems
Packed Unpacked
Group GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Unpacking Effect • Out of sight is out of mind – Explicitly stating an option makes it available and therefore increases its estimated likelihood – Unpacking a general category into specific subcategories therefore increases the total likelihood assigned to that category despite the two being logically equivalent
• The Planning Fallacy – A specific instance of the unpacking effect – Refers to the tendency of people to underestimate completion times for complex tasks
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Summary of Planning Fallacy & Unpacking Effect Planning Fallacy: “If everything goes to plan we’ll be on-line in 2 years” a tendency for people and organizations to rely on best-case scenarios, leading to optimistic estimates of how long it will take to complete complex tasks, even when they have experience of similar tasks over-running (their own or others).
Packing/Unpacking Effect: “lets not make this too complex and go with a high-level breakdown of tasks” When broad tasks, or causes, are broken down (unpacked) into explicitlyidentified sub-components, the estimates of times, costs, %-contribution for each task, and thus the overall total for all tasks, is more accurate.
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Mitigating Planning Fallacy & Unpacking Effect • If you insist on using single-point estimates – make a “best guess” estimate rather than a “best case” estimate • Unpack general categories (such as “all other problems”) by just listing their sub-components – this will improve the estimate of the general category, even if you do not estimate each of the subcomponents
• Take an “outside view” – look at the performance of other people/teams/companies, or how you think others would perform if doing your projects
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Major Psychological Issues Relevant to Project Cost and Time Estimates and Outcomes •
Overconfidence, Optimism and Superiority Biases
•
Anchoring
•
Unpacking & the Planning Fallacy
•
Availability, Recency and Vividness
•
Hindsight Bias
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*Availability and Vividness • Are there more words in English that have the letter ‘k’ as their first letter (e.g., kill) or as their third letter (e.g., ink)? a) first letter b) third letter
• Experimental results (Tversky and Kahneman): – 2/3 of people asked thought that words with the letter k in the first position were more probable
• Reality: – There are approximately twice as many words with k in the third position as there are words that begin with a k. GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*Availability and Vividness •
Which of the following is more likely to kill someone in the USA? a) Shark Attack? b) Falling airplane parts? – Falling airplane parts 30 times more likely than shark attack in the US (cited in Plous, 1993)
•
Which of the following caused more deaths in Australia in 2003 a) Renal Failure? b) Car and other transport Accidents? – Renal failure - 15000 compared to 2100 for transport accidents (Australian Bureau Statistics, 2003)
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Availability and Vividness Bias – Seeing What We Believe • The tendency people have to base estimates of frequencies (probabilities) on the most readily available, recent and vivid information they can remember – how many events of a particular type are available to memory – more available events are judged more likely
• Memory is limited to 7 “chunks” GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Bias from the Availability Heuristic & Saliency Heuristic • This heuristic leads to bias where it is easier to recall events for reasons other than their frequency, e.g. news coverage – Shark attacks provoke a media feeding frenzy making them easier to recall – Car accidents often make the news over long weekends with multiple deaths whereas renal failure only ever affects one person at a time and gets no coverage
• Another contaminating factor in our ability to use the availability heuristic accurately is the salience of events, e.g. how relevant they are to us personally – People whose houses have burnt down inflate the likelihood of house fires when asked – Knowing someone who has died of a particular cancer tends to cause inflation of estimates of its occurrence
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Bias from the Availability Heuristic & Saliency • Tversky and Kahneman (1974) – “decision makers assess the frequency of a class or the probability of an event by the ease with which instances or occurrences can be brought to mind.”
• Managers conducting performance appraisals: – Working from memory, vivid instances of an employee’s behaviour (either positive or negative) will be most easily recalled from memory and will appear more numerous than more commonplace instances. – Managers give more weight to performance during the three months prior to the evaluation than to the previous nine months of the evaluation period. GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Bias from the Availability Heuristic & Saliency • Engineers who are evaluating durations for potential project tasks. • Commonly done by comparing the tasks with tasks that have been done previously and where, as a consequence, the durations are known. • Potential biases induced with this strategy are – previous personal experience with particular risks can be overweighted – no comparison is made with other risks that have not been experienced but that will potentially effect the tasks GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Summary of Availability, Recency and Vividness Availability, Recency & Vividness: “I have an excel spreadsheet handy with cost estimates from our last project .....” causes individuals to over-weight the most easily accessed, remembered or recent information when assessing the likelihoods of the possible outcomes of future events
• Difficult to overcome due to the amount of information that can be contained in the memory • Highlights that memory can not be relied on when making estimates • When recalling from memory, write down a list and try to think of the most common occurrences (not the most vivid or recent) • Search for the most pertinent data, not the most available GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Major Psychological Issues Relevant to Project Cost and Time Estimates and Outcomes •
Overconfidence, Optimism and Superiority Biases
•
Anchoring
•
Unpacking & the Planning Fallacy
•
Availability, Recency and Vividness
•
Hindsight Bias
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Hindsight Bias: Remembered Probabilities of Once-Future Things • Prior to President Nixon’s trip to China and Russia in 1972, students were asked to consider 15 possible outcomes: – “the US will establish a permanent diplomatic mission in Peking, but not grant diplomatic recognition,” – “Nixon will meet Mao Tse-tung at least once,” – “Nixon will meet Soviet demonstrators,” – …
• The students assigned probabilities to each outcome.
Ref: Fischhoff and Beyth (1975)
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*Hindsight Bias: The “We Knew it All Along” Phenomenon • After the trip the students were asked in hindsight to assess the likelihood of these various outcomes and also asked to remember or reconstruct their original probabilities.
• Two weeks between pre- and post-trip – 67% thought their original estimates were closer to the truth then they really were.
• Four to eight months between pre- and post – 84% thought they had predicted the outcome.
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Hindsight Bias: The “We Knew it All Along” Phenomenon • After the trip the students were asked in hindsight to “people even misremember their own assess the likelihood of these various outcomes and also predictions soreconstruct as to exaggerate asked to remember or their original in probabilities. hindsight what they knew in foresight” Fischhoff and Beyth
• Two weeks between pre- and post-trip
•
– 67% thought their original estimates were closer to the truth How will this bias impact our ability to, then they really were. in hindsight, judge the quality of our predictons? Four to eight months between pre- and post – 84% thought they had predicted the outcome.
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Summary Hindsight Bias Hindsight bias: “I knew that was likely to happen, ...”; the inclination, in retrospect, to assign higher chances to outcomes that have already occurred than were assigned before the event took place – can apply to one’s own, or other peoples, estimates of the chance of the outcome occurring
• Makes learning from previous estimates much more difficult • Highlights that memory can not be relied on when doing “lessons learnt” workshops • Records of previous predictions/estimates, and reasons for them, should be made at the time of the prediction/estimate and checked against actual performance to help with calibrating the mind
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Outline •
Introductions, Questionnaire
•
Overview of Performance of O&G Industry Capital Investment Decisions
•
Underlying Concepts
•
Major Psychological Factors
•
Limits of Intuition in Complex & Uncertain Situations
•
–
Uncertainty Propagation
–
Updating Estimates with New Information
–
Complexity
What can we do… RISC Process
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Estimating Odds
• Gilbert Video (2005G) Pt 1 – Odds 00.00-8:19
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Judging likelihoods of events • Linda is a 31 years old, single, outspoken and very bright. She majored in philosophy. As a student she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations. Which is the more likely alternative? a) Linda is a bank teller b) Linda is a bank teller and active in the feminist movement. Answer:______
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*Probability Rules: Multiplication Rule for Independent Events When event A and Event B can occur together Not A or B
A
Joint probability = P (A and B)
B
• Multiplication Rule for independent events P(A and B) = P (A) * P(B)
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Discussion of Linda Question
•
Nearly 90% of respondents choose the second alternative (bank teller and active in the feminist movement), even though this is logically incorrect bank tellers
feminists
feminist bank tellers
Junctions (“ands”) areTeller always likely P(Bank Teller) > P(Bank ANDless Feminist) than stand-alone statements. GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Cognitive Illusions
•
The description of Linda is more representative of a feminist bank teller so people, wrongly, conclude it is more likely that she is a feminist and a bank teller
•
Kahneman & Tversky (1982) – “As the amount of detail in a scenario increases, its probability can only decrease steadily, but its representativeness and hence its apparent likelihood may increase.” – “The reliance on representativeness, we believe, is a primary reason for the unwarranted appeal of detailed scenarios and the illusory sense of insight that such constructions often provide.”
•
Implications: consider a “rich” description of a reservoir depositional environment
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
High production days •
Daily production has been tracked for two oil fields for a year and the average production computed for each –
The first oil field has 45 wells and the second 15.
•
The number of days when production was above average for 60% or more of the wells in a field has been calculated.
•
Which field do you think would have recorded more such days over the course of a year? a) The 45-well field. b) The 15-well field. c) Approximately the same (within 5%).
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Insensitivity to Sample Size Results – Industry Respondees 70
No of Participants
60 50 40 30 20 10 0 Larger
Smaller
Equal
Answer
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Insensitivity to Sample Size • This question requires intuitive judgement as no data for calculation is available • Central Limit Theorem: – The smaller the sample, the more likely it is to be a deviant sample -> one with a mean and standard deviation significantly different from those of the population from which the sample was drawn – Answer b, the 15-well field, is the most likely
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Insensitivity to Sample Size • People tend to ignore sample size they are dealing with when drawing conclusions from it. • This is not consistent with the Central Limit Theorem which describes the changing characteristics of samples according to their size. • Example – which is more likely: – Getting 6 heads in 10 flips of a coin, or – Getting 6000 heads in 10000 flips of a coin.
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Sequences - Representivity Heuristic: The “Law of Small Numbers” •
I have just tossed a fair coin 7 times. You have not seen the result
•
You are invited to play a betting game to guess which of the three sequences below is the one I actually observed.
•
Which sequence would you bet on
•
Using multiplicative rule for independent events P(A&B&C&D.) = P(A)*P(B)*P(C)*P(D) ….
•
a) HHHHTTT
P = (1/2)7 = 1/128
b) THHTHTT
P = (1/2)7 = 1/128
c) TTTTTTT
P = (1/2)7 = 1/128
ALL sequences have the SAME probability and are thus EQUALLY likely (or equally rare!)
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
The “Law of Small Numbers” (Tversky & Kahneman) • We expect to see the same behaviour in small sequences that we would observe in large sequences – the mathematical “Law of Large Numbers” informs us of behaviours that are approximately true for large sequences, and rigorously true for sequences near to infinity
• Some sequences are seen as more balanced or more “typical” and are thus thought to be more probable. Typicality is mistaken for probability. – with the result that we over-estimate probability
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Sequences of Events, 2 • You are playing a game with a fair coin: – Heads you win $10, Tails you loose $10 (EV=$0)
• You have played 10 times so far and have had a lucky streak of 8 wins and 2 losses, so you are up $60 – What is your expected value if you play a total of 1000 times
• You have played 10 times so far and have had an unlucky streak of, 3 wins and 7 losses, so your cumulative position is down $40 – What is your expected outcome if you play a total of 1000 times
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*Sequences of Events are not self-correcting : The “Gamblers Fallacy” • You are playing a game with a fair coin: – Heads you win $10, Tails you loose $10 (EV=$0)
• You have played 10 times so far and have had a lucky streak of 8 wins and 2 losses, so you are up $60 – What is your expected value if you play a total of 1000 times
– $60
• You have played 10 times so far and have had an unlucky streak of, 3 wins and 7 losses, so your cumulative position is down $40 – What is your expected outcome if you play a total of 1000 times
– $-40 GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Sequences of Events are not self-correcting : The “Gamblers Fallacy” • After holding bad cards on ten hands of poker, the poker player believes he’s “due” for a good hand.
• Tversky and Kahneman:
• After–winning in the viewed lottery, as a woman “Chance$10,000 is commonly a selfchanges her regular lotteryinnumber after all, how correcting process which a– deviation likely is it that the same number will come up twice? in one direction induces a deviation in • The “hot thehand” opposite direction to restore the – If your favourite player made his last fourare shots, equilibrium. Inhas fact, deviations notis the probability of making next shot higher, lower, or the corrected as the a chance process unfolds, same as the probability of his making a shot without the theyfour arehits? merely diluted.” preceding • After 6 successive failures in an exploration play with a 30% chance of success “the next well is bound to be a discovery” GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
The Law of Small Numbers – We are not consistently inconsistent. • The Law of Small Numbers is ubiquitous in the world of business decision-makers, where it lends unfounded credibility to the claims of those who have been successful for a few years in a row. • An employee who does well several years in a row is surprised if performance is thought to be meanreverting. GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Probability Concepts: Multiplication Rule • The case of two dependent events P(A and B) = P (A) * P(B|A) can be generalized for many events to P(A and B and C..) = P (A) * P(B|A) * P(C|BA)… • If events are independent P(A and B) = P (A) * P(B) P(A | B) = P (A)
and
P(B | A) = P (B)
which can be generalized for many events to P(A and B and C…) = P (A) * P(B) * P(C) … GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Using P50s to assess completion times • Suppose a project is made up of five, parallel, independent tasks, all of which must be completed for the project to be completed. • You have estimated the P50 time for completion of each task to be 6 weeks. • What is the chance that the project will be completed within 6 weeks? • By definition, the probability of each task being completed on or before the P50 time (6 weeks) is 50%. Since the tasks are independent, then, P(All 5 completed by 6 weeks) = 0.55 ~ 3% GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Reliability of Predictors • Historical estimates suggest one in every 1000 blow-out preventers (BOPs) has serious cracks. • Suppose x-ray analysis is a very good, but not perfect, detector of these cracks. – If a BOP has cracks, x-rays will correctly say it has them 99% of the time – If a BOP does not have cracks, x-rays will wrongly say that it has them 2% of the time
• A BOP has been x-rayed at random and the result was positive! – What is your intuitive estimate of the chance that is has cracks?
Estimated Answer GeoMet 2013, Brisbane
%
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Use Probability Rules: in this case, Bayes’ Rule • Given a collection of n mutually exclusive and collectively exhaustive events B1, B2, …. Bn and another event A P(Bi|A) =
P (A|Bi) P(Bi) P (A)
=
n
i=1
P (A|Bi) P(Bi) P (A|Bi) P(Bi)
• The denominator (summation) is the total probability of A, that is, all the ways that A can occur and P(A|Bi) is called the likelihood function (likelihood) * (prior probability) posterior probability = {(likelihood) * (prior probability)}
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Use Probability Rules: in this case, Bayes’ Rule • Given a collection of n mutually exclusive and collectively exhaustive events B1, B2, …. Bn and another event A P(Bi|A) =
P (A|Bi) P(Bi) P (A)
=
n
i=1
P (A|Bi) P(Bi) P (A|Bi) P(Bi)
• The denominator (summation) is the total probability of A, that is, all the ways that A can occur and P(A|Bi) is called the likelihood function (likelihood) * (prior probability) posterior probability = {(likelihood) * (prior probability)}
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
What’s the relevance? We make extensive use of conditional probabilities, and their formulation as Bayes’ Rule, to answer questions such as How should prior probabilities be revised in the light of new information? - e.g. how should we revise initial CoS estimate as a drilling program progresses
What is the value of taking 3D Seismic? What is the value of adding extra slots to a platform in case OOIP is higher than thought? What is the value of coring a well?
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*Insensitivity to Base Rate • P(BOPC) = 1/1000 = 1x10-3 • P(test positive | BOPC) = 99% • P(test positive | no BOPC) = 2% • A = test positive, B = BOPC
P( A | B) P( B) P ( B | A) P( A | B) P( B) P( A | B ) P( B ) 3
0.99 10 P ( SHRD BOPC | test pos ) 3 0.99 10 0.02 0.999 0.047 4.7% GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Insensitivity to Base Rate Results 90 80
No of Participants
70 60 50 40 30 20 10 0 0
20
40
60
80
100
Estimated Posterior Probability
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Uncertainty Propagation Assessed Uncertainties
Propagated Uncertainty
Single Value
Decision Objective
Decision Criterion
eg NPV
E[NPV]
Input 1
Model
Input n
Equations (not brain!)
For non-linear models with uncertain inputs: the correct value of the decision criterion must be calculated by propagating the input uncertainty through to the decision objective uncertainty GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Uncertainty Propagation Assessed Uncertainties
Propagated Uncertainty
Single Value
Decision Objective
Decision Criterion
eg NPV
E[NPV]
Input 1
Model
Input n
Equations (not brain!)
For non-linear models with uncertain inputs: the correct value of the decision criterion must be calculated by propagating the input uncertainty through to the decision objective uncertainty GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Averages don’t always work - misplaced
•
Mean = 10
For non-linear processes,
Mean = 5
–
reservoir simulation – volumetrics with cut-offs – development alternatives
3
even if only a single, “best” estimate is required, we still need to use complete range of inputs - cannot use an average input
•
x
21
1
z
8
Model Y = X2 Z2
/
Simulation Result
Also, P10 (P90) results are NOT given by taking P10 (P90) inputs and running them through the model
GeoMet 2013, Brisbane
True Mean Result ~ 7.8 Y=4 0
Carrasco Lecture: Uncertainty, decisions models & people
Y
30 Steve Begg
The Flaw of Averages: • Unless model (f) is linear (“model” is a calculation) average of f( x,y,z,…. )
f( average(x), average(y), average(z),….) where x,y,z,… are uncertain quantities, and f
GeoMet 2013, Brisbane
is a calculation using x, y, z ....
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Flaw of Using Averages (after Savage)
NCF, Log, ...
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Insensitivity to Base Rate: Frequency view Applying the following reliabilities (CONDITIONAL probabilities) Real world P(T|R) Has Has Not Has 99% 2% Test says Has Not 1% 98% 100%
100%
to a sample of 100,000 people gives the following frequencies Real world
Test says
GeoMet 2013, Brisbane
Has Has Not
Has 99 1
Has Not 1998 97902
100
99,900
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Outline •
Introductions, Questionnaire
•
Overview of Performance of O&G Industry Capital Investment Decisions
•
Underlying Concepts
•
Major Psychological Factors
•
Limits of Intuition in Complex & Uncertain Situations
•
–
Uncertainty Propagation
–
Updating Estimates for New Information
–
Complexity
What can we do… RISC Process
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Change Detection and Blindness • People fail to notice large changes to visual arrays and scenes if they are briefly occluded – e.g., during saccades, eye blinks, or presentations using simple masks – the task is called ‘change detection’ – the inability to perceive change is called ‘change blindness’
• The previous simple example works for about a third to a half of people, and was created by Mondy and Coltheart (2002)
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Counting Passes: Basketball movie Basketball Movie
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
The Monty Hall Problem • US TV Game show in early 1970s: “Let’s Make a Deal!” – Show host Monty Hall
• Later discussed in newspaper column by Marilyn vos Savant – Created an enormous response – More than 10000 letters (many from professional mathematicians) denouncing her answer. – Discussed in “The American Statistician.” GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
The Monty Hall Problem A
B
C
1/3
1/3
1/3
There are three doors. One of these doors contains a prize. The other two do not. Therefore, the probability that any one of the doors contains the prize is 1/3. You choose one door, say, door A.
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
The Monty Hall Problem A
B
C
1/3
1/3
1/3
There are three doors. One of these doors contains a prize. The other two do not. Therefore, the probability that any one of the doors contains the prize is 1/3. You choose one door, say, door A. I open one of the two remaining doors, say, door C, and reveal to you that it does not contain a prize. Given the option, should you stick with your original choice of door A, or switch to door B? Or does it matter? GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*Solution: The Monty Hall Problem • If your original guess was correct and you switch, you lose.
Not behind A = 2/3 A
• If your original guess was wrong and you switch, you win. • Since your original guess would be wrong two out of three times, if you switch you’ll win two out of three times.
B
1/3 GeoMet 2013, Brisbane
C
2/3 Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Use Probability Rules: in this case, Bayes’ Rule • Conceptually, Bayes tells you how to update your prior probability of some event occurring (or model being true) after getting some information/data
based on the likelihood (probability) of observing the information/data, given that the event/model is true
posterior probability =
(likelihood) * (prior probability) {(likelihood) * (prior probability)}
• The denominator (summation) is the total probability of the data/information, that is, all the ways that it can occur GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
The Monty Hall Problem: Bayes’ Theorem Bayes’ Theorem P(X and Y) P(X |Y) P(Y) P(Y | X ) P(X) P(X)
We want to know the probability of the prize lying behind A or B, given the host opens C
P ( A and open C ) P ( A) P (open C | A) P ( A | open C ) P(open C ) P(open C )
P ( B and open C ) P( B) P(open C | B) P( B | open C ) P(open C ) P(open C ) GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Remember: Decomposing Total (or Marginal) Probability B3
B1
BA B2
In “Monty Hall”: A = open C B1 = prize behind A B2 = prize behind B B3 = prize behind C
P(A) = P(A&B1) + P(A&B2) + P(A&B3) = P(A|B1)P(B1) + P(A|B2)P(B2) + P(A|B3)P(B3) n
P(A) = GeoMet 2013, Brisbane
i=1
P (A|Bi) P(Bi)
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
The Monty Hall Problem: Probability that host will open door C The a priori probability that the prize is behind door A, B, or C P(A) = P(B) = P(C) = 1/3 This image cannot currently be display ed.
The probability that host opens door C if the prize is behind A: P(open C|A) = 1/2 The probability that host opens door C if the prize is behind B: P(open C|B) = 1 The probability that host opens door C if the prize is behind C: P(open C|C) = 0 The total probability for host opening door C is then P(open C) = P(A)*P(oC|A) + P(B)*P(oC|B) + P(C)*P(oC|C) = 1/2 + (1/3)*(0) = (1/3)*(1/2) + (1/3)*(1) GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
The Monty Hall Problem: Solution using Bayes’ Rule P(open C) = P(A)*P(oC|A) + P(B)*P(oC|B) + P(C)*P(oC|C) = (1/3)*(1/2)
+
(1/3)*(1)
+
(1/3)*(0)
= 1/2
So the probabilities we require are:
P ( A ) P ( oC |A) P ( A | oC ) P( oC )
13 1 2 1 12 3
1 1 2 P ( B ) P ( oC |B) 3 P ( B | oC ) P( oC ) 12 3
Switch to door B! GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
21 movie 21 Movie
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*Alternative Solution: List all possibilities You Choose A A A B B B C C C
Prize Behind A B C A B C A B C
Host opens B or C C B C A or C A B A A or B
Outcome (no Switch) Switch to Outcome Win B or C Lose Lose B Win Lose C Win Lose A Win Win A or C Lose Lose C Win Lose A Win Lose B Win Win A or B Lose
When you switch, you win 2/3 of the time - if you stick you win 1/3 of the time GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Some of the things the letters said • “I’m very concerned with the general public’s lack of mathematical skills. Please help by confessing your error.” » Robert Sachs, PhD, George Mason University
• “There is enough mathematical illiteracy in this country, and we don’t need the world’s highest IQ propagating more. Shame!” » Scott Smith, PhD, University of Florida
• “I’m in shock that after being corrected by at least three mathematicians, you still do not see your mistake.” » Kent Ford, Dickinson State University
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Some of the things the letters to vos Savant said: • “…Your solution is the correct one and any REAL mathematician can produce a proof of its correctness. REAL mathematicians consider this a trivial problem… WHAT discipline do these respondents have their PhDs in? Is it adolescent behavior? If it is in mathematics, my second question is what institution granted it?” » Professor Stephen J. Turner, Babson College
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Some of the things the letters said
• “In a recent column, you called on math classes around the country to perform an experiment that would confirm your response to a game show problem. My eight-grade classes tried it, and I don’t really understand how to set up an equation for your theory, but it definitely does work!” » Pat Gross, Ascension School, Missouri
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Some of the things the letters said
• “After considerable discussion and vacillation here at the Los Alamos National Laboratory, two of my colleagues independently programmed the problem, and in 1 million trials, switching paid off 66.7 percent of the time. The total running time on the computer was less than one second.” » G.P. DeVault, PhD, Los Alamos National Laboratory
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Outline •
Introductions, Questionnaire
•
Overview of Performance of O&G Industry Capital Investment Decisions
•
Underlying Concepts
•
Major Psychological Factors
•
Limits of Intuition in Complex & Uncertain Situations
•
–
Uncertainty Propagation
–
Updating Estimates with New Information
–
Complexity
What can we do… RISC Process
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
A generic US Independent Executive DecisionMaker Buy More Blocks in Viet Nam
Sell South Louisiana
Trade GOM Shelf for Indonesia
Develop Strategy and Set Financial Goals
Partner with XX in Deepwater
• Complex problem Manage the Balance Sheet
Explore in Colombia
-7 chunks of data
• “Optimum” is not intuitive Manage Human Resources
Use Project Finance for Middle East
GeoMet 2013, Brisbane
-models can help
Develop Venezuelan Heavy Oil Produce in the Rockies
Build LNG Capability in Alaska
Achieve 60/40 Oil/Gas Portfolio Split
Carrasco Lecture: Uncertainty, decisions models & people
Assume Operatorship in CO2 Flood
Steve Begg
*Intuition: Steel Band Problem • Assume the earth is perfectly smooth and a band of steel is places around the equator, which has a circumference of 50,000 km. • Now we add in an extra 10m of steel, which slightly forces the band off the surface of the earth. • What is your intuitive estimate of how much?
• Answer: 1.6 metres
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Geometry Problem Imagine two one-mile long pieces of railroad track, put end to end, and attached to the ground at the extremes. When it gets hot, each track length expands by one inch, forcing it to rise above the ground. 1 mile + 1 inch
x? 1 mile
How high is the track off the ground at peak? Give a high and low estimate such that you are 90% sure the correct answer lies between them.
Ref: Richard Thaler
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*Intuition: Geometry Problem
• Give a high and low estimate such that you are 90% sure the correct answer lies between them. • Typical Answers: – Median Low Guess:
½ Inch
– Median High Guess:
2 Inches
– Ranges containing True Value:
15.9%
Ref: Richard Thaler GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*Normative Rule • Pythagorean Theorem
z2 x2 y2 x x
z2 y2
(1 m ile 1 in c h ) 2 (1 m ile ) 2
(5, 2 8 0 * 1 2 1) 2 (5, 2 8 0 * 1 2 ) 2
1 2 6 , 7 2 1 3 5 5 .9 8 in c h e s 2 9 .6 fe e t!
• Descriptive reality: – Most people underestimate x – Why: We anchor on 1 inch and adjust insufficiently
Ref: Richard Thaler
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Intuition is Overrated • Many decision-makers believe that intuition, repeated experience and their general intelligence will see them through • Human beings are imperfect information processors • We can’t always trust our intuition and perception. particularly in an uncertain environment! • We need to use the appropriate tools and frameworks to address the uncertainties and decisions. GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Intuition • Intuition can be useful when – It is educated by making the same kind of decision multiple times and observing the outcome – that is, when you are well calibrated – The possible outcomes of the decision are not important
• Intuition can lead you astray when – Even slight degrees of complexity are involved – Uncertainty is involved, particularly if there are multiple uncertainties which you have to combine to reach a conclusion – Or when it is trained in one situation but then applied to another GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
When to trust intuition
After a McKinsey Quarterly article (May 2010) based on “Conditions for intuitive expertise: A failure to disagree”, American Psychologist, Sept 2009
• Daniel Kahneman (Heuristics & biases, prescriptive DA) and Gary Klein (naturalistic decision-making) debated when you can trust intuition or gut-instinct. They agreed that to protect decisions against bias, four tests should be passed: – The familiarity test: Have we frequently experienced identical or similar situations? – The feedback test: Did we get quick & reliable feedback on the outcomes of past decisions/judgments? – The measured-emotions test: Is our thinking clouded by emotions we have experienced in similar or related situations? (“no”= pass!) – The independence test: Are we likely to be influenced by any inappropriate personal motivations or biased thinking (“no”= pass!)
• If a situation fails even one of these four tests, we need to strengthen the decision process to reduce the risk of a bad outcome. GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
*Probability Training • In terms of practical applicability, probability theory is comparable with geometry; – both are branches of applied mathematics that are directly linked with the problems of daily life.
• While most people have a natural feel for geometry (at least to some extent), many people clearly have trouble developing a good intuition for probability. • In no other branch of mathematics is it so easy to make mistakes as in probability theory. – Conditional probabilities, and Bayes theorem in particular, are especially difficult
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Probability Training • In terms of practical applicability, probability theory is comparable with geometry; “The theory of probabilities is at bottom nothing but – both are branches of applied mathematics that are common sense reduced to calculus; … directly linked with the problems of daily life.
It teaches us to avoid the illusions which often • While most mislead us; people have a natural feel for geometry (at least to some extent), many people clearly have trouble goodworthy intuition for probability. … theredeveloping is no scienceamore of our contemplations nor aofmore useful one is forit admission • In no other branch mathematics so easy to to our mistakes system of public education.”theory. make as in probability Laplace –probabilities, Theorie Analytique Probabilites – Conditional anddes Bayes theorem in particular, are especially difficult
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Developing subjective Probabilities Have you made repetitive forecasts of such an event in the past
Did you receive timely feedback on the accuracy of your forecasts
Go ahead and assess subjective probability
GeoMet 2013, Brisbane
Is there a reference class of events that is similar and on which relative frequency information exists
Use the relative frequency information as subjective probability
Carrasco Lecture: Uncertainty, decisions models & people
Beware of impact of potential errors due to biases and inappropriate heuristics Steve Begg
Eliciting Subjective Probabilities • The de Finetti game: – Bruno de Finetti u
An Italian statistician (1906 – 1985)
u
Worked in the middle ground between mathematics and psychology
– A device to objectively measure subjective probability
• Most people ”lie” about probability without even being aware of it – they even lie to themselves.
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Eliciting Subjective Probabilities • Suppose your friend just took an exam and feels good about how she did. • She might tell you: – ”I aced it; I’m one hundred percent sure I’ll get a perfect score.”
• The de Finetti game is a way to measure how sure she really is about having aced the exam. • We need to ask your friend a series of questions to assess her true subjective probability of having aced the test.
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Eliciting Subjective Probabilities •
Tell you friend the following: – ”Let’s play a game. You have a choice. 1. you can either draw a ball from a bag that has 98 red balls and 2 black balls. If you happen to draw a red ball, I will give you one million dollars, or 2. you can decide to wait to see how you did on the exam, and if you receive a perfect score on that exam I will give you one million dollars. – What’s you choice: draw or wait?”
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Eliciting Subjective Probabilities •
If your friend says: – ”Draw from the bag.” – She then isn’t 100% sure that she has aced the exam – actually she must be less then 98% certain that she has aced the test if she chooses to draw from the bag.
•
Now you ask the next question: – ”Now there are 80 red balls in the bag and 20 black ones.” – ”Do you want to draw, and if you obtain a red ball get a million dollars, or wait to see how the exam went?”
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Eliciting Subjective Probabilities •
If your friend says: – ”Wait for the exam result.” – Then you know that he is more than 80% sure that she really aced the test.
•
Now choose a value in between, such as 90 red balls and ask: – ”Now there are 90 red balls in the bag and 10 black ones.” – ”Do you want to draw, and if you obtain a red ball get a million dollars, or wait to see how the exam went?”
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Eliciting Subjective Probabilities •
At some point, say at 83 red balls (83%), your friend may say: – ”I’m indifferent between drawing a ball and waiting for the exam result.”
•
This (83%) is then her subjective probability of having aced the test.
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Perceptual Limitations – visual illusions are a metaphor for cognitive illusions
• •
•
Previous examples were Cognitive/Probability Illusions Cognitive biases or illusions are similar to optical illusions in that the error can remain compelling even when one is fully aware of its nature. – Awareness of the bias, by itself, does not necessarily produce a more accurate perception. Cognitive biases, therefore, are also exceedingly difficult to overcome.
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
What Does This Have to Do With You and Me? • These biases, which are well known to psychologists, have been shown to influence real decision-making behaviour. • Evidence confirms oil & gas professionals are as prone to these biases and illusions as anyone else. • In any decision-making situation it pays to pause and ask a few key questions – What are the non-intuitive factors might be present, particularly with respect to uncertainty assessment and probability – Am I motivated to see things a certain way? – What expectations did I bring into the situation? – Would I see things differently without these expectations and motives? – Have I consulted others who don’t share my motives and expectations? GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
De-biasing • Create awareness of the problems & recognition of situations when they may be present – Anchoring – Overconfidence – Availability, Vividness & Recency
• Don’t rely on memory alone – write lists • Actively challenge ourselves. – Stop to consider reasons why your judgement might be wrong.
• Abandon false comfort of single-point predictions. – Use ranges instead of single-point estimates. – Use multiple anchors.
• Calibration. – Feedback and accountability. GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
Understanding the Limits of Our Knowledge
• To know that we know what we know, and that we do not know what we do not know, that is true knowledge Confucius
“It’s not what we don’t know that gets us into trouble, it’s what we know that ain’t so” Will Rogers
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
The Pioneers
•
Kahneman and Tversky – 2002 Nobel Price in Economics
•
Russo and Schoemaker – Decision Traps – Winning Decisions
•
Thaler – The Winner’s Curse
•
Bazerman – Judgment in Managerial Decision Making
•
Plous – The Psychology of Judgment and Decision Making
GeoMet 2013, Brisbane
Carrasco Lecture: Uncertainty, decisions models & people
Steve Begg
