System Safety: why is it so difficult?

System Safety: why is it so difficult?

Missing the system when looking at the system  Richard I. Cook, MD

Professor of Healthcare System Safety

Royal Institute of Technology

Stockholm, SWEDEN Disclaimer: The opinions expressed are those of the speaker and do not represent those of any organisation, institution, or government.

Disclosure: The speaker has no commercial or financial interests related to nuclear or energy industries.

Technical Meeting on Interaction between Individuals, Technology and Organization

A Systematic Approach to Safety in Practice J8-TM-47706

International Atomic Energy Agency

IAEA Headquarters, Vienna, AUSTRIA, 10-13 June 2014 Copyright © 2014 by R.I.Cook for CTL

System Safety: why is it so difficult?

Behind Human Error Distancing Through Differencing

1st ed. 1994! 2nd ed. 2010

Resilience engineering series Copyright © 2014 by R.I.Cook for CTL

System Safety: why is it so difficult?

Distancing Through Differencing How Complex Systems Fail A Tale of Two Stories Going Solid

s a e l b a l i a e v a h t l l t A a s F PD site www.ctlab.org b e w Copyright © 2014 by R.I.Cook for CTL

System Safety: why is it so difficult?

You have… ! ~20% chance of being a hospital patient this year.! ~10% chance of being harmed during that hospitalisation. [1:50]! ~0.1% chance of dying or suffering irreparable damage as the harm. [1:5000]! There is nothing you can do to reduce these risks.! Everyone working in the hospital knows this.! Good luck. Copyright © 2014 by R.I.Cook for CTL

System Safety: why is it so difficult?

The future seems implausible, the past incredible. David D. Woods, testimony to the ! Columbia Accident Investigation Board! 2003

Copyright © 2014 by R.I.Cook for CTL

System Safety: why is it so difficult?

Summary of the lessons learned from other domains… ! 1. Accidents arise from multiple sources.! 2. These sources are… ! …products of the system ‘physics’! …consequences of conflicting goals, continuous change, etc.! …mostly well managed (familiar, visible, in-frame)! …rarely overwhelming (no pattern, ‘special’ cases, etc.) !

3. Accidents tend to follow failure cascades.! …difficult to foresee! …almost never modelled or simulated !

4. Programmatic approaches are weak.! …simple, compelling, feel good, superficially successful! …ineffective, unsustainable, badly suited to real work! Unchanged since TMI

Copyright © 2014 by R.I.Cook for CTL

System Safety: why is it so difficult?

We design our procedures and guidance based on the world we imagine. After accidents our studies reveal that the world is far different from what we imagined. Current nuclear safety is almost entirely reactive. We are quite good at preventing past accidents. “System” decomposition techniques fail because they are based on the world we imagine, not the world that is. Copyright © 2014 by R.I.Cook for CTL

Tale of Two Stories, 1998

First stories: “...a kind of story we... tell after the fact in order to learn from the failure and to decide what kinds of changes are needed. In telling that story, stakeholders focus on a few of the factors and actors that could be seen as contributing to the sequence of events.” Second stories: “...examine how changes in technology, procedures, and organizations, combine with economic pressures to create new vulnerabilities and forms of failure at the same time that they create new forms of economic and therapeutic success.” Copyright © 2013 by R.I.Cook for CTL

Bridging gaps

Main points:

1. Safety is dynamic.

2. Workers detect and avoid hazards — they create safety.

3. Production pressure encourages risk taking.

Copyright © 2014 by R.I.Cook for CTL

Bridging gaps

1. Safety is dynamic.

Available free at www.ctlab.org

Copyright © 2014 by R.I.Cook for CTL

Bridging gaps

1. Safety is dynamic.

Operating point Accident

Copyright © 2014 by R.I.Cook for CTL

1. Safety is dynamic.

Bridging gaps

ACCIDENT BOUNDARY

Counter-gradient

•New rules •Recent accidents •Safety campaigns *The operating point tends to move towards the accident boundary over time. Copyright © 2014 by R.I.Cook for CTL

Bridging gaps

1. Safety is dynamic.

ACCIDENT BOUNDARY

Getting close to the margin is a signal that the operating point needs attention. Copyright © 2014 by R.I.Cook for CTL

Bridging gaps • •

•

•

2. Workers create safety.

You know most risks. You are already doing the important things in your areas. You have been working on safety since you began your training. Only bureaucrats imagine the system is safe.

Copyright © 2014 by R.I.Cook for CTL

Bridging gaps

• • • • • • •

2. Workers create safety.

World is not nice Constant change Flaws are so common Maintenance is continuous Procedures don’t match conditions Production pressures unbalanced New hazards appearing The main problem with a systems view of safety is that there is no system there. Paraphrasing that great! philosopher Donald Rumsfeld Copyright © 2014 by R.I.Cook for CTL 15

Bridging gaps

2. Workers create safety.

The risks that matter cannot be seen from the office. We depend on workers

• to recognize emerging hazards

• to anticipate future hazards

• to exploit available opportunities

• to react to changing conditions

• to learn from experience

These are all features of resilience. Copyright © 2014 by R.I.Cook for CTL

Bridging gaps

3. PP encourages risk taking

ACCIDENT BOUNDARY

Production pressure

Copyright © 2014 by R.I.Cook for CTL

Bridging gaps

3. PP encourages risk taking

We usually don’t know where the accident boundary is. ACCIDENT BOUNDARY

MARGIN

Copyright © 2014 by R.I.Cook for CTL

Bridging gaps

3. PP encourages risk taking

Where is ACCIDENT BOUNDARY? Copyright © 2014 by R.I.Cook for CTL

Bridging gaps

3. PP encourages risk taking

Copyright © 2014 by R.I.Cook for CTL

Bridging gaps

3. PP encourages risk taking

Normalization of deviance… Dianne Vaughn

ACCIDENT
 BOUNDARY

…moves the marginal boundary closer to the accident boundary. Copyright © 2014 by R.I.Cook for CTL

Bridging gaps

3. PP encourages risk taking

The result is often a slipping of the operating point towards the accident boundary.

This is “flirting” with the margin Copyright © 2014 by R.I.Cook for CTL

Bridging gaps

3. PP encourages risk taking

• Managers pretend the process is a ‘system’! • But the real world is not a system! • Workers bridge the gaps! • But training is in procedure following! • Rasmussen: “touch the boundaries”

Resilience engineering Copyright © 2014 by R.I.Cook for CTL

Bridging gaps

3. PP encourages risk taking

Main points:

1. Safety is dynamic.

2. Workers detect and avoid hazards — they create safety.

3. Production pressure encourages risk taking.

Copyright © 2014 by R.I.Cook for CTL