Drift into failure
"What does the collapse of sub-prime lending have in common with a broken jackscrew in an airliner’s tailplane? Or the oil spill disaster in the Gulf of Mexico with the burn-up of Space Shuttle Columbia? These were systems that drifted into failure. While pursuing success in a dynamic, complex environment with limited resources and multiple goal conflicts, a succession of small, everyday decisions eventually produced breakdowns on a massive scale." (More from publisher)
Stats for nerds
7recommend
22mentions
12people
2013–2025years
59% recommend32% neutral9% critical
Mentions per year
What Hacker News says
- A long time ago I had a colleague turn me on to Sidney Dekker’s “Drift Into Failure”, which in many ways covers system design taking into account the “human” element. You could think of it as the “realists” approach to system safety.
At the time we operated some industry specific, but national scale, critical systems and were discussing the balance of the crucial business importance of agility and rapid release cycles (in our industry) against system fragility and reliability.
Turns out (and I take no credit for the underlying architecture of this specific system, though I’ve been a strong advocate for this model of operating) if you design systems around humans who can rapidly identify and diagnose what has failed, and what the up stream and down stream impacts are, and you make these failures predictable in their scope and nature, and the recovery method simple, with a solid technical operations group you can limit the mean-time-to-resolution of incidents to <60s without having to invest significant development effort into software that provides automated system recovery.
The issue with both methods (human or technical recovery) is that both are dependent on maintaining an organizational culture that fosters a deep understanding of how the system fails, and what the various predictable upstream and downstream impacts are. The more you permit the culture to decay the more you increase the likelihood that an outage will go from benign and “normal” to absolutely catastrophic and potentially company ending.
In my experience companies who operate under this model eventually sacrifice the flexibility of rapid deployment for an environment where no failure is acceptable, largely because of an lack of appreciation for how much of the system’s design is dependent on an expectation of the fostering of the “appropriate” human element.
(Which leads to further discussion about absolutely critical systems like aviation or nuclear where you absolutely cannot accept catastrophic failure because it results in loss of life)
Extremely long story short, I completely agree. Aviation (more accurately aerospace) disasters, nuclear disasters, medical failures (typically emergency care or surgical), power generation, and the military (especially aircraft carrier flight decks) are all phenomenal areas to look for examples of how systems can be designed to account for where people may fail in the critical path.
- It's a thought terminating cliche.
Systems thinking is much more nuanced and productive IMO. Drift Into Failure by Sidney Dekker is a good introduction.
- This reminds me very much of Sidney Dekker's work, particularly The Field Guide to Understanding Human Failure, and Drift Into Failure.
The former focuses on evaluating the system as a whole, and identifying the state of mind of the participants of the accidents and evaluating what led them to believe that they were making the correct decisions, with the understanding that nobody wants to crash a plane.
The latter book talks more about how multiple seemingly independent changes to complex loosely coupled systems can introduce gaps in safety coverage that aren't immediately obvious, and how those things could be avoided.
I think the CAST approach looks appealing. It seems as though it does require a lot of analysis of failures and near-misses to be best utilized, and the hardest part of implementing it will undoubtably be the people, who often take the "there wasn't a failure, why should we spend time and energy investigating a success" mindset.
- "would be blacklisted, deplatformed, or deamplified by consolidated authorities"
Sorry. Not true. You have Black Swan (Taleb) and Drift into Failure (Dekker) among many other books. These ideas are very well known to anyone who makes the effort.
- Please consider reading Sidney Dekker's books, The Field Guide to Understanding Human Error[1] and Drift Into Failure[2]
[1] - https://www.amazon.com/Field-Guide-Understanding-Human-Error...
[2] - https://www.abebooks.com/servlet/BookDetailsPL?bi=3123581163...
- Drift Into Failure is a good read about how systems fail, albeit a little light on solutions.
Related
The Field Guide to Understanding Human Error
23 recommend · 57
The logic of failure
15 recommend · 26
Thinking, fast and slow
373 recommend · 1,165
2011–2026
Thinking Fast and Slow by Daniel Kahneman
99 recommend · 157
Succeed
6 recommend · 7
The Power of Habit
127 recommend · 223
