12 videos · 7 hr 16 minPlaylists
How Things Fall Over
Curated by HackerTube · 12 videos · 7 hr 16 min
HN's reading list for the day after an outage. It opens with short, funny disaster stories, moves into detailed incident accounts and the craft of debugging under pressure, then to designs that limit the damage. It ends with Richard Cook's two short Velocity talks, which several commenters say they rewatch every few months.
Play all on YouTube- 1
How Toy Story 2 Almost Got Deleted: Stories From Pixar Animation: ENTV
@HOLLYWOOD · 551K views · 14 years ago
Two minutes to start: Pixar nearly lost a film to one command and was saved by a home-computer backup, a cautionary tale HN tells to anyone in storage or SRE.
- 2
Dev Deletes Entire Production Database, Chaos Ensues
Kevin Fang · 3.6M views · 3 years ago
Kevin Fang's comic but accurate retelling of GitLab's 2017 incident, where every backup layer failed in turn, which HN prescribes for a bad ops day.
- 3
dotScale 2014 - Robert Kennedy - Life in the Trenches of healthcare.gov
dotconferences · 13K views · 12 years ago
A rescue engineer's first-hand account of a failure that was organizational, not technical: nobody owned the whole site.
- 4
Who Destroyed Three Mile Island? - Nickolas Means | The Lead Developer Austin 2018
LeadDev · 762K views · 8 years ago
A minute-by-minute account of a nuclear accident, which HN uses to push back whenever an outage gets blamed on human error.
- 5
Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017
GOTO Conferences · 101K views · 9 years ago
The postmortem HN rates above almost every written one: a mistyped command reboots a whole Joyent datacenter, and Cantrill walks through what came after.
- 6
Debugging with the Scientific Method - Stuart Halloway
ClojureTV · 25K views · 10 years ago
A calmer counterpart on method (form hypotheses, try to falsify them, write everything down), which several commenters say changed how they debug.
- 7
Zebras All the Way Down - Bryan Cantrill, Uptime 2017
Code & Supply · 49K views · 9 years ago
Cantrill again, on the rare failures (firmware, disks, opaque hardware) that keep turning out to be the real cause, a talk HN calls entertaining and sobering.
- 8
AWS re:Invent 2018: How AWS Minimizes the Blast Radius of Failures (ARC338)
Amazon Web Services · 37K views · 7 years ago
Moving from response to design: cell-based architecture and shuffle sharding, which HN links whenever someone asks how to contain an outage.
- 9
The Do's and Don'ts of Error Handling • Joe Armstrong • GOTO 2018
GOTO Conferences · 74K views · 8 years ago
Erlang's creator makes the case for letting things crash and recover, a conceptual talk rather than a code-heavy one.
- 10
TigerStyle! (Or How To Design Safer Systems in Less Time) by Joran Dirk Greef
TigerBeetle · 87K views · 3 years ago
The most demanding answer here: TigerBeetle's method of assertions, up-front design and deterministic simulation for software meant to be defect-free.
- 11
Velocity 2012: Richard Cook, "How Complex Systems Fail"
O'Reilly · 60K views · 14 years ago
A patient-safety physician explains to ops engineers why complex systems always run partly broken, and one HN commenter rewatches it every couple of months.
- 12
Velocity NY 2013: Richard Cook, "Resilience In Complex Adaptive Systems"
O'Reilly · 47K views · 13 years ago
The capstone, in 19 minutes: Cook's model of pressures pushing a system toward the edge of failure, which one commenter called among the most important talks they had seen.
