GOTO Conferences
8 HN recommenders · recommended in 16 months, 2018–2025
52:35 · 101K views · May 10, 2017
Bryan Cantrill's GOTO Chicago 2017 talk on debugging production systems under pressure, centered on the incident where an operator's mistyped command rebooted an entire Joyent datacenter. It covers human factors, automation and architecture.
A perennial HN recommendation for outage and postmortem threads, praised as funny, educational and one of the best postmortems anyone has seen.
17 comments from Hacker News
The best postmortem I've seen was actually a conference talk: "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" by Bryan Cantrill (in the GOTO conference, 2017) Here's a YouTube video of it: this video Here's the slides: [LINK] It goes into detail about a pretty bad outage (when an entire data center was brought down), the human aspects, automation, how they handled it, the various risks, architectures, how things fail and about software development in general.
Time to share one of my favorite talks (and speakers) ever - "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" (Bryan Cantrill, GOTO 2017) this video
Perhaps not famous, but Bryan Cantrill, who gives my favorite talks, has an interesting and funny talk on one of the Joyent outages: this video
> I run a YouTube channel that is over 10 years old , has over 770,000 subscribers , almost 600 public videos , 120,000,000 views When writing software involved in managing a live, public, massively multi-user system, the traditional unix-style commands that are immediate, often silent, and capable of damaging effects become a really easy way to shoot yourself in the foot. Worse, some commands might let you accidentally shoot everyone's foot on a typo. The traditional example is accidentally typing something like "rm -f * .bak" (note the extra space after the star). For a good discussion of this type of problem, I highly recommend Bryan Cantrill's talk[1] about the time an operator accidentally rebooted an entire datacenter with a single miss-typed command. The general solution to this is building sanity checks into the software. The user just asked to reformat 500 hosts, but almost all previous uses olf the 'reformat' command affected less than 10 hosts. Maybe we should ask for verification from an actual human if they really intended to run this unusually destructive command. Why doesn't YouTube have this kind of sanity check in their automated takedown/strike/channel-deletion tools? Google wrote automation that can decide to delete a channel with a long history and many successful videos. Why doesn't that automation have basic sanity checks that ask for operator input when asked to do an unusually destructive action like deleting a 10 year old channel with a huge history?
I suppose it all depends on how much infra needs to be stood up for the absolute necessities of the business to operate. Does the company need that internal ticketing system in place to process external client transactions? Probably not, but it'll need it eventually (so maybe that moves to 2nd tier restore process?). My company's RTO is 24hrs to processing new client transactions. Restoring old ones will definitely take longer, but at least new ones can proceed. If your own company's RTO is 2w, that sounds like a lot needs to be in place. Part of the business continuity/ disaster recovery is getting management to sign off on those types of numbers, big or small. Make sure they're realistic. You're right that this type of recovery is not fun. Bryan Cantrill gives a great presentation about managing an outage ( this video ). One of my biggest takeaways, if you're looking at a sweeping outage and a long haul of a recovery, do sleep management asap with your team. Dead tired people are more likely to make brain dead decisions.