Tom Limoncelli, a sysadmin and author, writes about system administration practice, Unix tools, networking and team operations.
Commenters say his books and blog helped them plan and structure systems, and HN links his tools like timetravelpdb.
Google actually does have a systematic solution: fault injection. Google's systems are designed so that you can (manually, if you have the right privileges) tell an RPC to fail regardless of whether it would otherwise have succeeded, and then test the response of the system as a whole.
The problem is that these failure cases are exercised much less frequently than the "normal execution" code paths are. For example, every year Google does DiRT [1] exercises which test system responses to a large calamity, eg. a California earthquake that kills everyone in Mountain View and SF including the senior leadership, and also knocks out all west coast datacenters. The half-life of code at Google (in my observation) is roughly 1 year, which means that half of all code has never gone through a DiRT exercise. The same applies to other, less serious fault injection mechanisms: they may get executed once every year or two, and serious bugs can crop up in the meantime. Automated testing of fault injection isn't really feasible, because the number of potential faults grows combinatorially with the number of independent RPCs in the system.
I'd be willing to bet that the two bugs that caused this outage were less than 6 months old. In my tenure at Google, the vast majority of bugs that showed up in postmortems were introduced < 3 months before the outage.
[1] http://everythingsysadmin.com/2012/09/devops-google-reveals-their-di.html
Yes, very constructive.
A more detailed reply:
3) really? the core of devops is to be data-driven in your decisions. how can you decide if you are maintaining the right uptime if you don't measure it?
8) I agree with you, as does the text. I think you may have reversed what I wrote.
9) The good ones write so that they "think before they do". On a larger team it is important to communicate what you are about to do, or what you have done. I prefer to write mini design docs. The team I'm on does this and I like it so much I want to spread the word.
11: Again, the team I'm on does this and it works so well I want to spread the word. I see I need to expand this out to explain why, not just how.
14: I'll clarify that the point is not to make big changes on your production system. Whether it is qa+live or a zillion steps including dev, qa, UAT, pre-prod, canary and prod. As long as it isn't zero steps.
22: refresh policy: This is for PCs (non-servers). I'll clarify.
23: The last part makes your point. I'll rewrite to make it more evident
28: "Anti-malware? Really? In 2011? I'm sure you have a whole blog post". Yes, I do: http://everythingsysadmin.com/2011/04/apt.html
Thanks for the reminder to add a link! (and if you are blown away that I had to list this, you can imagine my surprise about finding sites that violated this one!)
It would be helpful if you could tell me where you got the impression that I was encouraging use on cloud-storage services. Was it in the README? the code? a post I made on my blog? I ask because I'd like to correct that doc/post/etc. so that others don't get the same impression.
Tom Limoncelli
http://the-cloud-book.com
http://everythingsysadmin.com
P.S. Blackbox was written to safely store the occasional secret file (SSL certs, mostly) in a Puppet repo stored on an in-company source code server.
Even though I personally knew everyone with access to the repo, root access to the repo server, or access to where the backup tapes are stored... it became essential to encrypt those secrets. Honest, I don't trust the people who handle our backup tapes... one of them is me and I'm the least trustworthy person I know.
Even more importantly, now that I could open up the repo to all my coworkers, it enabled me to collaborate across the company. They could read my code, even submit pull requests, and I didn't have to worry that they had access to SSL certs.
The path of a lot of system administrators from amateur to professional resembles that of a child who is exceptionally gifted at building erector sets being hired to construct a pedestrian bridge. Then, if the bridge doesn’t fall, the kid gets to build bridges designed to handle interstate traffic.
This is the key point. It's SO HARD to grow as a sysadmin, because there are very few people who can pass on information. We have pretty much one textbook (TPoSNA: http://everythingsysadmin.com/ ), and a few organizations like LOPSA and USENIX, but there are almost no Windows or Mac specific sysadmins who join.
There are also places where regulation is already happening, albeit in a top down manner, like HIPPA. I doubt top down "you must do this" regulation would help, but accreditation where people have to know and ought to follow best practices would likely do more good.
I'm a programmer, but believe it's important to be well-rounded. To balance my programming skills with sysadmin skills, I've started reading "The Practice of System and Network Administration":
http://everythingsysadmin.com/aboutbook.html
http://www.amazon.com/Practice-System-Network-Administration-Second/dp/0321492668
The book was praised by multiple HN commenters ( http://news.ycombinator.com/item?id=1665915 for example), and I'm glad I listened to their advice. I've only read a few chapters so far (it's a big book!), but I already like it. It contains a lot of good insights, and I like that it's not focusing on a specific OS / platform. Also, you don't have to read it in a linear fashion, you can pick and choose chapters depending on your current needs (moving into a new data center? implementing a security policy?). Thumbs up.
The first quote is difficult to describe for me, but let's call it "poetic", as my brain is too fried to think of a better term for it. Basically, my interpretation of what he's saying is: if your bakery isn't tech-savvy, even at a basic level, they could be getting killed by reviewers (or some other bogeyman) online without knowing it.
The second quote is pretty easy to support with sources. Google has three _official_ languages that can be deployed into production, and Python is one of them. See [this post from a Googler](http://everythingsysadmin.com/2011/01/python-is-better-than-perl6.html), for instance. I have no idea how hard a rule this is, but many a Googler has mentioned it. I think it's worthy of being called a 'fact', and thus completely valid in the article in question.