James Hamilton

I have helped scale (non web) low latency systems in some pretty stressful situations.

In addition to the sage advice about monitoring / metrics, mentoring and not scaling until necessary I think the following are useful:

* Design Services not Software. In particular read "On Designing and Deploying Internet-Scale Services" (http://mvdirona.com/jrh/talksAndPapers/JamesRH_Lisa.pdf) and at least the first chapter of "The Art of Unix Programming" (http://catb.org/~esr/writings/taoup/html/)

* Get to grips with debugging and profiling so you can figure out what's really happening. Tools like sar, sysstat, [d|k|s]trace, tcpdump, gdb etc and the equivalents for your datastore & application frameworks are invaluable and unfortunately for whatever reason you inevitably won't have all the metrics and monitors you need.

* Do try to understand every layer of your service. I have helped debug scale out related issues from Layer 2 to Layer 8. I have also had to debug many Layer 1 issues while bringing up a new Site or similar. I may not be a DBA, Network Engineer or Software Engineer but in the past I have had to wear those hats while scaling.

* Despite comments elsewhere about learning through Mentoring and baptism-by-fire there is a lot of real engineering & science theory you can lean on. Looking back on courses I took in school while I didn't do any courses on Scalable Web Programming over the years I have used content from courses on Computer Architecture, Math including Queuing…

Amazon is expanding their EC2 feature set so rapidly... The pace is mind blowing to me. Last year, Randy Bias estimated EC2 was pulling $220M revenue/yr:

http://cloudscaling.com/blog/cloud-computing/amazons-ec2-generating-220m-annually

And he estimated an overly conservative 10-20% annual growth. But given the EC2 buzz this year, and personal anecdotes from my friends and colleagues using it, my gut feeling tells me the 2010 revenues will have increased 50-100% over 2009 revenues.

Is EC2 profitable to Amazon? Likely very profitable if you want my opinion. It is well accepted in the industry that the dominant cost in large scale datacenters is power and cooling --not hardware, not human resources-- and I keep running numbers in my head and the hourly prices of all instance types are well above power & cooling.

Just as an example, we know that this new GPU instance has two 95W Xeon X5570 and two 247W Tesla M2050; assuming (1) a max TDP of 50W for the motherboard and rest of the server, (2) instances are run under full load 100% of the time and always reach these max TDP numbers (unlikely, but follow me for the sake of the argument), (3) Amazon uses servers with 80PLUS power supplies (80% efficient or more), (4) a rather good datacenter with a PUE of 1.3 (power usage effectiveness, which includes overhead from power distribution and cooling; numbers in the range of 1.2-1.4 are often quoted by James Hamilton from the AWS team: http://perspectives.mvdirona.com/), and (5)…

James has put out a lot of great material. He's given a talk series and wrote a conference paper called "On Designing and Deploying Internet-Scale Services" which is a trove of valuable design insights:

https://www.usenix.org/legacy/event/lisa07/tech/full_papers/hamilton/hamilton_html/ (somewhat condensed)

Slides from a talk: http://mvdirona.com/jrh/talksAndPapers/JamesRH_AmazonDev.pdf

He writes some of the more practical and useful advice I've seen about how to run a successful business based on high-scale distributed systems.

One of his mnemonics that's stuck with me is the "Four Rs" of recovery-oriented computing: Restart, Reboot, Reimage, Replace (slide 6 in PDF). A human shouldn't be engaged to troubleshoot a problem until the platform has first tried in successing to restart the software, reboot the OS, re-image the machine, and finally replace the hardware entirely. After these auto-recovery steps have failed, only then is it time to engage a human. He describes the connection between these techniques and significantly lower operations costs.

Eventual consistency often gives up the ability to develop applications against the system, because more reasoning about the state of the system has to be embedded in the application logic. For a better description of this problem, see this post from an AWS employee:

http://perspectives.mvdirona.com/2010/02/24/ILoveEventualConsistencyBut.aspx

Note that his "right answer" for performance (not availability, so it's a bit of an aside from your comment) is to dynamically repartition, which is what Clustrix does. Note that we can actually do this type of repartitioning on just the "hot" data, using MVCC to avoid any downtime (unlike Mongo, which blocks all writes).

By the way, the Google Megastore project I believe you're referring to is not actually eventually consistent in the same way that, say, Cassandra is. Megastore is fully ACID within an entity group, meaning that lowering eventually consistency down to the per-node level was not the solution that Google went with. BigTable is also not eventually consistent.

*Clustrix employee

The linked slides are a lot more informative than the summary on that page: http://mvdirona.com/jrh/TalksAndPapers/KushagraVaidMicrosoftDataCenters.pdf

It's interesting that in contrast to the likes of Google and HP who have been stating that the cost of power will soon be greater than the cost of servers — e.g. http://bits.blogs.nytimes.com/2008/11/03/kill-an-energy-hog-save-your-job/ — Microsoft seem to be having a very different experience with servers making up 61% of the TCO and power consumption and distribution/cooling only taking up 16% and 14% respectively.

It's also interesting that they haven't (seemingly) adopted the Google strategy of shifting workloads from data centers based on the external weather conditions, e.g. http://www.datacenterknowledge.com/archives/2009/07/15/googles-chiller-less-data-center/ — This sounded like an ingenious idea when I first heard about it and am surprised that everyone didn't copy it straight away.

Recovery-Oriented Computing has hit HN before. Instead of surfing the overview page, check out the toplevel with links to a LOT of fantastic and accessible papers and presentations:

http://roc.cs.berkeley.edu/

Then, move on to James Hamilton[1]'s "On Designing and Deploying Internet-Scale Services" which is a 10 minute read that's HOWTO on putting Recovery-Oriented Computing into practice:

http://static.usenix.org/event/lisa07/tech/full_papers/hamilton/hamilton_html/

That paper changed the entire way I design and evolve systems. And I keep seeing the lessons from it come up in less-distilled forms in other places. For example, Zach Holman of Github's recent and great "How To Build A Github"

http://zachholman.com/talk/how-to-build-a-github

[1] Vice President and Distinguished Engineer on the Amazon Web Services team where he is focused on infrastructure efficiency, reliability, and scaling. http://www.mvdirona.com/jrh/work/