Blog of SmugMug, a photo hosting company, mainly written by CEO Don MacAskill on infrastructure, storage and Amazon S3.
HN commenters call it one of the best technical blogs and link posts on S3 outages, Sun hardware troubles and brand lessons.
Hilarious! Yes, I have seen this way too many times. Great find!
Here's a personal account of ball of mud code:
(from my comment on http://blogs.smugmug.com/don/2007/04/07/how-to-wreck-your-brand/ )
I worked with them [Sony] on the development of their http://musicstore.connect.com/ site that was their attempted 'iTunes Store killer' (just saying that makes me chuckle). Wow, what a mess. Talk about spaghetti code that had mushroomed into a giant ball of mud. It was way over budget and took much longer than anyone anticipated. And a year later, it still only supports Internet Explorer and Windows.
My take on what went wrong in this instance: they did not manage complexity.
They let it grow and grow and did not believe that 'code debt' was a reality. It soon took 10 times longer to make even the simplest changes because of the very poor state of the code.
I have seen this happen at several companies and it's usually the same story. There are B and C level coders that don't understand the dangers or code complexity or code debt and thus create loads of both. The A level coders that do understand this try to explain the growing problem to management. The management doesn't know any better, so they see code cleanup as a needless expense that will only slow down rolling out new features and so they side with the B and C level coders. Of course, the irony is that development will eventually crawl to a near halt because the code is undecipherable. Along the way the A level…
From my perspective, the issue was with execution, not strategic values. Recall the SmugMug guys struggle to use Sun hardware. Not only could they not get someone from sales to return their call, but they had a heck of a time even getting the devices set-up in their datacenter.
http://blogs.smugmug.com/don/2007/05/16/sun-honeymoon-update-storage/
How many things must have gone wrong so that a config UI requiring a non-obvious, redundant magic string could have gotten out the door?
Edit: The "jumped the shark" moment was when, at JavaOne, Sun claimed it would build the world's largest app store by installing nagware along with the JVM.
Compared to the cost of building and maintaining your own system I think it is cheaper. Add the cost of servers, colocation, bandwidth, people.
Also, on S3 you pay exactly for what you use, if you use your own server you are paying for them up front. Another bonus is the using their CDN if you have frequently downloaded files.
Check this comment by Don MacAskill, CEO of SmugMug which stores a huge amount of data on S3: http://news.ycombinator.com/item?id=422574. Check his blog too: http://blogs.smugmug.com/don/tag/s3.
Don MacAskill (SmugMug CEO) has posted a response on his blog: http://blogs.smugmug.com/don/2008/01/28/your-private-photos-are-still-private/
Says Don: "To us, privacy and security are two separate, but related, issues. One analogy we use often is that security is like locking your front door and arming your alarm (no-one can get in without a key), and privacy is like closing your window blinds (no-one can look in from the outside, but you can tell people where you live and they can visit without a key)."
'... We don`t make it as easy to get your photos AND metadata back out of SmugMug ...'
This one sucks in my view. But I dont think it matters as much as it does with flickr which seems to attract a geekier crowd. Instead of beating themselves up over something that doesn't matter to users they concentrate on more useful things like openid ~ http://blogs.smugmug.com/release-notes/2007/02/23/new-features-february-23rd-2007/
It's possible thanks to the magic of dtrace.
In a storage appliance, you wouldn't expect the load to be overwhelmingly CPU bound[1], so performing real time instrumentation isn't a noticeable performance hit: http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_Overhead
[1]: assuming you aren't using ZFS gzip compression: http://blogs.smugmug.com/don/2008/10/13/zfs-mysqlinnodb-compression-update/