Bret Taylor

I'm a GAE user for a personal app -- basically I use it for serving comments on my personal blog, similar to whatever Disqus is doing but I prefer the control. I also built on it 2 other personal apps that I later shut down.

IMHO, GAE is great for quick prototypes. Even for apps built for personal usage is great as the free quota is still reasonable, and compared to shitty PHP hosting services it's heaven-like. As far as the free quota goes it's also a lot better than Heroku.

However, I feel very restricted on GAE. My current project at work that pays my bills would never fit into it. Even if it did, because of the stuff we do asynchronously and because of how Google charges for API calls, the bills would go through the roof.

the only PaaS offering that can scale the data-tier

Yes it is. However, you can also scale the data-tier by yourself. Yes, it takes engineering, but it really isn't that difficult if you're planning ahead a little, which you have to do with GAE anyway. I think this old article from 2009 about how Friendfeed used MySQL to do sharding is a classic by now, but in case you haven't read it: http://bret.appspot.com/entry/how-friendfeed-uses-mysql

My biggest problem with GAE is that it is hard to migrate from when it no longer suits your needs. Not impossible, not extremely difficult, but difficult nonetheless taking away precious resources you'd rather invest in something else.

But as far as PaaS goes if you're in to that, it works great.

We're (Inkling) successfully using Cassandra in production to store most of our data.

I wrote the first version of our data store modeled off of FriendFeed's architecture (back when we were five people). There is an excellent blog post about it here:

http://bret.appspot.com/entry/how-friendfeed-uses-mysql

When Cassandra came along, we had long debates internally about the risk of hitching ourselves to their wagon. Eventually we decided that an open source project with a similar (enough) architecture to what we'd been building ourselves was preferable to maintaining a python library in-house.

So far we have no regrets. We have had no production issues and we have an architecture that is built to handle a large amount of incoming data (every note and highlight you take in an Inkling book is synched to our servers). We had already stripped away most SQL functionality by using FF's architecture, so we didn't lose much. And we gained a lot - the ability to tune the durability of different writes depending on the use case, far better fault tolerance, a conceptually simpler indexing strategy, etc.

It definitely has a learning curve - error and status reporting is not wonderful, you generally need to be over-provisioned as a strategy, etc. But I'm happy we learned those lessons sooner rather than later - I'd hate to scramble to implement Cassandra in a strained production environment (my sympathies to a certain website who suffered through this).

The OP's suggestions are pretty bad, as jasonkester points out. But for a specific class of problems (ones that are both performance or JOIN intensive and require the awesome indexing/sorting capabilities in SQL solutions), schema-less MySQL, when done "correctly" is awesome. I designed an application that needed to do graph traversal in real-time, so normalized SQL wasn't an option for me. I tried various NoSQL solutions, but they all either lacked important features or ran insanely slow. I ended up settling with a design similar to how FriendFeed uses MySQL:
http://bret.appspot.com/entry/how-friendfeed-uses-mysql

Not only was the solution faster, but it made code so much more concise, because you can just serialize Python objects to the database rather than write a huge database abstraction layer. For me this is even more valuable than the performance ramifications.

If you're using Python+MySQL, be sure to check out Tornado's database wrapper (http://github.com/facebook/tornado/blob/master/tornado/database.py) because it handles all sorts of sharp corners that straight up MySQLdb won't (like working with binary data, reconnecting when necessary, etc.)

friendfeed was built in 08, before amazon had released EBS as a feature. there is no way in hell the instance-store disk i/o would have kept up with friendfeeds mysql demands (you may remember the post Bret himself wrote on their nifty approach to mysql: http://bret.appspot.com/entry/how-friendfeed-uses-mysql ) Frankly, even EBS probably wouldn't have. Plus, back in 08 the larger ec2 instances were still pretty low on ram compared to what you could cheaply cram in a colo'd server, putting even more pressure on the disk i/o.

I think Bret's really wishing what almost everyone is, that these problems would just go away and you could pay a reasonable premium for that, but we're not there yet and certainly weren't in 2008. I don't think he really checked his numbers on this one.

edit: however there is a happy medium. managed service providers will gladly charge you ec2 prices (250/month for a low end server, 500/month for a beefy one) while handling all midnight colo trips for you.

I'm not a 10x coder, but over the years I've learned to deal with buggy 3PLs by avoiding them, or encapsulating them once and for all. The problem with buggy 3rd-party libs is that the bugs are random. They quite literally keep you dumb by forcing to you memorize random trivia instead of new concepts.

Rewriting the same functions yourself in a better and more efficient way teaches you a lot, especially about 1) how to spot good libraries, and 2) when not to write it yourself.

If you look at the code of really effective hackers, the one thing they have in common is how small the code is compared to what it does. Look at web.py, or bret taylor's datastore built on top of mysql: http://bret.appspot.com/entry/how-friendfeed-uses-mysql

I'm always reluctant about going completely schemaless. Reminds me of the blog post by FriendFeed about how they use MySQL, highly recommended: http://bret.appspot.com/entry/how-friendfeed-uses-mysql

I feel like like there needs to be a better middleground for having some schema but being able to augment it easily with metadata that you're not querying against (yet). Then later extracting the metadata into queryable columns.

I wrote a post outlining some ideas of how to do this: https://github.com/shazow/everything/blob/master/idea/arbitrarily-structured-data-in-rdbms.md

I've only implemented bits and pieces of this in practice, huge convenience so far.