Richard Jones

These "NoSQL" comparisons are a tricky thing, inasmuch as this is a very volatile space with a lot of really interesting development happening. Furthermore, benchmarking a handful of storage options that are really only alike in that they share the same thing which they are not (relational, SQL) is a lot like running road tests on a handful of cars that are not Honda Civics.

I made the foolish decision a while back to choose a data storage option based on benchmarks and hype, and I'm currently paying for it by having achieved spectacular launch failure and now completely rearchitecting my backend.

NoSQL emerged only after a decade or so of webscale challenges helped us finally realize that everything only looked like relational nails because the only tool we had was SQL hammer, and that tackling every data warehousing problem with SQL was coming at the cost of money, engineering hours, and headaches. Now that viable, production-ready alternatives have become available it's important that design decisions to go beyond seeing problems as not-nails and seeing them instead for what they are.

Are you looking for reliable, versioned master-master replication for embedded devices? Might want to consider Couch. An obscenely fast caching tier that need not persist to disk? Sure, check out Scalaris. A fully distributed and pluggable map-reduce architecture capable of storing a metric fucktonne of data? Hbase might be for you. And, it should go without saying, there are still and…

Here are my top ten, mostly web-related in some way because of my interests. These are the "loud alpha geeks" - the ones who broadcast their interests via regularly-updated blogs or curated links. (There are plenty of "quiet alpha geeks" who are just as "alpha" but not as easy to follow online.)

Brad Fitzpatrick (http://brad.livejournal.com/) - creator of LiveJournal and Danga, now at Google working on PubSubHubbub, the Social Graph API, etc.

John Resig (http://ejohn.org/, http://news.ycombinator.com/user?id=jeresig) - jQuery, Mozilla, lots of other JavaScript and browser stuff.

Paul Bucheit (http://news.ycombinator.com/user?id=paul) - creator of GMail and FriendFeed, now at Facebook.

Ryan Tomayko (http://tomayko.com) - Ruby, Sinatra, now working at GitHub, has an excellent linkblog.

Simon Willison (http://news.ycombinator.com/user?id=simonw) - Django, Python, also has an excellent linkblog.

Leslie Orchard (http://decafbad.com/) - Previously at Delicious/Yahoo, wrote books on JavaScript, Dojo, RSS, now at Mozilla. Follow his bookmarks at: http://delicious.com/deusx

Mark Pilgrim (http://diveintomark.org/) - Atom, HTML5, Linux, Python, Greasemonkey, formerly at IBM, now at Google. Bookmarks at: http://delicious.com/wearehugh

John Gruber (http://daringfireball.net/) - Apple news and punditry, linkblog.

Sam Ruby (http://intertwingly.net/blog/) - Atom, Python, Ruby, Rails, XML, HTML5, SVG. Read what he reads at: http://planet.intertwingly.net/

Kevin Kelly…

Interesting stuff. I read a few more articles here and walked away with more questions than answers. CassFS definitely looks fun learning project.

What I'd really love to see, is an article that explains the goals/niche/maturity/strengths/weaknesses of various distributed storage solutions.

Can anyone here offer a recommendation? (For example, here's something similar: http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/ )

I've been casually researching this topic for months and I still feel so overwhelmed! Not many people are giving a clear picture of what they're trying to accomplish, when/how their system should be used, and how they compare to others.

I'm mostly asking because I recently ended up writing my own. It's just a Python script that sends a UDP message to all our servers asking who has the file. First person to respond wins and the file is downloaded via nginx. Replication is accomplished with durable message queues which are consumed during off-peak hours. It took days to code and suits our needs perfectly.

I learned from this project that reliable, scalable, distributed data storage is very simple, if you focus on solving a very specific use-case scenario.

We use Freenode, #grooveshark is an open channel for random techie users who want to talk directly to developers (or each other), and people working on 3rd party extensions/add ons to Grooveshark.

We have other internal invite-only channels for coordinating with external developers on internal projects. We also have github spamming a channel now whenever changes to a repo that a lot of people are coordinating on get pushed.

The #redis channel saved our asses at least once when Pieter Noordhuis wrote a script to help us with a particular issue we were having. :) And the #facebook channel has been a useful resource as well.

I was wondering if anyone here uses IRCcat: http://www.metabrew.com/article/how-we-use-irc-at-lastfm
I know flickr also uses it and it seems like it would be insanely useful to have notifications about all kinds of things going into various channels. I know flickr also uses it: http://velocityconference.blip.tv/file/2284377/

Haven't spent much time working on it but so far the bot gets kicked for flooding immediately. May only be a problem because we're using freenode for testing. Long term want to run our own IRC server, but for testing it was easier to just use freenode...

This has been posted twice before[1][2]. It should be noted that this benchmark was done over localhost, i.e. packets did not actually leave the computer, so who knows what kind of optimizations were done by the kernel / networking stack.

When looking at examples where packets actually leave the computer, at least one person achieved 1M concurrent connections with Erlang in 2008, and WhatsApp were hitting 2.8M concurrent connections with Erlang in a production system in 2012[4]. Of course memory usage was a lot higher in these examples, but at least they were exercising the networking stack.

[1]: https://news.ycombinator.com/item?id=5127251

[2]: https://news.ycombinator.com/item?id=5474331

[3]: http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-1

[4]: http://www.erlang-factory.com/upload/presentations/558/efsf2012-whatsapp-scaling.pdf

Keep in mind that select basically sucks for handling more than a few sockets -- it's slow and limited to max. 1024 concurrent sockets (unless you change FD_SETSIZE and recompile everything, which may cause unexpected bugs).

An almost always better choice is using epoll -- it's usually faster and can handle any number of concurrent sockets (up to a million and more).

* http://docs.python.org/library/select.html#edge-and-level-trigger-polling-epoll-objects

* http://scotdoyle.com/python-epoll-howto.html

* http://blog.urbanairship.com/blog/2010/08/24/c500k-in-action-at-urban-airship/

* http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-3