MyNoSQL

I'm at the beginning of a research cycle on graph databases

Pretty much the same here. I am not an expert on any of these projects yet, but I can share some of what I've found and been researching so far.

and am curious to get some suggestions of your favorite open source projects?

Depends on exactly what you are looking for. There are graph "databases" and there are graph "processing engines" and there are subtle differences between them. Some of these things gain massive scalability by running on top of Hadoop, others implement Bulk Synchronous Parallel processing, etc.

A few interesting choices you could look at include:

Apache Giraph - http://giraph.apache.org/

Titan - http://thinkaurelius.github.io/titan/

Faunus - http://thinkaurelius.github.io/faunus/

Apache Hama - http://hama.apache.org/ - not strictly a graph database OR a graph analytics engine, but a generic BSP platform, written in Java, that works with HDFS. Does include some built in support for Graph processing: http://hama.apache.org/hama_graph_tutorial.html

Hypergraphdb - http://www.hypergraphdb.org/index

OrientDB - http://www.orientdb.org/

GoldenORB - http://goldenorbos.org/

You may also be interested in Blueprints, which is more of "just an API" but is supported by several of the various graph database projects out there. https://github.com/tinkerpop/blueprints/wiki

For what it's worth, I'm leaning towards focusing most of my initial energy on Giraph and/or Hama, but I'm also pretty excited…

While SQL can scale, I think this argument is a little spurious.

I truly dislike the NoSQL stance of "never SQL;" it has its place, and its place isn't necessarily at the twitters or the facebooks of today. SQL scales very well with datasets that make sense for RDBMS. CRUD style applications. Core business apps. Data that doesn't necessarily need to be mined furiously. Trying to shoehorn, say, a high-volume message system or intensely self-referential (graph) dataset into an RDBMS is a recipe for disaster, however. Many of the performance issues people see with RDBMS seem to stem from this, I believe.

If your app is hugely real-time datadriven and the datamining (if necessary) can be offloaded to cron jobs, a K/V store is great. And can scale very quickly and relatively cheaply.

If you're doing something hugely relational that, if loaded into a SQL server would require an immense number of self-joins (I'm looking at you, graph analysis) that must be done in real-time and not offloaded to a cron job, a graph database is probably the way to go. They're harder to scale to a huge amount of data, but certain datamining tasks are made much easier - and don't require distributed map-reduce execution. Scaling will get much easier once some systems come out that use k-means (or similar) to cluster and shard the data. That kind of smart scaling would be nigh impossible on either a KV store or a traditional RDBMS. Google gets away with it with BigTable because they can throw so much…

These are all fantastic advise for any open source project. A couple of days ago, I was arguing that for NoSQL projects in particular there is an additional required step: making sure that it is easy to be used from major frameworks and that this integration is "as seamless as possible" (you can read the whole thing here http://nosql.mypopescu.com/post/299775877/nosql-to-people).
With all the NoSQL solutions around, right now it is quite difficult to pick the one that fits your problem. Making it simple to try it out will generate a lot of use cases and that will make further adoption even easier.

:- alex

ps: (excuse and/or ignore what probably can be seen as a plug) on MyNoSQL (http://nosql.mypopescu.com) I am trying to follow some of the most promising NoSQL projects and Redis is one of them. And according to the stats, there is a lot of cool activity around Redis.

Currently (mongodb 1.6.x) sharding is a killer feature that might kill yourself. It's just a marketing propaganda.

I once tried switch to sharding to avoid high write lock ratio(MongoDB use DB level lock currently, only one write operation allowed at the same time). After sharding, MongoDB even don't know how to count my colleciton. db.mycollection.count() return values at random. mongorestore also failed in sharding setup, there's no error message when I was restoring millions of documents, but after it reported successfully restored, I checked the DB, no documents there.

We switched back to more reliable master/slave setup finally. So before trying sharding, you should watch this video http://nosql.mypopescu.com/post/1016320617/mongodb-is-web-scale . It's so true to some extend.

Datomic's immutable storage and time-travel query capabilities are awesome, and I often miss them in other DBs. But Datomic currently isn't designed for write-intensive workloads. And while you can shard Datomic's transactor and then combine multiple DBs in a query (http://nosql.mypopescu.com/post/19310504456/thoughts-about-datomic#comment-465580077), that's only going to get you so far.

However, Apache Geode lets you add custom indexes so it might not be too hard to add Clojure's persistent data structures as a custom index scheme and hook in Apache Geode as a backend to Clojure Datalog:

Clojure Datalog: https://github.com/fogus/bacwn

Datscript: https://github.com/tonsky/datascript

Clojure's Persistent Data Structures for Java: https://github.com/grignaak/clj-ds

First, your friends with MongoDB are fsync and safe. These are both documented and discussed in more than a few places: http://nosql.mypopescu.com/post/1052759609/mongodb-safe-and-fsync-explained

So, assuming you are writing an ecommerce application, here's where I think these flags come in.

- Session data: fsync = true. Wait for a response, and ensure it's written to disk

- Internal web analytics: safe = false. Who cares if it's written, I've got an application to serve!

- Orders: fsync = true. I know, RDBMS, transactions, blah blah blah.

People tend to look at NoSQL and wonder why it doesn't function like MySQL, then they loudly complain how bad the software is. Nobody is writing articles about how Memcached doesn't function like MySQL.