Datomic Blog

> I'm curious to know what the actual data structure is.

This tweet from Rich provides the missing puzzle pieces:

"The log is not a btree - used for durability, not query. Separate indexes combine memory with batch-updated storage."
https://twitter.com/richhickey/status/420910382538948608

Restating in my own words:

The transaction log is essentially just a stream of asserts and retracts with metadata. That's used for durability first and echoed to clients/peers and the indexing engine. The indexing engine asynchronously batch updates indexes (which probably are b-trees, but have no need to be streaming append-only style in this context). The peers query a joined dataset of the latest consistent indexes plus the set of unindexed transactions.

> As append-only B tree is persistent.

You're right, it can be. However, the popular append-only b-tree systems (such as CouchDB) are actually append-only B+-tree systems. That "+" means the leaves of the tree are intrusively linked together, and so are not persistent. You can not "fork" such trees cheaply.

> losing information is a specific feature that was added to git

http://blog.datomic.com/2013/05/excision.html

Yeah, it kinda sucks -- mostly one learns to create new copies of definitions/code when doing substantial changes, which the notebook interface makes quite convenient. Or use a full blown editor like Eclipse.

The holy grail as I see it would be storing the entire edit history of the document as a git-like tree of modifications, which is entirely possible because Mathematica notebooks are themselves built out of Mathematica expressions (see for example Cell http://reference.wolfram.com/mathematica/ref/Cell.html). This would be similar to Rich Hickey's work on language-aware version control (http://blog.datomic.com/2012/10/codeq.html)

Wow, very cool indeed. We worked on an open-source community[1] tool using Datomic/Codeq[2], and the ops portion was/has been pretty unpleasant. I'd love to see Datomic-as-a-Service popup, even as a only-meant-for-starter-projects, just so that experimenting with it could be that much easier. That might be possible with the new license, depending on how the Cognitect team sees it.

[1] The service is a frontend to Codeq, having imported many Clojure repos: http://www.jidaexplorer.com/

[2] http://blog.datomic.com/2012/10/codeq.html

EDIT: Added links.

https://blog.datomic.com/2012/10/codeq.html

>codeq ( 'co-deck') is a little application that imports your Git repositories into a Datomic database, then performs language-aware analysis on them, extending the Git model down from the file to the code quantum (codeq) level, and up across repos. By doing so, codeq allows you to:

>Track change at the program unit level (e.g. function and method definitions)

>Query your programs and libraries declaratively, with the same cognitive units and names you use while programming

>Query across repos

Rich Hickey's (of Clojure fame) Datomic database uses Datalog as its query language.

http://docs.datomic.com/query.html

http://blog.datomic.com/2015/01/datalog-enhancements.html

Nathan Marz's (of Twitter startup fame) Cascalog library (also implemented in Clojure) uses Datalog as its query engine to generate Hadoop jobs.

http://alexrobbins.github.io/cascalog-intro/#/step-14

http://clojure.com/blog/2012/02/03/functional-relational-programming-with-cascalog.html

While the author mentions that he just doesn't have the time to look at all the databases, none of the reviews of the last few years mention immutable and/or bi-temporal databases.

Which looks more like a blind spot to me honestly. This category of databases is just fantastic for industries like fintech.

Two candidates are sticking out.
https://xtdb.com/blog/launching-xtdb-v2 (2025)
https://blog.datomic.com/2023/04/datomic-is-free.html (2023)