Valentin Waeselynck writes about Clojure, Datomic, REPL-driven development, and data modeling with DataScript.
Commenters point to his posts on Datomic's history features and on what makes a good REPL, calling the REPL post the best on the topic.
There is so much to dig into for one short article. So much. Too much for an HN comment, I think. A few bullet points:
- It's not an event sourced system, it's a relational system with an absolutely terrible audit log.
- The audit log is (was?) serialised and blocking for every transaction, okay for low volume and low latency transactions.
- 30 minute transactions!? That is NOT low latency. That needs engineering to manage contention.
- The last updated event information is updated in a transaction and definitionally is always the most recent event… what's it actually for?
- The solution is exactly equivalent to a boolean "is_applied" column on the event table… and is still always set "true" inside a transaction.
> That sounds like a recipe for disaster when your replay code and update code are not actually in sync?
It reads like their replay code is their update code, but they have absolutely no confidence that their event history is complete and consistent ("A lot of things can go wrong with a large history, foreign keys, exceptions, serialization mismatches, events out of order etc.") so it's unlikely they'll ever risk replaying events.
The bit that I think has the most interest to consider, though, is the granularity of events. The article refers to database transactions as a monoid, which is to be expected since the application's architecture is built around database transactions and not events, but events can also be treated as a monoid. Making events extremely…
The claim that there's declining interest is absurd, and that's what I'm arguing. Google trends are simply not a measure of anything interesting in practice. It's about as useful as throwing around TIOBE rankings.
>Tooling is a big problem, not because the once you learn emacs and cider it is insufficient, but because it is a pain to setup and learn emacs and cider. Not because Atom and proto-repl are bad tools, but because to get it set up you need 10 plugins and 20 config setting and it ends up being fragile - complected in a major way.
It's quite clear you haven't used Clojure tooling if that's your impression of it. Again, Clojure has some of the best tooling I've used in any language and I've been doing development for around 2 decades now. Intellij and Cursive are a fantastic combination that takes minutes to install. Calva is another great option for VS Code and it's also a one click install. Both these projects are actually being funded. Cursive is a commercial product and Calva is funded via Clojurists Together. Both Cursive and Calva show just how much the community has grown, since people and companies are willing to pay for tooling nowadays. That's a real measure of growth.
>Another part of the problem is that Clojure advocates vastly overvalue the what unique benefits Clojure adds things like hot reloading and the REPL. Clojure and ClojureScript might have better or more pure implementations, but everyone has some form of these features now.
Again, this is…
> The Events here give up on the naming, granularity, and semantics problem: they're extremely low level fine-grained changes to fields in a database.
You say "fields", I say "facts".
Which of a set of nominal events or a graph of facts is higher-level will ultimately depend on the domain, but using Datomic over several years now has led me to think that it's much more often the latter than conventional Event Sourcing has trained us to believe.
In fact, it may well be that Event Sourcing has traditionally been relegated to those (few) use cases where the "set of nominal Event Types" approach is better, being the only situations where conventional Event Sourcing is practical.
> Events themselves are no longer interesting or semantically meaningful, because they're a single atomic change to a database field. A change in the way state is represented means a different set of events is produced. Subscribing to meaningful occurrences in this model is difficult,
Datoms are not 'single' or 'atomic' - they're packed coherently together in a Transaction, and are immediately relatable to entire database values. Subscribing is just pattern matching, and is not hard, as the following section of the article tried to show: https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html#processing_events_with_datomic.
> [...] and probably will eventually result in the creation of a "meta-event" for each action that contains the semantic intent of the…
To me, being able to make changes to your code while keeping the current state in the REPL is key to interactive development.
My workflow is generally to build up state, and then experiment with functions on that state until I get the correct output.
This workflow is very natural in Clojure, Common Lisp and even Python (with IPython and autoreload).
However, in Racket you have to restart everything on every change. This works ok for smaller applications, but if for example, your state is a large dataset that you pull from a remote database, it becomes a little more difficult.
There are possibly workarounds, and I'm not saying that Racket is bad because of this. There are definite advantages to this approach, mainly for keeping everything simple and predictable. However, this was a roadblock for me, and the main reason why I didn't spend more time working on it.
I also miss the ability to make changes to code, refresh the browser and instantly see the changes (instead of having to restart the server after each change).
Here's a good discussion (in my opinion) of what makes a good REPL for interactive development:
http://vvvvalvalval.github.io/posts/what-makes-a-good-repl.html
This is often called "event sourcing". The other commenter mentions Datomic, which probably is the 'latest and greatest' form of event sourcing (I've never used it).
https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event-sourcing-without-the-hassle.html
If you want something built on postgres, I don't have any specific recommendations, but you can build a simple event sourcing system yourself. I worked somewhere that used event sourcing on postgres, and the core log was basically just a table with aggregate IDs, event sequence numbers, and a JSONB column for the event payload. I recommend starting with just one part of your application if you're going to adopt event sourcing. You will quickly find that there are a lot of new considerations and pitfalls that you don't have with traditional RDBMS usage. Overall, event sourcing is hard to get right, so you should consider the trade-offs carefully. There are easier ways to get audit logs, for example: https://www.pgaudit.org/
I will also recommend this as a way to start to understand the tricky aspects of event sourcing: https://leanpub.com/esversioning
For me, the real game changer is the REPL-driven workflow you get with both Clojure and ClojureScript, but it's really hard to describe the details of how it's different than hot-reloading in the React ecosystem. I've tried many times, but my explanations seems to mostly confuse people rather than enlighten them.
So let's use the words of others instead, here is the best explanation I know about why the Clojure REPL rocks: https://vvvvalvalval.github.io/posts/what-makes-a-good-repl.html
Short version is: you can send snippets of code to your REPL from your editor, and because of the common structure of Clojure apps, you can usually evaluate content in the middle of a function without having to change any code. Basically unlimited introspection into a running program.
I used to program in PHP, C#, Golang, Ruby, NodeJS/JavaScript (both backend/frontend) and some others, and no language comes close to the dev experience of Clojure. I'm sure other lisps are similar as well, but haven't used anything else than Clojure substantially.