Martin Kleppmann, author of Designing Data-Intensive Applications, writes on distributed systems, databases, local-first software, and formal verification.
HN commenters recommend his book as a great primer, and cite his accounting article and his analysis of book royalties.
It's been a long time since I looked into this: is there now a way to configure a cluster of Redis instances such that you won't lose messages on node failure? If not, all the nice at-least-once delivery (or "effectively once" when you add message dedupe) you get with something like Kafka/Kinesis/GCP PubSub is gone.
If not, either people's messages don't matter /that/ much (which is fine, just not great for most of my usecases at the moment) or everyone's in for another round of "oh shit, where did the data go?"
Edit: Just in case we end up in CP vs AP datastore wars, please go read https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html
At-least-once delivery requires neither CAP consistency (linearisability) nor CAP availability (any non-failed node must return a response in a non-infinite time), but is a very useful property!
Great improvements, performance was probably the major concern of automerge prev versions, I have begin reading about CRDTs and this is the correct movement field at the moment.
Local first in my mind means your users can still work even if your server is down for a small amount of time for any reason, which means the users can also jump on an airplane and decide to review/update their price table or check out their sales commissions if they want.
It benefits both sides of the software economy (users and providers)[1].
I recommend the reading the paper from 2019 about local first it is not too academic and gives a good view of the challenges [2].
[1] https://mateusfreira.github.io/@mateusfreira-2022-12-04-my-thoughts-on-local-first-software/
[2] https://martin.kleppmann.com/papers/local-first.pdf
There's another very nice description, with useful pictures, by Martin Blais (as part of the documentation of Beancount) at https://beancount.github.io/docs/the_double_entry_counting_method.html (example picture that I would say is the heart of the idea: https://beancount.github.io/docs/the_double_entry_counting_method/media/19e2cc49a057dfeea1cf5254610eab4c9a124488.png )
Another one, with completely different graphs, is by Martin Kleppman at https://martin.kleppmann.com/2011/03/07/accounting-for-computer-scientists.html — despite the name ("Accounting for Computer Scientists"), on reflection / trying it out, I find it less useful than the Beancount one.
(Both of these are actually linked in the second paragraph of the posted article, but they're independently worth mentioning / good.)
Yes, it is one of the most entertaining and insightful articles I have read on computing and programming language for a long time.
All the best for your new Oil shell, and I agree that we need to have similar to POSIX standard for distributed systems but it must put local-first software as the first class citizen [1].
Personally I think the local-first software cannot happen soon enough given that what we have endured for the past 30 years with regard to inefficiency and drudgery of the web based applications.
Just wondering about your thoughts on SDLang [2] as the type alternative to JSON, YAML etc, and lightweight alternative to XML?
[1]https://martin.kleppmann.com/papers/local-first.pdf
[2]https://sdlang.org/
While Git is great at asynchronous collaboration, its a bit clunky to try and work on the same commit with other devs or across devices. This is because Git tracks what changed, not how it changed.
It'd be cool if you were on the same branch as somebody else, or another device, and your working directories could be synced. It'd also be cool for the commit history to be a bit richer, so you could see who, what and when for a change at a keystroke level.
So I'm working on real-time sync for Git! I'd represent the working directory as a tree CRDT [1] and sync that through FUSE and p2p networking.
Not sure whether this is actually a good idea! This is a POC :)
[1] https://martin.kleppmann.com/papers/move-op.pdf
+1 to OP's book, which is the best beginner guide I've found for understanding Beancount / plaintext accounting.
I was also confused about double-entry accounting for most of my life until I read the article, "Accounting for Computer Scientists"[0] by Martin Kleppman (author of Designing Data-Intensive Applications). It explains double entry accounting in a surprisingly accessible way be putting it in terms of graph theory. I don't even like graph theory that much or consider myself competent in it, but Kleppman's explanation was extremely effective.
[0] https://martin.kleppmann.com/2011/03/07/accounting-for-computer-scientists.html