Aphyr

Look at the Raft protocol. In general you’d rather not integrate at the protocol layer; it’s standard to use a consistent store like etcd that implements Raft for the coordination/data that needs to be serializable.

(Kubernetes uses etcd so it scales pretty well for a strongly-consistent k/v store.)

You said “multiple datastores” so I’m assuming you have heterogenous data and something like CockroachDB isn’t an option.

> I'm a beginner in this topic and I find this topic interesting.

Not trying to gatekeepe but rolling your own is dangerous. See https://aphyr.com/ for the gold standard in testing (great educational material). You can use Jepsen to test your distributed systems. But better to just use datastores that Kyle has shown are solid.

This. The article seems like "one weird trick that message queue companies HATE" as it's utilizing, as far as I understand, some SQL semantics in a very specific way to cobble together a way of achieving what other software is designed to do out of the box. It seems fine for a toy system, but I wouldn't stake the success of a real company on this approach.

One could also use DNS TXT as an RDBMS with some interesting fault tolerance and distribution schemes. That doesn't mean it's a good idea or the best way to solve a problem.

If you haven't seen them already, the Jepsen analyses are really worth a read:
https://aphyr.com/posts/293-jepsen-kafka
https://aphyr.com/posts/282-jepsen-postgres
https://aphyr.com/tags/jepsen

> Why do you recommend against using a modern distributed database?

Call me maybe. This is a newer technology and the growing pains have been dramatic. I assume you've followed Jespen?

https://aphyr.com/tags/jepsen

I use MongoDB all the time, for a lot of projects. But I'm also keenly aware of how many years they have struggled to pass the Jespen tests. I can also recall several times engineers from MongoDB have said "We have finally resolved all of the issues raised by Jespen" and then another bug was discovered.

Again, I use MongoDB all the time. But I'm aware it is simpler and safer to use Postgres, for many, many use cases.

Good writing is utilitarian. Boring but it serves the purpose of communication.

Great writing is much more. It communicates and entertains. Here are 2 of my favorites relating to computerz:

https://aphyr.com/posts/341-hexing-the-technical-interview

http://scholar.harvard.edu/files/mickens/files/thenightwatch.pdf

Although I do agree the advice does make sense for most of us, who just aren't very good at writing. But if you do have the skill to write well, do it! I love great writing and love people who aspire to it, even if they fail! Everything you do does not have to be packaged up to maximize mass appeal and resume points.

Question: The author makes this article available on the web as a series of posts, a PDF (https://aphyr.com/data/posts/411/the-future-of-everything-is-lies.pdf) and an EPUB. What do you suppose their tooling looks like?

Is there a single "document containing all the words," and it updates the website, pdf, and epub whenever you change it?

What struck me was that the presentation is beautiful. It seems worth emulating. But that raises the question of what format you'd write your original words in. Do you suppose they just use Markdown files, or something more elaborate?

> Filesystem software, especially databases, can't be ported to Amazon S3

This seems mistaken. Porting databases that run on local disk to S3 seems like a good way to get a lashing from https://aphyr.com/

Can any databases do it correctly?

If so, I doubt they work with the model of partial overwrites. They probably have to do something very custom, and either sacrifice a lot of tail latency, or their uptime is capped by the uptime of a single AWS availability zone. Doesn't seem like a great design.

(copy of lobste.rs comment)