Jack Vanlightly

You got it! By approximate category and in no particular order:

## MISCELLANEOUS

- [Low Tech Magazine](https://solar.lowtechmagazine.com), a solar-powered website showcasing the amazing low-tech things you can do with a little ingenuity,

- [Stephen Wolfram's Blog](https://writings.stephenwolfram.com) on math, compsci, and everything else,

- [Cool Macrophotography](https://www.barrywebbimages.co.uk/Images/Macro/Slime-Moulds-Myxomycetes) for amazing pictures of tiny things (WARNING: Time Sink!),

## COMPUTER SCIENCE

- [While-True-Do](https://blog.while-true-do.io) has lots of stuff on Linux administration and self-hosting,

- [Backend Banter](https://www.backendbanter.fm) for a great podcast on backend development,

- [Two Wrongs](https://two-wrongs.com) for statistics and functional programming,

- [Math ∩ Programming](https://www.jeremykun.com/) where math and programming intersect! Written by a Googler...

- [Paul Bourke's site](https://paulbourke.net/) where you can get happily lost exploring old pages on every topic CS touches,

- [Jason Punyon's blog](https://jasonpunyon.com) for some fun algorithms and things,

- [Yossi Kreinin's blog](https://yosefk.com/blog) for C, Python, and plenty of non-technical topics,

- [Adrian Sampson's academic blog](https://www.cs.cornell.edu/~asampson/blog) for an academic's perspective,

- [Cool polyhedra viewer project](https://georgehart.com/virtual-polyhedra/vp.html) if you like geometry,

- [Another academic blog on…

Thanks for your reply and thanks for your insight and wisdom.

500 million requests per second. I think that's awesome for _intra thread communication_. I have a sharded bank simulation which naively sends money between accounts and it gets 700 million transactions a second which is the aggregate of 12 threads all transacting local to a thread, so 500 million across threads seems good (I split money up rather than accounts per thread). I see that some web frameworks can handle up to 600,000 requests per second on techempower benchmarks. So we need to combine the best of each!

Especially the part about one writer. I have a journal entry called "Golden concurrency" inspired by someone's comment on HN (I really should have linked it!) talking about writing to a database from 3 processes and they decided to split the database into 3 databases rather than suffer contention from concurrency control.

"Golden concurrency" is my term or desire that degradation caused by multiple producers/multiple consumers in different configurations could be optimised for and could be a scenario that is handled without performance degradation. I suspect you need a different design if you have 1 producer:1 consumer producers to consumers, or 1 producer:N consumers or N producers:1 consumer or N producers:N consumers.

I enjoyed the following benchmarks with Kafka and RedPanda which made me think of this…

It just feels like two widely different scenarios we're talking about here.

https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-doesnt-need-fsync-to-be-safe talks about the case of a single failure and it shows how (a) Raft without fsync() loses ACK-ed messages and (b) Kafka without fsync() handles it fine.

This post on the other hand talks about a case where we have (a) one node being network partitioned, (b) the leader crashing, losing data, and combing back up again, all while (c) ZooKeeper doesn't catch that the leader crashed and elects another leader.

I think definitely the title/blurb should be updated to clarify that this is only in the "exceptional" case of >f failures.

I mean, the following paragraph seems completely misleading:

> Even the loss of power on a single node, resulting in local data loss of unsynchronized data, can lead to silent global data loss in a replicated system that does not use fsync, regardless of the replication protocol in use.

The next section (and the Kafka example) is talking about loss of power on a single node combined with another node being isolated. That's very different from just "loss of power on a single node".

> What is the relation between "start with why" and "databases are becoming a commodity"?

My assumption is that the author is painting out a longer-term vision both for Confluent and its audience.

It probably makes more sense in the context of other posts, e.g.

> This trend towards object storage is not just happening at Confluent but across the data ecosystem. Many different types of cloud data systems are integrating object storage into their architecture, and I have covered a few of these such as Neon and ClickHouse Cloud. There is also a flurry of start-ups doing logs (such as the Kafka API) over object storage directly (six and counting). There is money being plowed into data systems that use object storage as the primary, and sometimes, only storage layer.

https://jack-vanlightly.com/blog/2024/5/2/hybrid-transactional-analytical-storage

In other words: managed Kafka is becoming a commodity and Confluent should "start with why" to figure out where to go. I figured the advice and perspective is of general enough interest.

Thanks for the report - we just deployed fix for the same.

No lock-in means it’s 100% open source (PromQL) compatible. You can swap out vendors or move to self-hosted open source solutions should you need to move away from Oodle. When you migrate out, you get to export all your data, dashboards and alerts. you don't need to make any code changes.

We support bringing your own bucket (BYOB) for large enterprise customers however, you cannot bring your own compute at this time. Our thoughts are along the lines of how Snowflake approached the problem - everything fully managed to keep the operational overhead minimal. https://jack-vanlightly.com/blog/2023/9/25/on-the-future-of-cloud-services-and-byoc

This is a great review of Pulsar by Jack Vanlightly. If you haven't read it, I highly recommend it:

https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works

> The thing I don't like is that we also need BookKeeper, but yeah.

Word on the street is now you can swap the consensus module in ZooKeeper. [update: just re-checked Pulsar's dev channel. Apparently one can use etcd instead of zookeeper.]

> sirixdb

Sirix sounds interesting but is a bit close to home to a (currently dormant) temporal DB project of my own. But the real issue is I usually do significant OSS contribution if I'm on the bench and not working.

Wish you success in your effort!