Confluent Blog

So much that we presume in the modern cloud wasn't a given when Apache Kafka was first released in 2011.

kevstev wrote just above about Kafka being written to run on spinning disks (HDDs), while Redpanda was written to take advantage of the latest hardware (local NVMe SSDs). He has some great insights.

As well, Apache Kafka was written in Java, back in an era when you were weren't quite sure what operating system you might be running on. For example, when Azure first launched they had a Windows NT-based system called Windows Azure. Most everyone else had already decided to roll Linux. Microsoft refused to budge on Linux until 2014, and didn't release its own Azure Linux until 2020.

Once everyone decided to roll Linux, the "write once run everywhere" promise of Java was obviated. But because you were still locked into a Java Virtual Machine (JVM) your application couldn't optimize itself to the underlying hardware and operating system you were running on.

Redpanda, for example, is written in C++ on top of the Seastar framework (seastar.io). The same framework at the heart of ScyllaDB. This engine is a thread-per-core shared-nothing architecture that allows Redpanda to optimize performance for hardware utilization in ways that a Java app can only dream of. CPU utilization, memory usage, IO throughput. It's all just better performance on Redpanda.

It means that you're actually getting better utility out of the servers you deploy. Less wasted / fallow CPU cycles — so better…

Delivery is communication between two actors. Processing is what one actor (the receiver) does with a message.

Communication is when two actors exchange a message. Communication is generally done over an unreliable medium because in practice there is no way to communicate without the potential of failure.

1. Exactly once communication between two actors over an unreliable medium is impossible. At the very least you have to account for the possibility of failure of the medium, so you might need to re-send messages.

2. At least once communication between two actors is possible -- just re-send a message until the receive acknowledges the message.

3. Because a message might be re-sent, the receiver must be able to cope with duplicate messages. This is what you're describing. This might be done by making message processing idempotent or tracking which messages you've seen before. In either case, you have achieve exactly once processing. That is, if a receiver is given the same message multiple times, only the first receive of the messages changes the state of the receiver.

---

> But exactly-once delivery is not only possible, it's trivial.

Considering that many in the field consider this problem to be impossible (or, at best, extremely difficult, e.g. https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/), this should be a huge red flag to yourself that you're missing something. Everyone has blind spots and that's okay, but…

Before I proceed, let me just state that I have no particular knowledge of either RDBMS' internals or Kafka, so this is nothing but my own amateur musings on the subject.

Now, I wholeheartedly agree that Kafka is more scalable, but I think the key point here is that there is no particular law of nature as to why that is the case. It may just be an historical accident of how RDBMS - and PosgreSQL in particular - have evolved. Further: many of the properties of Kafka are in fact also desirable properties for the PostgreSQL transaction log.

My take on the inopinatus observation, together with the Samza article [1] mentioned on this discussion, are as follows. You can think of Postgres as two "products" (bounded contexts, if you like):

- a stream-based, possibly replicated, transaction log;

- a projection of that transaction log into relational calculus, plus all of the associated machinery.

Thus far we never had the need to think of these as clearly separate "products", but Kafka makes it obvious that they are. In truth, the amount of tools processing WAL outside of PostgreSQL were already hinting in this direction; Kafka just made it obvious.

From this perspective, it seems a tad expensive to take the original transaction log, convert it to a RDBMS representation, then convert it to events and, in some cases, then store it as an event stream in Kafka. It would be much more efficient to simply use the original transaction log directly - and this is why, to me, even…

As someone who’s spent a lot of time working on data pipelines, I think this is a great breakdown of the complexity most data engineers are facing. However, I think there’s two more keys to tidying up messy pipelines in practice:

1. You need to colocate both stream processing for the data pipeline and real-time materialized view serving for the results.

2. You need one paradigm for expressing both of these things.

Let me try to describe a bit why that is.

1. You virtually always need both stream processing and view serving in practice. In the real-world, you ingest data streams from across the company and generally don’t have a say about how the data arrives. Before you can do the sort of materialization the author describes, you need to rearrange things a bit.

2. Building off of (1), if these two aren’t conceptually close, it becomes hard to make the whole system hang together. You still effectively have the same mess—it’s just spread over more components.

This is something we’re working really hard on solving at Confluent. We build ksqlDB (https://ksqldb.io/), an event streaming database over Kafka that:

1. Let’s you write programs that do stream processing and real-time materialized views in one place.

2. Let’s you write all of it in SQL. I see a lot of people on this post longing for bash scripting, and I get it. These frameworks are way too complicated today. But to me, SQL is the ideal medium. It’s both concise and deeply expressive. Way more people are…

I prefer to see the whole system as a succession of stream transformations (https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/). If you view the sequence of input events as first-class and the "current state of the world" as derived, then a lot of problems go away. You need a datastore that can give you a consistent answer as to what order events occurred in, but it's a lot easier to make appending to an append-only list atomic than to make arbitrary state computations ACID.

For downstream derived computations, basically you either make the causal relationship explicit, or accept that you have eventual consistency. The only case where you can have inconsistency is where you have a "diamond" in your computations (i.e. you compute B that's derived from A and C that's also derived from A, and you compute D that's derived from B and C). So you figure out the business implications and either accept it or eliminate the diamond (by computing (A, B) from A and (B,C) from (A, B) instead of computing B and C separately from A). You will also get inconsistency if you do some totally ad-hoc query that's not part of your existing pipelines, but usually that's the kind of reporting query that doesn't need to be 100% consistent; if you do need a consistent version of that then the best approach is to take a regular snapshot, which can be consistent.

Basically you have a lot more precise control, you have causal relationships where you have explicit…

M U INC | Senior Data Engineer (aka "Director of I/O Bound Stuff") | Rust / Kafka | Austin, TX | ONSITE | Full Time | 100K-130K w/ EQUITY

We are a small independent startup (fully funded by our MVP's proceeds) focused on managing, enriching and presenting large and complex Energy Data.

Our technical capabilities translate DIRECTLY into commercial success, we are thus obsessed with coupling Engineering Excellence with Pragmatism. We strive to continuously "co-optimize" architecture and technical decisions with the company's long term business strategy.

Our MVP has been extremely successful. We are actively looking for someone ready to develop and take charge of our next generation Data Platform.

We've fallen in love with Martin Kleppmann's "Building Data-Intensive Applications" book and his "Turning the database inside-out..." [0] talk. You will be building a Kappa Architecture from scratch. This platform will become the data backbone of our business, supporting most of what we do.

This is the stack we already have and/or envision:

- Apache Kafka
- Rust
- LMDB/RocksDB
- Docker/Kubernetes
- Python

You will report directly to the CEO (aka Chief Nerd Officer) and become a member of our small core team. Our success and your success will be extremely aligned as you will be making substantial contributions to our business (data is critical for us).

This is an ONSITE position only here in Austin, TX. Let's talk! Email us at hello@m-u-inc.com

[0]…