Apache Flink Blog

I'm in the same boat - looking to consolidate a few smaller data pipelines that have organically grown over the years into a single pipeline and taking the opportunity to try and get the org to rethink how we emit events in various parts of the stack. As teams operate fairly independently, we need something that enables a flexible data structure whilst still enabling the ability to join data on common well-defined fields.

The current path we are going down is looking at using a 'nested' JSON structure that enables sub-processes in the various systems to inherit values from the parent. Something similar to:

{ "type": "schemaX", "version": 1, "payload": { "k1": "v1", "type": "schemaY", "version": 3, "payload": { "kk1": "v1" } } }

The structure itself will be documented using JSONSchema and hopefully we will be able to verify the validity of events as they are processed, though this might be too expensive to do in high volume scenarios.

The goal is to then build a low latency router that takes in routing policies to forward subsets of events to further data pipelines. The policies themselves will be defined using some kind of DSL (possibly using JSONPath?).

As a whole, this seems to be a fairly common problem[0] that other companies are trying to solve, but a lot of the low level details are not spoken about. One thing that does seem to be a common component to a service like this is Apache Flink.[1]

[0] Netflix Keystone -…

Restate is built as a sharded replicated state machine similar to how TiKV (https://tikv.org/), Kudu (https://kudu.apache.org/kudu.pdf) or CockroachDB (https://github.com/cockroachdb/cockroach) are designed. Instead of relying on a specific consensus implementation, we have decided to encapsulate this part into a virtual log (inspired by Delos https://www.usenix.org/system/files/osdi20-balakrishnan.pdf) since it makes it possible to tune the system more easily for different deployment scenarios (on-prem, cloud, cost-effective blob storage). Moreover, it allows for some other cool things like seamlessly moving from one log implementation to another. Apart from that the whole system design has been influenced by ideas from stream processing systems such as Apache Flink (https://flink.apache.org/), log storage systems such as LogDevice (https://logdevice.io/) and others.

We plan to publish a more detailed follow-up blog post where we explain why we developed a new stateful system, how we implemented it, and what the benefits are. Stay tuned!

Don't forget about the Java machine learning & deep learning "renaissance":
Deeplearning4j https://deeplearning4j.org/, Konduit Serving https://serving.konduit.ai/, Amazon's open-source DJL https://djl.ai/ Apache Mahout https://mahout.apache.org/ Apache Arrow https://arrow.apache.org/

TensorFlow can run on any JVM for building, training and running machine learning models. They have created recently https://github.com/tensorflow/java

PyTorch also supports inference on the JVM but you still need to train with Python. https://www.infoq.com/news/2020/02/pytorch-releases-java-bindings/

I think something like Scala will be used in the future instead of Python even for prototyping.

Streaming is also big business on the JVM:
Apache Kafka https://kafka.apache.org/ (80% of Fortune 100 companies use it)
Apache Flink https://flink.apache.org/
Apache Storm https://storm.apache.org/

Hard to think that .NET ecosystem will ever reach the ecosystem of Java when it comes to open-source.

Many commenters seems to state ah, if company makes $25M it can afford to pay.

First, certain amount of revenue, even if it looks high for you, does not mean it is able to pay ANY amount vendor require. Paying Oracle $50.000 per CPU Core to run Oracle (for 100s of thousands of CPU cores) would be steep even for Facebook/Meta

This idea what there are those large evil companies do not contributes to Open Source, and there are same large VC funded copyright holders are just struggling to survive is quite wrong - it is not that simple and it is Open Source which is one of the ways which create balance of power

Second, it is not about money. Having any restriction on who can use it makes it impractical for using as component of other Open Source software, which Is where I think Open Source power really shines. Check out this article for example

https://flink.apache.org/news/2022/09/08/akka-license-change.html

Flink's event-time support is coming along nicely. Their first round of true event-time support came in November (https://flink.apache.org/news/2015/11/16/release-0.10.0.html), and much more is on the way. Flink will be an excellent platform for Beam, both batch and streaming.

As I understand it, Spark has event-time support coming soon as well. I think basic stuff is landing in 1.7. Not sure precisely what they have planned, but I can only imagine that Spark will also become an excellent platform for executing streaming Beam pipelines in due time. In the meantime, the streaming runner for Spark can either target those features which Spark does support well (i.e., processing-time windowing, in this case), or try to emulate those it doesn't (such as how it was done in the article).

> why Flink and not Kafka ?

I assume you mean "Kafka Streams" when you say this, as Kafka is just a event bus and Flink can be used to read messages from it.

The biggest advantage of Flink (IMO) is that you can write the Flink logic once, and reuse it for both batch and stream processing. So if I write a Flink job that consumes a Kafka stream and produces some aggregated outputs, that same job can be run against my data lake in S3/GCS/Azure Blob Storage, etc.

Kafka Streams does not support batch processing, or working on top of anything other than Kafka. Flink supports building on top of other message buses like Pulsar as well: https://flink.apache.org/news/2019/11/25/query-pulsar-streams-using-apache-flink.html