data Artisans, the company behind Apache Flink, blogs about stream processing, benchmarks, and Flink use cases.
HN links it for Flink stream processing posts, such as transactions on streaming data, and for its job listings in Berlin.
This is pretty cool stuff. I have been working with Apache Flink where the computational model is exactly once (which is still hugely useful) but it still comes with caveats about duplicate messages downstream, which makes writing to downstream db or having multiple flink processes tricky. Often times this is done by windowing data and producing complete snapshots of that window (i.e. you need to keep all the state for a given window) such that the message downstream is idempotent and then aggregating windows together (which can be non trivial for certain aggregations).
Once flink (and other systems) support this, I think it will really be a game changer. It will allow for doing things like sending an increment downstream rather than needing to keep all that state. Instead of mostly seeing stream processing as a thing we do for analytic use cases, it can really become the backbone for streaming applications.
Event Sourcing/CQRS has been an idea for a while, but in practice, is difficult to do because to the inherent difficulty in dealing with message semantics and consistency (see https://data-artisans.com/blog/drivetribe-cqrs-apache-flink for a good write up about such an app). The ability to independently optimize for both reads and writes while also not having to make all messages be idempotent, in my mind, will make this feasible for a broad range of teams and won't require a huge amount of work in thinking up clever message formats or working through as many failure…
data Artisans | ONSITE (Berlin, Germany) and REMOTE
Hiring Process: Application, Phone interview, and On-site interview
What: Data processing, Big Data
data Artisans is building the next-generation platform for programming data-intensive applications.
We are tackling some of the biggest technical challenges in big data and data streaming today. We love open sourcing our code, and working together with the open source community that has formed around Apache Flink, one of the biggest projects of the Apache Software Foundation in its category.
Positions:
- Software Engineer (https://data-artisans.workable.com/jobs/241346)
- Senior Data Engineer (https://data-artisans.workable.com/jobs/325667)
- Software Engineer (Support) (https://data-artisans.workable.com/jobs/241347)
Benefits:
- Competitive salary and stock options
- Flexible working hours and vacation
- Pick your own tech gear
More details here: http://data-artisans.com/careers
data Artisans | ONSITE (Berlin, Germany)
Hiring Process: application, remote interview, on-site interview.
data Artisans is building the next-generation platform for programming data-intensive applications.
We are tackling some of the biggest technical challenges in big data and data streaming today. We love open sourcing our code, and working together with the open source community that has formed around Apache Flink, one of the biggest projects of the Apache Software Foundation in its category.
Positions:
- Software Engineer (https://data-artisans.workable.com/jobs/241346)
- Support Engineer (https://data-artisans.workable.com/jobs/241347)
- Community Evangelist (https://data-artisans.workable.com/jobs/241348)
Benefits:
- Pick your own tech gear
- Competitive salary and stock options
- Flexible working hours and vacation
More details here: http://data-artisans.com/careers
Counting views/impressions in combination with Apache Kafka sounds like the ideal use case for a stream processor like Apache Flink. It supports very large state which can be managed off-hand. This should enable you to count the exact number of unique views in real time with exactly once semantics. Here is a blog post on large scale counting with more details. It also includes a comparison with other streaming technologies like Sanza and Spark: https://data-artisans.com/blog/counting-in-streams-a-hierarchy-of-needs
Also check out this blog post by a Twitter engineer on counting ad impressions: https://data-artisans.com/blog/extending-the-yahoo-streaming-benchmark
"data Artisans Streaming Ledger [...] [extends] Apache Flink [...] to perform serializable transactions from multiple streams across shared tables and multiple rows
of each table." [1]
This sounds like a hybrid approach if you need streaming to handle the speed and volume of data, but still want to use ACID on a small subset of events.
LMAX [2] architecture may be another approach to consider for your application.
[1] https://data-artisans.com/wp-content/uploads/2018/08/2018-08-31-dA-Streaming-Ledger-whitepaper.pdf
[2] https://martinfowler.com/articles/lmax.html
True. In most streaming systems, exactly-once refers to the state that is maintained and controlled by the streaming system. For that, Flink can guarantee such semantics. That state can also be exposed externally, see for example here: http://data-artisans.com/extending-the-yahoo-streaming-benchmark/
For interactions (writes) with external systems, you need transactional integration. There is WIP to do this for transactional databases, for non transactional systems, there are cases where it cannot be strictly solved.