Redpanda Blog

I am the founder of RisingWave (http://risingwave.com/), an open-source SQL streaming database. I am happy to see the launch of Warpstream! I just reviewed the project and here's my personal opinion:

* Apache Kafka is undoubtedly the leading product in the streaming platform space. It offers a simple yet effective API that has become the golden standard. All streaming/messaging vendors need to adhere to Kafka protocol.

* The original Kafka only used local storage to store data, which can be extremely expensive if the data volume is large. That's why many people are advocating for the development of Kafka Tiered Storage (KIP-405: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage). To my best knowledge, there are at least five vendors selling Kafka or Kafka-compatible products with tiered storage support:

-- Confluent, which builds Kora, the 10X Kafka engine: https://www.confluent.io/10x-apache-kafka/;

-- Aiven, the open-source tiered storage Kafka (source code: https://github.com/Aiven-Open/tiered-storage-for-apache-kafka);

-- Redpanda Data, which cuts your TCO by 6X (https://redpanda.com/platform-tco);

-- DataStax, which commercializes Apache Pulsar (https://pulsar.apache.org/);

-- StreamNative, which commercializes Apache Pulsar (https://pulsar.apache.org/).

* WarpStream claims to be "built directly on top of S3," which I believe is a very aggressive approach that has the potential to drastically reduce costs, even compared to tiered…

Redpanda | Full-time | Remote | https://redpanda.com/careers

Come shape the future of streaming! We are the startup behind Redpanda, a Kafka-compatible event streaming platform. With a compelling combination of no data loss, operational simplicity, and consistent performance at scale, Redpanda takes advantage of modern hardware to deliver the modern streaming platform for mission critical workloads.

Redpanda is looking for the following roles within engineering:

* Analytics Engineer - https://jobs.lever.co/redpanda.com/6f52eb53-b4ca-4763-8c68-dba7279bc770/

* Backend Software Engineer, Cloud - https://jobs.lever.co/redpanda.com/36d3b98e-533e-41be-822b-f3f0c093480e/

* C++ Core Engineer - https://jobs.lever.co/redpanda.com/b2f26147-9b9f-4088-9867-ccfdb97cc782/

* Developer Experience Engineer - https://jobs.lever.co/redpanda.com/913d6556-d827-4c1a-9a4f-2b37e9b095cc/

* Developer Productivity Engineer - https://jobs.lever.co/redpanda.com/c9780ee7-036b-4ccc-975f-388fd4c42c7d/

* Developer Support Engineer - https://jobs.lever.co/redpanda.com/5eb54061-289b-4e38-8855-4fe5b08514ae/

* Engineering Manager, Cloud - https://jobs.lever.co/redpanda.com/59ab1690-8469-45a1-9b95-aafc5c82f64c/

* Engineering Manager, Core - https://jobs.lever.co/redpanda.com/efac50ea-a925-4fe6-badc-84c1199bf598/

* Information Security Engineer - https://jobs.lever.co/redpanda.com/9f45fd56-443b-4887-a8b8-fdf5be6c484d/

* Site Reliability Engineer -…

Redpanda Data (formerly Vectorized) | Remote | Full-time | https://redpanda.com/careers/

Come shape the future of streaming! Redpanda, a Kafka-compatible event streaming platform. With a compelling combination of no data loss, operational simplicity, and consistent performance at scale, Redpanda takes advantage of modern hardware to deliver the modern streaming platform for mission critical workloads.

We are a team of doers, seasoned engineers, hackers and builders, working on the future of streaming. A fully distributed, diverse team, with members of Medellin - Colombia, London, San Francisco, St. Petersburg - Russia, NYC, San Diego, Germany, Canada, Poland and growing!
Redpanda is looking for the following roles:

C++ Core Engineer

Developer Support Engineer

Distributed Systems Engineer

Information Security Engineer

Lead Engineer, WASM

Site Reliability Engineer

Test Infrastructure Engineer

Director, Product Marketing

Sr. Engineer, Developer Content

Sr./Principal Product Marketing Manager

Product Manager

To apply, visit https://redpanda.com/careers/ click on the role, then APPLY FOR THIS JOB

Redpanda Data | REMOTE | Full-time | https://www.redpanda.com/careers

Redpanda is pioneering the Agentic Data Plane (ADP) - a new category in AI infrastructure that makes it simple and secure to connect AI agents with enterprise data and systems. Built on a multi-modal data streaming engine, Redpanda empowers agentic applications that reason and act in real-time with speed, autonomy, and precision.
Global leaders including Activision Blizzard, Cisco, Moody's, Texas Instruments, Vodafone and 2 of the top 5 banks in the U.S. rely on Redpanda to process hundreds of terabytes of data a day.
Backed by premier venture investors Lightspeed, GV and Haystack VC, Redpanda is a diverse, people-first organization with teams distributed around the globe.

Software Engineer, Core Enterprise - https://job-boards.greenhouse.io/redpandadata/jobs/4652843005

Software Engineer, Core SQL - https://job-boards.greenhouse.io/redpandadata/jobs/4659296005

AI Agent Engineer - Poland - https://job-boards.greenhouse.io/redpandadata/jobs/4678043005

I hope you like this blog post illustrating how Generative AI can be integrated into streaming data environments using just 45 lines of code, leveraging DataSQRL and Redpanda Data.

DataSQRL is a tool that simplifies building data applications on streaming data. What's remarkable is how it enables handling complex streaming data processes in a surprisingly concise manner - the entire implementation showcased in the blog is only 45 lines of code!

Additionally, it explores Redpanda Data, a streaming data platform that serves as an efficient alternative to Apache Kafka.

The post talks about the technical implementation but hints at broader applications in predictive analytics and real-time decision-making across various industries.

For those who are into AI, data streaming, or just love efficient coding, this might be a compelling read. If anyone here has streaming data and AI, please let me know your thoughts.

Originally posted on RedPanda's website

https://redpanda.com/blog/real-time-clickstream-gen-ai-aws-datasqrl

For Kafka alternative written in C++ there's Redpanda [1],[2].

Redpanda claim of better performance but benchmarks showed no clear winner [3].

It will be interesting to test them together on the performance benchmarks.

I've got the feeling it's not due to programming language implementation of Scala/Java (Kafka), C++ (Redpanda) and Rust (Walrus).

It's the very architecture of Kafka itself due to the notorious head of line problem (check the top most comments [4].

[1] Redpanda – A Kafka-compatible streaming platform for mission-critical workloads (120 comments):

https://news.ycombinator.com/item?id=25075739

[2] Redpanda website:

https://www.redpanda.com/

[3] Kafka vs. Redpanda performance – do the claims add up? (141 comments):

https://news.ycombinator.com/item?id=35949771

[4] What If We Could Rebuild Kafka from Scratch? (220 comments):

https://news.ycombinator.com/item?id=43790420