Engineering blog of the Feldera team, who build an incremental SQL computation engine, covering Rust, streaming, and SQL internals.
Commenters describe the team as friendly and the engine's incremental approach as novel; HN also links it for the profiler write-up and Datalog-to-SQL history.
Feldera (https://feldera.com | REMOTE (US) | Full-Time
RELIABILITY AND PERFORMANCE ENGINEER: https://jobs.ashbyhq.com/feldera/709c14e4-1fa9-46b4-9ff8-0783e137ed78
- Strong background in systems engineering, performance testing, or site reliability engineering. - Fluency in Python and Linux fundamentals. Rust experience is a plus. - Experience with distributed systems and database concepts (consistency, fault tolerance, transactions). - Experience with CI/CD/Infrastructure as Code: GitHub Actions, Docker, Kubernetes. - Hands-on experience running large-scale and long-running workloads, preferably in a cloud-native environment. - Curiosity, rigor, and the ability to design experiments that simulate messy real-world conditions. SOLUTION ENGINEER: https://jobs.ashbyhq.com/feldera/544aff74-263f-4749-a4d0-af07d75cd404
- 5+ years experience in solution architect, customer engineering or solution engineering roles. - Strong background in distributed systems, databases, cloud infrastructure, and modern data platforms. - Experience with data-intensive systems in production (e.g., Kafka, Delta Lake, Iceberg, Kubernetes, monitoring/observability stacks). - Exceptional debugging and problem-solving skills, especially in customer-facing contexts. - Excellent communication skills, both for customer-facing and internal interactions. - Ability to write and maintain high-quality technical docs and playbooks. https://jobs.ashbyhq.com/feldera
Feel free to email…
This triggered some associations for me.
Strongest was Cells[0], a library for Common Lisp CLOS. The earliest reference I can find is 2002[1], making it over 20 years old.
Second is incremental view maintenance systems like Feldera[2] or Materialize[3]. These use sophisticated theories (z-sets and differential dataflow) to apply efficient updates over sets of data, which generalizes the case of single variables.
The third thing I'm reminded of is Modelica[4], a language where variables are connected by relations (in the mathematical sense). So while A = B + C can update A on when B or C change, you also can update just A and B, then find out what C must have become.
[0] https://cells.common-lisp.dev
[1] https://web.archive.org/web/20021216222614/http://www.tilton-technology.com/cells_top.html
[2] https://www.feldera.com
[3] https://materialize.com
[4] https://modelica.org
The correct way to think about the problem is in terms of evaluating joins (or any other queries) over changing datasets. And for that you need an engine designed for *incremental* processing from the ground up: algorithms, data structures, the storage layer, and of course the underlying theory. If you don't have such an engine, you're doomed to build layer of hacks, and still fail to do it well.
We've been building such an engine at Feldera (https://www.feldera.com/), and it can compute joins, aggregates, window queries, and much more fully incrementally. All you have to do is write your queries in SQL, attach your data sources (stream or batch), and watch results get incrementally updated in real-time.
There are lots of options to stream data out of Postgres, including:
- https://electric-sql.com (disclaimer: co-founder) - https://feldera.com - https://materialize.com - https://powersync.com - https://sequinstream.com - https://supabase.com/docs/guides/realtime/broadcast - https://zero.rocicorp.dev
Etc.
Definitely not a skill issue if you read the details in their post - https://www.feldera.com/blog/the-pain-that-is-github-actions - they're clearly very experienced with using GitHub Actions and have run into legitimate challenges due to the complexity of what they're using it for.
Rust has long compilation times because it doesn't prioritise compilation times.
What does [0] have to do with the borrow checker?
[0] https://www.feldera.com/blog/cutting-down-rust-compile-times-from-30-to-2-minutes-with-one-thousand-crates