Dagster Blog

Author here.

Agreed — I think adding some comparisons to other database partitioning strategies would be helpful.

My 2 cents, specifically about manually partitioning Postgres/MySQL (rather than using something like Citus or Vitess):

SQLite-on-the-server works similarly to Cassandra/DynamoDB in how it partitions data. The number of partitions is decoupled from the number of databases you're running, since data is automatically rebalanced for you. If you're curious, Dagster has a good post on data rebalancing: https://dagster.io/glossary/data-rebalancing.

With manual partitioning, compared to automatic partitioning, you end up writing a lot of extra complex logic for:

- Determining which database each piece of data lives on (as opposed to using partitioning keys which do that automatically)

- Manually rebalancing data, which is often difficult and error-prone

- Adding partitions manually as the system grows

- (Anecdotally) Higher operational costs, since matching node count to workload is tricky

Manual partitioning can work fine for companies like Notion, where teams are already invested in Postgres and its tooling. But overall, I think it introduces more long-term problems than using a more naturally partitioned system.

To be clear: OLTP databases are great — you don’t always need to reach for Cassandra, DynamoDB, or SQLite-on-the-server depending on your workload. But I do think SQLite-on-the-server offers a really compelling blend of the developer experience of…

We only use KubernetesOperators, but this has many downsides, and it's very clearly a 2nd thought of the Airflow project. It creates confusion because users of Airflow expect features A, B, and C, and when using KubernetesOperators they aren't functional because your biz logic is separated. Eg., if your biz logic knows what S3 it talks to in an external task, how can Airflow? So now its Dataset feature is useless.

There are a number of blog posts echoing a similar critique[1].

Using KubernetesOperators creates a lot of wrong abstractions, impedes testability, and makes Airflow as a whole a pretty overkill system just to monitor external tasks. At that point, you should have just had your orchestration in client code to begin with, and many other frameworks made this correct division between client and server. That would also make it easier to support multiple languages.

According to their README: https://github.com/apache/airflow#approach-to-dependencies-of-airflow

> Airflow has a lot of dependencies - direct and transitive > The important dependencies are: SQLAlchemy, Alembic, Flask, werkzeug, celery, kubernetes

Why should biz logic that just needs to run Spark and interact with S3 now need to run a web server?

[1] Anecdotes from various posts - https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753 - https://eng.lyft.com/orchestrating-data-pipelines-at-lyft-comparing-flyte-and-airflow-72c40d143aad -…

Dagster Labs | Software Engineers, Product Managers, Designers | SF, Minneapolis, NYC or Remote | Full-time | Open-source

Dagster is the data orchestration platform built for productivity. Data engineers all around the world, in every industry, use Dagster to build scalable and maintainable data applications like ETL pipelines, ML training pipelines, data integrations and similar systems.

Data applications are the true core of AI and ML training systems, and our goal is to make Dagster the de facto standard for structuring these systems.

We’re an early stage, well-funded startup team with a proven track record of shipping open source software with global adoption. We put a premium on respectful, clear, and complete communication, and we expect each other to be creative, curious, effective, and empathetic.

More info: https://www.dagster.io/careers

I have worked with Airflow during the past three years, but recently we adopted Dagster and I have been using it for the past 3 months. I have found it quite joyful to use and the experience has been very positive. Its main advantages compared to Airlfow (IMO):

- A great UI
- It forces you to clearly define inputs, outputs and types.
- Separation of concerns: Between configuration and data, between processing and IO, and between code and deployment options.
- It allows you to define flexible dags which you can configure at runtime, which makes it easiy to run locally or in k8s, or to switch the storage backend depending on the environment.

This blog post by the founder outlines the differences between the two in much more detail: https://dagster.io/blog/dagster-airflow

Snowflake isn't local, you have to pay for cloud ... Airflow is Airflow, complexity, steep learning curve, there is a whole industry trying to be as/more powerful than Airflow with less complexity and cleaner integration with modern things you might want to do like k8s, although of course Airflow is still super popular and powerful.

the Dagster folks have some comparisons, of course there are popular modern alternatives other than Dagster

https://dagster.io/blog/dagster-airflow

https://dagster.io/vs/dagster-vs-airflow