Astronomer, a company selling a managed Apache Airflow platform, writes about Airflow, data platforms, and its own engineering choices.
HN mentions are mostly from Astronomer staff and fans: job postings, product recommendations, and a note that Airflow is hard to run at scale.
i have a really good friend who did the whole UI/UX design bootcamp during the explosion of UX/UI jobs. he did okay, he's probably hopped jobs 2-3 times now and is now without a job.
i actually feel for him, it's definitely one of the career paths that's looked at as excess/waste now while companies slim up to reappropriate money for AI. but i do think there was something there, he was genuinely passionate about what he did and it's just really hard to find work doing it now.
i feel guilty saying this but i've let him talk me through some of what he does, show me how he sees and approaches design (the bulk of what he did was design the interfaces for publicly used webapps and mobile apps) and... idk. i feel like it's all acquired taste and almost a "good app developer will think of these things when they design the front end" and a lot of his insight to me broadly looked like a lot of stuff i would've considered myself as a mere sidequest and my general thought process to deliver a good app. the difference is im building the app and designing the user experience, but his entire career is silo'd to just building the user experience.
im not against breaking out the design to a dedicated resource whether thats one designer or one team who wants to try and maintain a consistent language for a company. i think this has upside to make the design experience not locked to a single developer or developer team, and opens it up to a lot more channels of input. but on the other hand,…
Astronomer | https://astronomer.io | Remote | Full-Time
Astronomer is the commercial developer of Apache Airflow, a community-driven open-source tool that’s leading the market in data orchestration. We’re a globally-distributed and rapidly growing venture-backed team of learners, innovators and collaborators. Our mission is to build an Enterprise-grade product that makes it easy for data teams at Fortune 500’s and startups alike to adopt Apache Airflow. As a member of our team, you will be at the leading edge of the industry and build solutions for some of the world's toughest data problems.
All roles are listed at https://jobs.lever.co/astronomer?lever-via=pXKhCU4CVs and the three below are specific to improving data engineer's developer experience. Feel free to reach out via email at julian [at] astronomer.io.
- Senior Software Engineer, Infrastructure: https://jobs.lever.co/astronomer/eb8cf141-a6f2-4207-b299-ee6b27aa0967?lever-via=pXKhCU4CVs
- Senior Software Engineer, Product Development: https://jobs.lever.co/astronomer/62250bee-2727-4941-8e2d-8065734a9712?lever-via=pXKhCU4CVs
- Software Engineer, UI/UX: https://jobs.lever.co/astronomer/cefedf9a-91fa-490b-87df-9e0abae5c538?lever-via=pXKhCU4CVs
DBT in simplest terms is just a way to orchestrate transformations in dependency order. Table "A" needs to be updated before table "B," because "B" selects from "A." It works well for that.
I've seen it used to get wild west SQL logic into version control. To replace scheduled SQL workbooks running entire data pipelines.
I've also seen over-engineered integrations with other orchestration tools in the "stack."
Once a DBT project grows to a certain size, roughly 500-1000 .SQL files, it gets hard to manage. Not impossible, though it takes intentionality about how to group things together and scale them out operationally.
"Slim CI" is a recent buzzword that has some nice ideas about build automations.
Cosmos is supposed to solve a lot of the automation gripes with dbt. I haven't tried it yet but would like to. https://www.astronomer.io/cosmos/
We use airflow mainly for ETL work, backfills, and batch processing. We do a lot of work with clickstream type data, be it taking data from analytics.js and loading it into redshift, or taking analytic data from redshift and loading into Google analytics. We have developed many pipes and connectors for airflow that allow us to connect to many data sources, both at the source and sink ends. I mainly work on the DevOps team, running our infrastructure and working on back end systems, so my knowledge of airflow is more high level. I just keep the system up so it can run ;P FYI I work for a startup out of Cincinnati, OH called Astronomer, you can find us at https://www.astronomer.io
Also, we weren't like all down, we just saw lots of time out issues when reading/writing to S3.
At Airbnb, we were using SubDAGs to try to manage large number of tasks in a single DAG. This allowed organizing tasks and drilling down into failures more easily but came with its own challenges.
In more recent versions of Airflow, TaskGroups (https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#taskgroups-vs-subdags, https://www.astronomer.io/guides/task-groups/ ) were made to help this a little bit. Hopefully that helps a bit.
At ~1k nodes in the graph introspection becomes hard anyway, as others have suggested, breaking it down if possible might be a good idea.
Disclosure: I work on Astronomer.io, an open source Airflow platform and SaaS [1][2], and also contribute to Airflow.
Our experience was similar to boulous' — Airflow is awesome once it's running but getting it running in an environment that scales to the point that you can deploy your first production DAG can take some effort. That's what led us to trying to do that work once for everyone.
Reusability and composability of components are some of my favorite aspects of working with Airflow.
[1]: https://www.astronomer.io/
[2]: https://github.com/astronomerio/helm.astronomer.io