Robinhood's engineering team writes about its data infrastructure, including Airflow, Kafka, Kubernetes, and Faust, plus mobile development.
During an outage, commenters used the blog to infer Robinhood's stack and cited its Kubernetes deployment post when guessing at scalability causes.
Since it's Day 2, def appears to be an architectural scalability issue, not some bugs. Quick scan of their engineering blog shows the following stack:
-Zookeeper, Kafka, Spark, Airflow, Faust (Kafka Streams in Python)
-Kubernetes (was/is SALT + Terraform)
-AWS Aurora, PG, ES, Influx
-Presto via AWS Athena, Redshift
-homegrown solutions for a data lake & managing k8s microservices
Fwiw, their team does appear to have a rigorous internal post-mortem process (SEV reviews) [0] drawn from industry best practices.
Let's hope a public one will be made when issues are resolved. (FINRA will be asking for one anyway.)
Not piling on. I genuinely feel for all the infra engineers staying up all day/night fixing this.
[0] https://robinhood.engineering/creating-a-sev-process-that-scales-with-robinhood-4b433f9c439b
Disclosure: I work on Astronomer.io, an open source Airflow platform and SaaS, and also contribute to Airflow.
IMO the pros / cons are really relative to what you're comparing it to and what your workflow needs are.
We've written some guides on "Airflow vs ___" [1] (currently AWS Glue and Oozie). Feel free to email me (in profile) if you'd like to see a comparison to something else.
I like these two posts on Airflow vs Luigi, Pinball, Azkaban, etc [2][3].
[1]: https://www.astronomer.io/guides/
[2]: http://bytepawn.com/luigi-airflow-pinball.html
[3]: https://robinhood.engineering/why-robinhood-uses-airflow-aed13a9a90c8
They recently had a blog about moving to kubernetes
https://robinhood.engineering/building-an-application-deployment-platform-on-kubernetes-f6af367d1e05
you might be right:
https://robinhood.engineering/building-an-application-deployment-platform-on-kubernetes-f6af367d1e05