DoorDash Engineering

I'm working on the infrastructure of a startup (Whova). Our backend is fully in Django, so our background tasks are run with celery since that's the main tool for that in the python community and a lot of legacy code is built around that. We process millions of tasks per day, for various things: cpu-bound logic, io (email/push notifications, 3rd part api calls, ...), scheduled tasks, ... These tasks are split across 20 queues processed by multiple workers on 6 dedicated machines.

My celery experience so far has been quite awful to say the least.

From the infrastructure point of view, celery has been the less reliable component of our stack by far.
As others have mentioned, one problem is that celery is frequently used for what it is not meant to be. And that's true for us too. If you check the documentation, deep down, celery was originally designed for short lived tasks, cpu bound, but turned out to be used for long lived tasks. And starting to process long lived tasks is the root of many problems until you find the correct settings to make it work. Of course, these settings are either not documented, or the documentation is useless at best (it sometimes creates even more confusion). There are also very few good quality resources online regarding this.

For several months, we dealt with celery workers getting stuck and not processing tasks, celery workers running out of memory, ... until we found the correct solution. And even now, we still have some random issues we…

Correct, we are super early so there is no self-serve yet.

The primary usecase for this is serving features for ML inference (since eventual consistency is ok and sacrificing write latency for reads is a fair tradeoff). Right now, this is done by using a traditional client-server DB at the moment (Redis/DynamoDB/etc) - or if you're a big tech company that cares about latency you can implement this on your own (https://doordash.engineering/2022/05/03/how-we-applied-client-side-caching/).

As far as self-hosting goes - yes writes will be definitely faster. IKV is fully open source so we're not opposed to it, just haven't figured out the details yet (since self hosting will mostly be useful to very large usecase)

At the core, we use in-memory hashmaps that reference memory-mapped files. So, when a dataset doesn't fit in RAM - it spills to disk automatically.

Cold start - the database is seeded with a "base image", that is built periodically by the backend. That's how a user can add new nodes to their cluster, and still avoid any RPCs.

That being said, if you don't have 10TB of disk, you have to partition IKV (and by extension your application). We support partitioning by allowing documents (the data) to declare partitioning keys. If one shard/partition cannot fit on disk - the store won't startup.

Like many organizations, DoorDash moved to a microservice architecture from a monolith in the last couple of years. Along with the benefits of better business isolation and increased development agility, the move to microservices also brings new kinds of failures that are difficult to handle.
I have been working on reliability projects at DoorDash for the last year, and witnessed firsthand how fires arise and are put out, and what DoorDash does to prevent outages from happening. To share what I learnt, my colleague and I have written a comprehensive overview about failures and their mitigation at DoorDash.
In particular, we discussed the advantage of using a centralized reliability control software, Aperture, which can launch coordinated mitigation plan across services during an outage. Unlike load shedders and circuit breakers, a centralized reliability software is in principle more effective in taming the outages as it has a global view of the system. What do you think of the centralized methodology for reliability control? Do they add extra value compared to existing localized mechanisms?
Blog link: https://doordash.engineering/2023/03/14/failure-mitigation-for-microservices-an-intro-to-aperture/

DoorDash | Remote Contract-to-Hire | https://get.doordash.com

We're looking for a highly-motivated junior or mid-career front-end engineer with experience in React, Next.js, Typescript -- key differentiators are a quantitative and qualitative approach to improving UX, a11y, i18n and documentation -- seeking Engineers who ask questions, unravel the thread, and take a holistic, long-term approach to solving problems.

We're a very small, but nimble team of Engineers working within Marketing at DoorDash.

To learn more about our team, please see our blog post: https://doordash.engineering/2022/02/22/building-a-marketing-engineering-platform-using-next-js-cloudflare-and-contentful

This position is remote and would start off as a contract (40 hours a week, with flexibility), with a desire to hire directly to our team if there is a good fit.

Please send your resume and GitHub (no cover letters needed) to harry[at]doordash[dot]com.

Thank you!

Lately, I've been enjoying a lot of posts in this blog right here:

https://eugeneyan.com/writing

My company just hired some new VPs from Amazon, and the writings in that blog convey a bit of the Amazon WOW. It's pretty insightful stuff.

Also, many posts from Julia Evans :D

More concretely, here are a few blog posts I go back to occasionally:

https://doordash.engineering/2019/02/20/experiment-rigor-for-switchback-experiment-analysis/

https://www.unofficialgoogledatascience.com/2019/12/humans-in-loop-forecasting-integrating.html

https://www.unofficialgoogledatascience.com/2016/10/practical-advice-for-analysis-of-large.html

https://eugeneyan.com/writing/ml-design-docs/

https://eugeneyan.com/writing/what-i-do-before-a-data-science-project-to-ensure-success/#first-draw-the-map-to-the-destination-one-pager

https://www.fharrell.com/post/addvalue/

https://www.fharrell.com/post/stat-ml2/

Utilizing image data opens up a number of opportunities for building new image recognition use cases. The challenge is often processing the image data and spinning up new models quickly. I wanted to share this blog post I wrote about how DoorDash built a solution to process image data with deep neural networks in order to quickly spin up new image recognition use cases. Check it out here https://doordash.engineering/2021/11/03/how-doordash-quickly-spins-up-multiple-image-recognition-use-cases/