Klaviyo's engineering team writes about scaling event processing, Graphite metrics, Flink performance, CI problems and using GPT-4 for coding.
HN links it mostly in Klaviyo hiring posts, and a Klaviyo engineer commented on how their RabbitMQ/Celery stack compared with DoorDash's.
This is fascinating. Doordash seems to have encountered many of the same problems we (Klaviyo) did using a RabbitMQ/Celery stack, but arrived at totally different answers. Klaviyo doubled down on making the tools work for us, rather than jumping to a new and untested (for us) broker/job framework technology and arrived at some pretty nice solutions:
Problems with observability? We added lots of custom StatsD and text logging instrumentation (Celery "signal" middleware), so that we could get e.g. accurate "how long do tasks like this spend waiting in the queue?" answers. Other than the inherent limitation of RabbitMQ being a strict queue (unlike Kafka, you can't "peek" at things in the middle of a topic without consuming that topic--we could do something hyper-complicated with exchanges and deliberate duplication of messages to address this limitation, but that doesn't sound worth it to me at all), observability of the brokers themselves seems pretty good. Coupled with Sentry reporting issues that occur during task processing, and some custom decorators to retry/delay/store tasks affected by common classes of systems issues, our visibility tends to be better than I've seen in any other asynchronous queue/message-bus system I've worked on.
Sneaky task disappearances turned out to be mostly bugs in the way we were starting/stopping workers, related to signal handling and rabbitmq "redelivery". By really understanding the Celery worker start/stop lifecycle and coupling that…
Klaviyo | Boston ONSITE | https://www.klaviyo.com | https://klaviyo.tech | http://bit.ly/klaviyocareers
Klaviyo is a data science company that helps brands grow faster. We're profitable, engineering-driven and growing incredibly fast. We have over 12,000 paying customers and are doubling year over year and ingesting an order of magnitude more data every year.
We're hiring across the board (Python/Java engineers, systems engineers, site reliability engineers, hands-on team leads, security engineers, data scientists, product managers, designers, and more). We run a DevOps culture and love full stack engineers who code code in both Python and Javascript (we use React and TypeScript).
Our tech stack is predominantly Python (Django), Java and JS/React running on AWS at massive scale. We run thousands of EC2 boxes, 100s of Aurora MySQL and PostgreSQL clusters and petabyte scale Cassandra clusters. We're heavy users of Kafka, RabbitMQ and Redis. We've just recently launched our first workloads on Kubernetes. We send billions of emails every month and fire customers that send spam--our goal is helping customers have more effective long-term relationships with their customers. We also do push notifications, website personalization and soon SMS.
We recently raised one of the largest fund-raising rounds in Boston history ($150m) but don't intend to change our core values which includes growing our business organically.
Feel free to reach out to me at…