Canva's engineering team writes about infrastructure, incidents, security, CI, data visualization and hiring practices at the design company.
HN links it for incident reports and scaling write-ups; commenters call the API gateway outage report interesting, with timeouts as a key fix, and cite its WebSocket and GPU posts.
I wrote about the way we handle WebSocket connections at Canva a while ago [1]. Even though some small things have changed here and there since the post was published, the overall approach has held up pretty well handling many millions of concurrent connections.
That said, even with great framework-level support, it's much, much harder to build a streaming functionality compared to plain request/response if you've got some notion of a "session".
[1]: https://www.canva.dev/blog/engineering/enabling-real-time-collaboration-with-rsocket/
The whole incident report is interesting, but I feel like the most important part of the solution is buried here [0]:
* "We're adding timeouts to prevent user requests from waiting excessively long to retrieve assets."
When you get to the size of Canva, you can't forget your AbortController and exponential backoff on your Fetch API calls.
--
0: https://www.canva.dev/blog/engineering/canva-incident-report-api-gateway-outage/#improvements-to-detecting-page-deployment-failures
Nix is the only solution for reproducible environments that I would call rock-solid.
It comes with costs and the gpu-related stuff is especially tricky e.g. https://www.canva.dev/blog/engineering/supporting-gpu-accelerated-machine-learning-with-kubernetes-and-nix/
I happened to prefer the original article: https://www.canva.dev/blog/engineering/canva-incident-report-api-gateway-outage/