Netflix engineers write about architecture, performance debugging, video encoding, and resilience on the company's engineering blog.
Commenters cite it as a good example of a tech blog that shares real information rather than marketing, and list it among strong engineering blogs.
This is a known and hard problem. A lot of companies are trying to internally do some sort of analysis of their own content libraries and intelligently tune their encoders. Netflix is definitely the most known for their tech related to video analysis and introduction of VMAF[1], but other metrics also exist which enable you to compare the original/master and the encoded variant (PSNR, SSIM, etc). Bottom line is that you need a lot of trial and error and fitting different curves on your bitrate/quality graphs and often times what metrics consider to be good quality human visual system doesn't agree 100% with. It's a very interesting problem nonetheless, and I recommend [2] if you want to learn more.
[1] https://en.wikipedia.org/wiki/Video_Multimethod_Assessment_Fusion
[2] https://netflixtechblog.com/toward-a-practical-perceptual-video-quality-metric-653f208b9652
This was a good read and great work. Can't wait to see the performance tests.
Your write up connected some early knowledge from when I was 11 where I was trying to set up a database/backend and was finding lots of cgi-bin online. I realize now those were spinning up new processes with each request https://en.wikipedia.org/wiki/Common_Gateway_Interface
I remember when sendfile became available for my large gaming forum with dozens of TB of demo downloads. That alone was huge for concurrency.
I thought I had swore off this type of engineering but between this, the Netflix case of extra 40ms and the GTA 5 70% load time reduction maybe there is a lot more impactful work to be done.
https://netflixtechblog.com/life-of-a-netflix-partner-engineer-the-case-of-extra-40-ms-b4c2dd278513
https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/
I don't know if it's a coincidence but we just released a major new feature in Metaflow a few days ago - composing flows with custom decorators: https://docs.metaflow.org/metaflow/composing-flows/introduction
A big deal is that they get packaged automatically for remote execution. And you can attach them on the command line without touching code, which makes it easy to build pipelines with pluggable functionality - think e.g. switching an LLM provider on the fly.
If you haven't looked into Metaflow recently, configuration management is another big feature that was contributed by the team at Netflix: https://netflixtechblog.com/introducing-configurable-metaflow-d2fb8e9ba1c6?gi=5523d54edf6e
Many folks love the new native support for uv too: https://docs.metaflow.org/scaling/dependencies/uv
I'm happy to answer any questions here
While I love ingenuity of developers at Hasura (because I've been personally through these scaling challenges), I always get a gag reaction with GraphQL. I've honestly tried hard to digest it, and I can tell you at large scale where single DB won't cut it, you would either need to develop a large federation layer like [Netflix](https://netflixtechblog.com/how-netflix-scales-its-api-with-graphql-federation-part-1-ae3557c187e2), or just rip-it out. Streaming might elevate the problem a little, but I real problem still lurks under the hood. The fact that front-end community wants to fit everything under GraphQL bothers me, because every backend developer knows that a single tool/technology is usually not the best tool for solving all problems of your life. Remember the golden words, THERE IS NO SILVER BULLET!
I don't see the benefit either. I believe Netflix made this idea popular with sidecar containers as a way to add
> “non-intrusive platform capabilities” [0]
to container stacks.
I understand that the idea might be popular among developers as it is easier to just add a database container to your stack rather than dealing with the db admin. I don't know if Netflix ever recommended databases as sidecar containers. But I have seen it in the wild where dev followed the Netflix model. I sometimes hear people arguing that they have to manage containers anyway so it would be less overhead to manage the db as a container as well.
0: https://netflixtechblog.com/prana-a-sidecar-for-your-netflix-paas-based-applications-and-services-258a5790a015
This article seems to be relevant
>This blog post sheds light on the groundbreaking series of A/B tests Netflix did which resulted in increased member engagement. Our goals were the following:
>Identify artwork that enabled members to find a story they wanted to watch faster [0]
Although it doesn't say that they give different thumbnails to different users, just that they tested different thumbnails for overall performance using A/B testing. So I'd like a source for customized per-user thumbnails. Although I only skimmed the article so I might have missed it.
[0] https://netflixtechblog.com/selecting-the-best-artwork-for-videos-through-a-b-testing-f6155c4595f6