Twitch Engineering Blog

Software for scaling up live-streaming CDN points of presence (POPs) is a pretty crazy domain. For on-demand video, you can think of a CDN as a cache, getting known-ahead-of-time chunks. But what about for live streaming? It's not feasible to stream frame-by-frame directly from your encoding backend to all the viewers of the World Cup, over something like RMTP - you'd want to use a CDN. So typically, you distribute meaty (multi-second) HLS segments as individual video files, or collections of files, to your CDN; once available, they then need to be requested by browsers/mobile clients as a whole segment, over HTTP(S). Works well with existing CDN infrastructure (provided they can handle the write volume and have big enough inbound pipes)... but the huge issue is that the length of the segment plus round-trips is a lower bound on effective latency. And when interactivity is required, multi-second delays can be horrible.

https://www.wowza.com/blog/hls-latency-sucks-but-heres-how-to-fix-it is a great writeup. Another overview of the problem, and a proposed solution, is in this excellent article by Twitter here:

https://medium.com/@periscopecode/introducing-lhls-media-streaming-eb6212948bef

> In HLS live streaming, for instance, the succession of media frames arriving from the broadcaster is normally aggregated into TS segments that are each a few seconds long. Only when a segment is complete can a URL for the segment be added to a live media playlist. The latency issue is…

I agree with this overall.

It seems that C is a language/compiler/toolchain combo that lets you shoot yourself in the foot, but gives you enough access to stuff to be maximially performant in a random (i.e. not specialized) use case. (C++ is in this camp too).

Rust is a language/compiler/toolchain combo that tries its best to not let you shoot yourself in the foot by presenting those footguns for you to fix up front, without giving up much of the access to stuff (C++ tried some things in this camp like destructors and smart pointers. GC is also something that's in this camp)

Golang is a language/compiler/toolchain combo that lets you shoot yourself in the foot in some circumstances, and does not give you enough access to be maximally performant in every random case, but rather it seems to posit to give you opinions that will get you really, good performance in most random cases (if you need to tune the GC, for instance, you're probably worse off, like here: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap/). Meaning it's making the tradeoffs of upfront complexity vs runtime issues, and Tim Toady vs The One True Way for you, because in the general case those are the most optimized tradeoffs. I don't disagree, especially when the closed-world ecosystem is good enough to not need to reach out to external tools whether those are via cgo or in the build toolchain.

In general, I think that if you reach the point where…

I work in a small shop, much "behind the times", and recently tried introducing gRPC. It went okay. The problem is it was a hard sell and for our limited needs, a lot of gRPC's features weren't something we didn't happily solve with REST. Some of the other more complex features that REST didn't support were things we didn't feel we needed. All together, gRPC was a big burden for me to push, with little payoff for people. I think I was the only one that liked it, and even I disliked parts of it.

So before pulling the switch too far, I decided to take a step back and assess with benefits we all got from gRPC. Mainly it was Protobuf spec, RPC design (ie not REST verbs and whatnot), and code generation.

Because those were the things we all agreed upon, I went with Twirp[0]. It's been a great experience so far. The rollout is just a bit ahead of where our gRPC was, and it's been mostly effortless.

The most effortless thing? I've got an (internal) API user switching from the old REST API to the proto RPC spec, and all they're changing is the URL and a few field naming schemes. It's still JSON for them. They're not generating code, they're not doing anything. It all "just works", which is pretty impressive.

TL;DR: We love simpler options that let us gain the benefits of Protobufs, without the complexity of gRPC when we don't need it.

[0]: https://blog.twitch.tv/twirp-a-sweet-new-rpc-framework-for-go-5f2febbf35f

> The only way to accomplish this at scale is to build something that is legit better and let the market decide. Anything else is just principled wishful thinking.

Classic neo-liberalism BS (pardon my french). Markets are not some natural law written in the atoms, it's a human construction, and we shape it the way we want. Countries can create or destroy markets just with laws, you put a tax here, you put a legal requirement there. That's for example the reason that big american tech companies have been kicked out of South Korea:

- https://www.theguardian.com/world/2025/aug/08/south-korea-google-maps-geographic-data-restrictions

- https://blog.twitch.tv/en/2023/12/05/an-update-on-twitch-in-korea/

Sure, if there are 2 competing companies that play with the exact set of rules, the mArKeT wIlL deCiDe, but that would be a really stupid decision from any government to not shape the rules in its favor. Europe is slowly waking up to this reality, better late than never I guess.

Did the "market decide" that Nvidia chips won't be shipped to China ? Did "the market decide" to put tariffs to get benefits from other countries ? Did "the market decide" to put embargo to Cuba, Iran, Venezuela.. ?

Hearing that regulations and laws is "wishful thinking" makes no sense at all. It's more the opposite, it's the only way to shape the markets the way you want to.

I can't speak for why Herman Li, or any other musician, would be banned for DMCA. But, its my understanding that Twitch pays for a live performance license from the major music publishers to enable musicians to stream on their platform [1]. Of course, this didn't work for Herman Li, but I think it at least speaks to their long-term goals for the platform.

Additionally, while they say to never play recorded music on stream, unless from a source where the copyrights are taken care of, their specific recommendations even in this "clarification" post are very opaque. Never play recorded music on stream, unless you own all the rights, but you probably don't, if you're not sure turn off your VODs. None of that is consistent with some overarching idea of why people get DMCA'd; would I get DMCA'd for playing recorded music live, but not having VODs? I would guess even Twitch doesn't know; its outside of their control.

In short, music on Twitch is REALLY risky, primarily because their rights enforcement is so immature. Its akin to them wildly shooting in the air with a machine gun. YouTube's Content ID is a bit more mature and predictable, so diversifying there may be a good idea.

[1] https://blog.twitch.tv/en/2020/11/11/music-related-copyright-claims-and-twitch/

V8 still uses STW for mark sweep. The benchmark image on this https://v8.dev/blog/concurrent-marking shows 50+ms pause time, quite bad compared to ~5ms or less in new Java collectors. This might be due to STW move phase though? They don't really explain, but the long pause times show that there's definitely still long STW pauses in V8.

For Go, I'm going to be a bad HN user and not read the whole article. Sorry, it's just too long for this time of night. It does appear that my understanding of Go GC is out of date. There's been many improvements in the last couple years. Some strange behavior due to not enough knobs to configure GC, but it appears to have a near constant GC pause? https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/

I'm annoyed that Google doesn't offer much benchmarking results for V8. Huge articles about improvements made with a single benchmark image. And they didn't use standard benchmarks for either so it's unclear what they're even benchmarking. The Go slides you linked include benchmarks from some guys production server he tweeted images of, a bunch of standard benchmarks but they only show % throughout improvement, and no pause times.