ClojureTV
4 HN recommenders · recommended in 7 months, 2015–2022
40:13 · 27K views · Apr 22, 2015
Zach Tellman's Clojure West talk covers the basics of queueing theory and its practical use in systems, including backpressure, how queues break down under load, and pitfalls of naive core.async use.
HN recommends it as a fascinating explanation of queues, flow control and overload that applies well beyond Clojure.
7 comments from Hacker News
Yeah this is true, typically you want to keep blocking functions outside this. By default, core.async allocates as many threads as cores in your CPU. In addition to this, I personally find core.async to deal poorly with flow control (e.g. slow subscribers slowing down the entire flow for every subscriber), and seems to not have a lot of telemetry / it’s difficult to find out what’s going on in the thread pool behind it. I’ve personally settled on Zach Tellman’s Manifold library, which provides a much more sane abstraction and is fully compatible with core.async. For some addition thought and material on these topics, I highly recommend watching this talk by Zach, “Everything Will Flow” this video
No. Check out this video by Zach Tellman. He talks about queues and how they break down under load. One of the least intuitive things he points out is that when you have more processors, the breakdown tends to be more of a step function: everything is running smoothly till the moment that it isn't. this video The point he makes arises from basic queue theory and is applicable to all kinds of systems, and how those systems react to load. It's got little to due with particular hardware and everything to do with basic math.
On dealing with backpressure, I would recommend what Zach Tellman writes about in his video "Everything will flow": this video Tellman also tells his own adventure about a system something like this: "a system that would latch up into an overload state because in overload it would log errors at a severity high enough to force the log file to be flushed on every line" He talks about some problems he faced when writing to Amazon S3, when S3 stopped accepting requests. It's a fascinating talk.
See also a very nice video from ClojureWest about queues in system architectures: this video The queue are everywhere - your messaging queue, the threadpool, the hardware threads, and other layers of the stack and APIs you use. The video adds the interesting detail that as you add more tellers (workers) you learn of impending disaster only in the outlier p99 (or higher) latencies; by the time your p85 latency rises, you're already about to stall out.
Anything by Zach Tellman: [LINK] [LINK] In terms of his ideas, watch "Always be composing": [LINK] And he offers some great thoughts about queues and backpressure in "Everything will flow": this video