Strange Loop Conference
12 HN recommenders · recommended in 16 months, 2019–2025
42:15 · 502K views · Sep 15, 2019
Emery Berger explains why performance is hard to measure and deliver on modern hardware, showing how code layout and alignment can swamp optimization effects. He presents Stabilizer for statistically sound evaluation and Coz, a causal profiler.
It is the standard recommendation for anyone puzzled by unexplained benchmark changes from tiny code edits, and for learning about causal profiling.
20 comments from Hacker News
Nobody should do benchmarking like this anymore. Memory layout of your code has such a big impact on performance on modern computers that measuring performance without removing that variable leads to wild goose chases, where you think you improved something, but in reality you incidentally got the compiler to move the code around a bit. Emery Berger has an excelent talk on this [1], and a causal profiler that they developed called Coz[2]. Branch miss-predicts might be somewhat invariant to that, but still, one of the main points of the talk is that people do too much eyeball statistics, mistaking the variance of the underlying stochastic process with actual signal. Coz is pretty trivial to set up with Zig too. 1. this video 2. [LINK]
I haven't had a chance to mess around with it much, but the coz-profiler [1] uses basically the same idea, but the other way around (i.e., slow down everything else to get a "virtual speedup") which helps quantify what the effect of optimizing some code would be. I learnt about this tool from Emery Berger's talk [2] on this (at strangeloop), which I highly recommend. Lots of really nice insight, even outside of this tool. [1] [LINK] [2] this video
Aside: this is the most amazing optimisation information I have seen: this video Unfortunate that it is a video... Summary: Modern CPUs have so many hidden causes of performance variation, that you require special tools to actually measure small performance gains. He finds an amazing but unobvious speed up in SQLite using the causal analysis tool they developed. And a surprising difference between -O2 and -O3 implying overfitting.
I would highly recommend Emery Berger's talks [1] for anyone interested in benchmarking software. The alignment and code layout in memory that huayra mentions has shockingly large performance implications. [1] this video
"Performance matters" by Emery Berger on StrangeLoop'19 is a great talk - this video Also, rest of his talks are also fascinating: - [LINK] - [LINK]