Chips and Cheese

Thanks for correcting my hand-wavy-ness on that. The thrust of my comment was that blog posts like this need to be handled with some care: this is just enough detail to be dangerous without enough context to be truly helpful.

Getting a bunch of people excited to go start relaxing their loads and stores with mostly-truths like "The CPU's store buffer must be flushed to L1 cache to make the write visible to other threads through cache coherency." is just leaving loaded guns lying around. I'm only mostly sure that some uarchs will read from the store buffer without engaging the cache-coherency stuff at least in an HT/ILP world, but that is kind of the point: playing intrinsics jazz is a very asymmetrical bet. You can win big if you absolutely nail it for your particular chip, but you usually lose it all when it's not correct.

For a slightly more detailed treatment I like this SO answer: https://stackoverflow.com/a/62480523/19734375. It sketches out a better intuition about how the store buffer and speculative stores play out in practice and is very well footnoted.

For those that want to go a level deeper I can't recommend Chips and Cheese highly enough. This treatment of Golden Cove is typical of their rigor in understanding what really pushes architecture-specific performance on cutting-edge gear: https://chipsandcheese.com/2021/12/02/popping-the-hood-on-golden-cove/.

Edit: Even this comment is self-contradictory. I pulled that SO answer out of my `x86_64 arcana` bookmark…

You say that like it's bad, but that's just life. We're trying to fit a whole universe into 3 pounds of meat. If we stopped to truly understand anything before taking action, we'd never get anywhere.

That's especially true in technology, where whole armies of people are working to complicate things as fast as possible. For the best of reasons, of course. But when I started out, I could read one short book and have a pretty good understanding of my computer's processor, and from there it wasn't much further to understanding the whole computer. Now I could spend days just understanding, say, processor cache strategies [1]. A field that is super interesting, but if I am to get any of my actual work done, I can't afford to dig into that and the many, many other similar subtopics. I'm going to get a casual understanding of something, do my best to make something useful, and only dig in further when I'm forced to.

When I do have to dig in, it comes in two cases for me. One is where there is something necessary complexity that I would have to learn about regardless. E.g., if something is too slow, I need to learn about what happens under the hood to do proper performance tuning. Great, fine, I will learn it.

But then there's the other bucket, which includes unnecessary complexity, bad abstractions, poorly considered UX, and the like. For me, git is clearly in that bucket. I intentionally have very simple development flows. Git can do a great deal, 98% of which I not only don't…

I feel like you are pulling all sorts of nonsense out of nowhere. Your numbers seem all made up. 6GB/s seems outlandishly tiny. Your justifications are not really washing. Zen4 here shows single core as, at absolute worst behavior, dropping to 57GB/s. Basically 10x what you are spinning. You are correct in that memory limits are problematic, but we also have had technology like Intel's Direct Data IO (2011) that lets the CPU talk to peripherals without having to go through main memory at all (big security disclosure on that in 2019, yikes). AMD is making what they call "Smart Data Cache Injection" which similarly makes memory speed not gating. So even if you do divide the 80GB/s memory speed across 16 chips on desktop and look at 5GB/s, that still doesn't have to tell the whole story.
https://chipsandcheese.com/p/amds-zen-4-part-2-memory-subsystem-and-conclusion https://nick-black.com/dankwiki/index.php/DDIO

As for SSD, for most drives, it's true true that they cannot sustain writes indefinitely. They often write in SLC mode then have to rewrite, re-pack things into denser storage configurations that takes more time to write. They'll do that in the background, given the chance, so it's often not seen. But write write write and the drive won't have the time.

Thats very well known, very visible, and most review sites worth a salt test for it and show that sustained write performance. Some drives are much better than others. Even still, an Phison E28 will let you keep…

In my eyes, it is less risky with AMD. When you're rooting for the underdog, they have every incentive to help you. This isn't a battle to "win" all of AI or have one beat the other, I just need to create a nice profitable business that solves customer needs.

If I go with Nvidia, then I'm just another one of the 500 other companies doing exactly the same thing.

I'm a firm believer that there should not be a single company that controls all of the compute for AI. It would be like having Cisco be the only company that provides routers for the internet.

Additionally, we are not just AMD. We will run any compute that our customers want us to deploy for them. We are the capex/opex for businesses that don't want to put up the millions, or figure out and deal with all the domain specific details of deploying this level of compute. The only criteria we have is that it is the best-in-class available today for each accelerator. For example, I wouldn't deploy H100's because they are essentially old tech now.

> Are you not interested in AI/ML customers?

Read these blog posts and tell me why you'd ask that question...

https://chipsandcheese.com/2024/06/25/testing-amds-giant-mi300x/

https://www.nscale.com/blog/nscale-benchmarks-amd-mi300x-gpus-with-gemm-tuning-improves-throughput-and-latency-by-up-to-7-2x

> The big question then is, why are ARM desktop (and server?) cores so far behind on wider SIMD support?

Very wide SIMD instructions require a lot of die space and a lot of power.

The AVX-512 implementation in Intel's Knight's Landing took up 40% of the die area (Source https://chipsandcheese.com/p/knights-landing-atom-with-avx-512 which is an excellent site for architectural analysis)

Most ARM desktop/mobile parts are designed to be low power and low cost. Spending valuable die space on large logic blocks for instructions that are rarely used isn't a good tradeoff for consumer apps.

Most ARM server parts are designed to have very high core counts, which requires small individual die sizes. Adding very wide SIMD support would grow die space of individual cores a lot and reduce the number that could go into a single package.

Supporting 256-bit or 512-bit instructions would be hard to do without interfering with the other design goals for those parts.

Even Intel has started dropping support for the wider AVX instructions in their smaller efficiency cores as a tradeoff to fit more of them into the same chip. For many workloads this is actually a good tradeoff. As this article mentions, many common use cases of high throughput SIMD code are just moving to GPUs anyway.

Thanks for the response, this made me dig around some more in this fascinating topic.

I found multiple papers that also simply use 100ns for RAM latency (e.g. this one[1]) but in reality it is a lot more complicated than one number. E.g. the time to first word for DDR3 is just above 6ns[2]. I'm not a CPU engineer but it looks like CPUs don't necessarily wait for the entire cache line and can utilize the first word straight away.

Also, the signal needs to travel both ways and the traces are not straight lines. I'm not an electronics engineer either so the best number for the maximum DRAM trace lengths I could find are 12-30cm, which would be 1-2ns indeed. This means that there is some room for improvement (in theory).

But even with ideal RAM it is physically impossible to beat register access at 0.17ns (i9-13900KS @ 6GHz) or even L1D cache at 0.7ns[3].

What is causing the extra 4-5ns latency of the RAM? Would it be physically possible to remove it?

[1] https://www.usenix.org/system/files/conference/hotcloud18/hotcloud18-paper-thomas.pdf

[2] https://en.wikipedia.org/wiki/CAS_latency

[3] https://chipsandcheese.com/2022/11/08/amds-zen-4-part-2-memory-subsystem-and-conclusion/