Tim Dettmers

nvidia specific, best write up I know: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/

Across vendors, generally, Nvidia still dominates currently. People are adding more support into ML libraries for other vendors via (second-class imo) alternate backends but expect to be patient if you're waiting for the day when there is healthy competition.

IMO, I'd say: if you can save up for it, get a 4090; if you can save up for half a 4090, get a 3090 - seen many going for 600-800 now. If you can save up for half a 3090, I'm not sure - depends on if you prefer speed or VRAM. If it were me, I'd pick more VRAM first.

re: compute capability, you can see here:

- which GPUs have what cc: https://developer.nvidia.com/cuda-gpus

- what cc comes with what features: https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#features-and-technical-specifications

I think the main qualitative change (beyond bigger numbers in the spec) for an enduser of machine learning libraries from 8.6 -> 8.9 (ie 3090 -> 4090) is this line:

> 4 mixed-precision Fourth-Generation Tensor Cores supporting fp8, fp16, __nv_bfloat16, tf32, sub-byte and fp64 for compute capability 8.9 (see Warp matrix functions for details)

ie new precisions will be builtin to eg pytorch with hw-level/tensor core support

edit: btw you probably ought to stick to a consumer gpu (ie not professional) if you want it to be generally versatile while also easy to use at home.

For a general guide, I recommend: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/

There's a subreddit r/LocalLLaMA that seems like the most active community focused on self-hosting LLMs. Here's a recent discussion on hardware: https://www.reddit.com/r/LocalLLaMA/comments/12lynw8/is_anyone_else_super_eager_to_upgrade_their/

If you're looking just for local inference, you're best bet is probably to buy a consumer GPU w/ 24GB of RAM (3090 is fine, 4090 more performance potential), which can fit a 30B parameter 4-bit quantized model that can probably be fine-tuned to ChatGPT (3.5) level quality. If not, then you can probably add a second card later on.

Alternatively, if you have an Apple Silicon Mac, llama.cpp performs surprisingly well, it's easy to try for free: https://github.com/ggerganov/llama.cpp

Current AMD consumer cards have terrible software support and IMO isn't really an option. On Windows you might be able to use SHARK or DirectML ports, but nothing will run out of the box. ROCm still has no RDNA3 support (supposedly coming w/ 5.5 but no release date announced) and it's unclear how well it'll work - basically, unless you would rather be fighting w/ hardware than playing around w/ ML, it's probably best to avoid (the older RDNA cards also don't have tensor cores, so perf would be hobbled even if you could get things running. Lots of software has been written w/ CUDA-only in mind).

This is a tough question to answer, because it depends a lot on what you want to do! One way to approach it may be to look at what models you want to run and check the amount of VRAM they need. A back-of-the-napkin method taken from here[0] is:

VRAM (GB) = 1.2 * number of parameters (in billions) * bits per parameter / 8 The 1.2 is just an estimation factor to account for the VRAM needed for things that aren't model parameters.

Because quantization is often nearly free in terms of output quality, you should usually look for quantized versions. For example, Llama 3.2 uses 16-bit parameters but has a 4-bit quantized version, and looking at the formula above you can see that will allow you to run a 4x larger model.

Having enough VRAM will allow you to run a model, but performance is dependent on a lot of other factors. For a much deeper dive into how all of this works along with price/dollar recommendations (though from last year!), Tim Dettmers wrote this excellent article: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/

Worth mentioning for the benefit of those who don't want to buy a GPU: there are also models which have been converted to run on CPU.

[0] https://blog.runpod.io/understanding-vram-and-how-much-your-llm-needs/

I think Arubis didn't push the point hard enough. Nvidia is a long way ahead of AMD in AI/ML hardware capability, from my very limited experience.

I got a Stable Diffusion thing up and running locally on a relatively old AMD RX 580 with 8GB of memory, and default settings of 512x512 with 10 (I think) passes, and it takes about 10 minutes to generate an image. On a similarly aged Nvidia 1060 with 6GB of memory that a colleague of mine uses for their local Stable Diffusion instance, it takes about 20 seconds to generate an image.

I don't know how apples and oranges this comparison is, my OS is Linux and his is Windows, for example, but 10 minutes versus 20 seconds is, for me, a massively disappointing difference.

I'm sure the gap has decreased in the intervening 6-7 years, but I don't know how much - such that I'd be opting for a second hand Nvidia over a new AMD (and I philosophically prefer AMD).

This may be a worthwhile read if you're looking to squeeze the most performance:
https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/

Thanks for extensive answer.

Do you have a few of these cutting-edge algorithmic advances papers in mind, could you list them?

I guess I got too pessimistic because of things like "emergent features" [1] / "grokking" [2] that seems to happen only with a lot of compute, and also the fact that the original (vanilla) transformer architecture remains (one of) the best, despite many additional ideas and "advances" (but that is only evident at large scale) [3].

Because of the points above, it's really hard for me, as a non-expert, to assess which papers are true advancements, and which were only published in pursuit of vanity metrics (e.g. publication counts) but actually represent overfit/cherry-picked results rather than robust progress.

[1] https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/

[2] https://www.lesswrong.com/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking

[3] https://twitter.com/YiTayML/status/1551657355036676096

Tim Dettmers [0] has been updating the linked article for a while now. It is a great primer -> deep dive into the world of GPU's for ML/DL tasks. I don't need 4x RTX 4090's but I really want them. As it stands now, I use an old M40 that I modified for water cooling. The only tricky part was configuring Win10 to run the card in WDDM mode as opposed to TCC mode without onboard graphics. To be honest, I am not entirely sure how it works as the pass-through is an older quadro card.

How do I justify the purchase of DGX-1 as a hobby programmer? Call it my mid-life crisis purchase? I mean, its not that much more than a mid-range 2023 Corvette Stingray.

[0] https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/