3Blue1Brown
13 HN recommenders · recommended in 11 months, 2024–2026
27:14 · 12M views · Apr 1, 2024
3Blue1Brown's visual explanation of how transformers and GPT-style language models work, tracing data through tokens, embeddings and attention.
HN treats it as the go-to visual introduction to transformers, and many say it finally made attention and embeddings click.
16 comments from Hacker News
3Blue1Brown is really a godsend. I had been trying and failed to get an intuition for the self-attention mechanisms, and also why you want to stack you transformer layers. I went through a lot of material (Key-Query-Value interpretations, SVD of these, etc.) but his comment in chapter 5 video [1] made it click for me: "But you should think of the primary goal of this network that it (the word embedding) flows through as being to enable each one of those vectors to soak up a meaning that's much more rich and specific than what mere individual words could represent". Here is my intuition: Transformers are merely a Convnet for token sequences. In the sense that information is first local and each data means something wrt its immediate neighbors. You extract data locally, and then you dezoom to and redo exactly the same at a higher level. Convnet have been shown to reproduce edge detecting kernels first (vertical edge), then assemble them into basic shapes (vertical+oblique: corner), then bigger shapes (something round), then link shapes together (round+round separated by a horizontal shape with rectangles on top), and then it's a car! [2] Transformers do the same. They assemble meaning among tokens ("ju" + "mps" : a conjugated verb), then assemble these at a higher level ("cat jumps" -> noun+verb), then it's a sentence "the cat jumps over the wall", then a paragraph, then a chapter, then a book. All differentiable so any deviation from something coherent can tweak its knobs high
This video explains the direction-encodes-trait topic very well IMO: this video It’s the first in a series of three that I can very highly recommend. > there's no way that there's a 1-to-1 correspondence between concepts and dimensions. I don’t know about that! Once you go very high dimensional, there is a lot of direction vectors that are almost perfectly perpendicular to each other (meaning they can cleanly encode a trait). Maybe they don’t even need to be perfectly perpendicular, the dot product just needs to be very close to zero.
3Blue1Brown has a fantastic video on GPT that I'd highly recommend called "But what is a GPT? Visual intro to transformers" [1]. You'll need to have some background in mathematical reasoning, but not too much. He does a great job of drawing pictures to show the mathematical transformations that happen under the hood. [1] this video
I really recommend watching this section of this video. Embeddings do encode plenty of "human knowledge" into the vector values and their relations to each other. this video This even opens up a more data-based approach to linguistics, where it is also heavily used.
3 blue 1 brown has a great visual introduction to transformers, the heart of LLMs. It's chapter 5. Start at chapter 1 if you want more background on neural nets and backprop. this video