3Blue1Brown
5 HN recommenders · recommended in 7 months, 2024–2026
26:10 · 4.9M views · Apr 7, 2024
3Blue1Brown's visual walkthrough of the attention mechanism inside transformers, covering self-attention, multiple heads and cross-attention. It is chapter 6 of his deep learning series.
HN treats it as the go-to visual explanation of attention, recommended to people who find text-based transformer explainers hard to follow.
8 comments from Hacker News
Because it doesn't work like how you think at all. You're still thinking it works like Chain of Thought. It doesn't. And the difference is key! It works by introducing probabilistic noise, and exploring N paths fully (each with noise) in parallel (all compressed). It's reasoning at a much, much smaller (probabilistic) level than running everything through the expensive large model (deterministic) and sometimes catching that it said, "I think 1.12 is greater than 1.9 because 12 is bigger than 9, final answer". The easiest way to think about it is: if you understand how hyper words work, it's as if it's searching for different versions of the hyper words that probilisticslly would lead to better outcomes IF it fed them to the LLM before it even does. That's not actually how it works exactly. But I think it is close enough to be helpful to understand where the gain is, a rough idea of what's happening (searching paths), and how it can potentially have huge orders of magnitude improvements (doing so without paying the full price of exploring the paths through the expensive and huge model). And also why it is so much harder to determine what it's "thinking". If you aren't familiar with hyper words, this is an amazing series: this video The general idea is that a token is a multidimensional vector to represent a word -> think like "man" is a [noun, singular, English, pronoun, masculine, contemporary, ...]. Each time is sees a new word, it mutates this word to mean some new token (ofte
This is good, but what bade me finally understand the transformer architecture [0] and attention [1], are 3Blue1Brown's videos. 0. [LINK] 1. this video
> when an LLM tells us that the sun rises in the east, it's not because it truly understands astronomy or the solar system. Rather, it has seen the phrase "sun rises in the east" repeated so many times in its training data that this becomes the highest probability answer. It's less about understanding and more about pattern recognition. This is not too different from most human person's experience with basic education ( [LINK] ). Though it is obvious that humans can do more than that too. And while "voting" that may be a close-enough metaphor for the multi-layer perceptron layers, it is not what is happening in the attention layers. I suggest checking out 3blue1brown's videos on gradient descent [LINK] and "How might LLMs store facts" [LINK] versus how attention works this video
There are many others that are better. 1/ The Annotated Transformer Attention is All You Need [LINK] 2/ Transformers from Scratch [LINK] 3/ Andrej Karpathy has really good series of intros: [LINK] Let's build GPT: from scratch, in code, spelled out. [LINK] GPT with Andrej Karpathy: Part 1 [LINK] 4/ 3Blue1Brown: But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning [LINK] Attention in transformers, visually explained | Chapter 6, Deep Learning this video Full 3Blue1Brown Neural Networks playlist [LINK]
If you want the actual maths instead of handwaving, I recommend: [LINK] For something gentler, 3Blue1Brown: this video (this is part of a series)