Brendan Bycroft

I think you are confused on how transformers work. You are confusing between static embeddings and the dynamic, contextual outputs calculated by transformers. While initial embeddings (representations BEFORE processing through the transformer layers) are relatively static, the output embeddings from each layer of the transformer are highly dynamic and context-dependent. They incorporate information from the entire input sequence via the attention mechanisms.

So when transformers process an input sequence, they don't just look at each token in isolation but consider the ENTIRE sequence's context through complex inter-token relationships. This makes the technique of caching outputs non-trivial and context-specific. The sequential processing does not imply independence of tokens but underscores the integral role of context and sequence in generating accurate and coherent outputs.

This should help you understand better how transformers work: https://bbycroft.net/llm

Focusing on Deep Learning specifically:
- Most LLMs currently use the transformer architecture. You can learn about this visually (https://bbycroft.net/llm), or through this blog post (https://jalammar.github.io/illustrated-transformer/), or through any number of Andrej Karpathy's blog posts and materials.
- To stay on top of papers that get published every week, I read a summary every Sunday: https://github.com/dair-ai/ML-Papers-of-the-Week
- To learn more about the engineering side of it, you can join Discord servers such as EleutherAI's, or follow GitHub discussions of projects like llama.cpp

Personally I think the best way to develop per unit time is probably to try to re-implement some of the big papers in the field. There's a clear goal, there are clear signs of success, there are many implementations out there for you to check your work against and compare and learn from.

Good luck!

Amazing work! Reminded me of LLM Visualization (https://bbycroft.net/llm) except this is a lot easier to wrap my head around and that I can actually run the training loops, which makes sense given the simplicity of the original microgpt.

To give a sense of what the loss value means, maybe you can add a small explainer section as a question and add this explanation from Karpathy’s blog:

> Over 1,000 steps the loss decreases from around 3.3 (random guessing among 27 tokens: −log(1/27)≈3.3) down to around 2.37.

to reiterate that the model is being trained to predict the next token out of 27 possible tokens and is now doing better than the baseline of random guess.

Innovation in data visualization? From a purely utilitarian view, the purpose of data visualization is to present data in a way people can understand it. If you’re constantly changing the method of visualizing the same thing it’s harder to do that. Sometimes a bar chart is best.

As far as cool visualizations go (that are better served as nonstandard visualizations) there are two recent ones that come to mind:

https://youtu.be/TkwXa7Cvfr8 (Especially around 16:56)

https://bbycroft.net/llm