Jay Alammar writes illustrated visual explanations of machine learning models, including transformers, BERT, GPT-2, word2vec and Stable Diffusion.
Commenters recommend it as a favorite sequence of tutorials from basic neural network math to GPT-2, and as human-made explainers worth reading.
Am I alone in really disliking Towards Data Science?
While their articles always look nice, their content is all written quickly by data scientists wanting to polish their resume with the ultimate aim of rapidly generating content for TDS that will match every conceivable data science related search. This post clearly exists solely so that TDS can get the top spot for "Word2vec explained" (which they have). As evidence of this tactic you can see that there already is a TDS post "Word2vec made easy" [0], offering nothing substantially different than this one.
The problem is that content is almost never useful, it just looks nice at first skim through. The authors, at no real fault of their own, are just eager novices that rarely have new perspective to add to a topic. It's not uncommon to find huge conceptual errors (or at least gaps) in the content there.
I personally encourage everyone at every level to write about what they can, but the issue is that TDS has manipulated this population of eager data scientists in order to dominant search results on nearly every single topic they can cover related to DS, which has made searching for anything tedious.
Compare this post to the fantastic work of Jay Alammar [1]. Jay's post is truly excellent, covering a lot of interesting details about word2vec and providing excellent visuals as well.
I'm assuming TDS will fold as soon as DS stops being a "hot" topic (which I think we'll be in the relatively near future), and will…
Content marketing ruined blogging, just like SEO ruined search.
I used to work with a lot of content marketers when it started to get hot. SEO was doing particularly poorly for a period (Google used to resist seo rather than reward it) and people realized that if you started writing "authentic" content, then people would organically link to it and it would increase your ranking.
Then social media furthered this problem by people mindlessly sharing content they never read but just looked engaging.
Marketing robots where just churning out content on blogs because eventually you would just hit search engine gold by nail the correct topic for some long tail search term. In my own domain, the atrocious "Towards Data Science" is a great example of this (except their "brilliant" innovation was to replace marketing robots with people eager to pad their resume). The content is largely garbage, but it looks about right and is easily shareable.
Today, whether you like it or not, if you want to make people to see your blog at all you have to think at least a bit like a content marketer to get seen at all. Consider Jay Alammar's fantastic blog [0], which is very high quality content, but still requires a tremendous amount of polish and marketing to survive. All of this has made it impossible to create the network required for a "global conversation among creative weirdos". Real content does get made, but has to be produced with the effort of a real marketer.
0.…
I've just recently been on a journey to understand transformers in and out, here are the resources I've found managed to drill it into my head:
1. Chapters 7, 9, 10
https://web.stanford.edu/~jurafsky/slp3/
This was really useful to really build up to the concepts of attention (although the actual attention section is still brief).
2. https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/
This was great for visualization and understanding that attention wasn't exclusively for transformers and actually was for RNNs first.
3. https://jalammar.github.io/illustrated-transformer/
Getting to understand how transformers actually work visually.
4. https://www.youtube.com/watch?v=S27pHKBEp30
This lecture by Leo Dirac was extremely helpful to finish off with, not only because it actually includes some pseudocode but it also revisits some key topics and covers why transformers are needed.
One of the big confusion points for me was that the concept of ATTENTION and SELF-ATTENTION are not the same thing.
Hope this helps.
I have been enjoying Natural Language Processing with Transformers [1]. It's largely focused on the Huggingface library, but Chapter 3 has a very nice walkthrough that builds up the encoder portion of an encoder-decoder Transformer from "scratch" (it still uses some primitives found in PyTorch like nn.Embedding). The decoder portion is covered in less depth and they instead refer folks to Karpathy's awesome minGPT [2], which implements a decoder-only (GPT-style) Transformer in ~300 lines of nicely-commented Python+PyTorch code.
For a higher-level conceptual view of how Transformers work, you can check out the now-classic "Illustrated Transformer" series [3] and this programmer-oriented explanation (with code in Rust) from someone at Anthropic [4].
[1] https://www.oreilly.com/library/view/natural-language-processing/9781098103231/
[2] https://github.com/karpathy/minGPT
[3] https://jalammar.github.io/illustrated-transformer/
[4] https://blog.nelhage.com/post/transformers-for-software-engineers/
I agree that Karpathy's YouTube video is an excellent resource for understanding Transformers from scratch. It provides a hands-on experience that can be particularly helpful for those who want to implement the models themselves. Here's the link to the video titled "Let's build GPT: from scratch, in code, spelled out": https://youtu.be/kCc8FmEb1nY
Additionally, for more comprehensive resources on Transformers, you may find these resources useful:
* The Illustrated Transformer by Jay Alammar: http://jalammar.github.io/illustrated-transformer/
* MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention: https://www.youtube.com/watch?v=ySEx_Bqxvvo
* Karpathy's course, Deep Learning and Generative Models (Lecture 6 covers Transformers): https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9Gv...
These resources cover different aspects of Transformers and can help you grasp the underlying concepts and mechanisms better.
Focusing on Deep Learning specifically:
- Most LLMs currently use the transformer architecture. You can learn about this visually (https://bbycroft.net/llm), or through this blog post (https://jalammar.github.io/illustrated-transformer/), or through any number of Andrej Karpathy's blog posts and materials.
- To stay on top of papers that get published every week, I read a summary every Sunday: https://github.com/dair-ai/ML-Papers-of-the-Week
- To learn more about the engineering side of it, you can join Discord servers such as EleutherAI's, or follow GitHub discussions of projects like llama.cpp
Personally I think the best way to develop per unit time is probably to try to re-implement some of the big papers in the field. There's a clear goal, there are clear signs of success, there are many implementations out there for you to check your work against and compare and learn from.
Good luck!