Peter Bloem

I'm the author of https://jalammar.github.io/illustrated-transformer/ and have spent years since introducing people to Transformers and thinking of how best to communicate those concepts. I've found that different people need different kinds of introductions, and the thread here includes some often cited resources including:

https://peterbloem.nl/blog/transformers

https://e2eml.school/transformers.html

I would also add Luis Serrano's article here: https://txt.cohere.com/what-are-transformer-models/ (HN discussion: https://news.ycombinator.com/item?id=35576918).

Looking back at The Illustrated Transformer, when I introduce people to the topic now, I find I can hide some complexity by omitting the encoder-decoder architecture and focusing only on one. Decoders are great because now a lot of people come to Transformers having heard of GPT models (which are decoder only). So for me, my canonical intro to Transformers now only touches on a decoder model. You can see this narrative here: https://www.youtube.com/watch?v=MQnJZuBGmSQ

not sure what 'mixing LLM models' entails but these are maybe some good starting points

- karpathy - https://www.youtube.com/watch?v=kCc8FmEb1nY

- https://towardsdatascience.com/beautifully-illustrated-nlp-models-from-rnn-to-transformer-80d69faf2109

- https://dzone.com/articles/a-deep-dive-into-the-transformer-architecture-the

- https://peterbloem.nl/blog/transformers

- http://nlp.seas.harvard.edu/2018/04/03/attention.html

- https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/

- https://blog.quickchat.ai/post/tokens-entropy-question/

- https://dugas.ch/artificial_curiosity/GPT_architecture.html

- https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/

- https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

- https://arxiv.org/pdf/2005.14165.pdf

- https://arxiv.org/pdf/2303.08774.pdf

- https://arxiv.org/pdf/2303.17564.pdf

For a while now, an answer I've seen is to start with "Attention Is All You Need", the original Transformers paper. It's still pretty good, but over the past year I've led a few working sessions on grokking transformer computational fundamentals and they've turned up some helpful later additions that simplify and clarify what's going on.

You can quickly get overwhelmed by the million good resources out there so I'll keep it to these three. If you have a strong CS background, they'll take you a long way:

(1) Transformers from Scratch: https://peterbloem.nl/blog/transformers

(2) Attention Is All You Need: https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html

(3) Formal Algorithms for Transformers: https://arxiv.org/abs/2207.09238

Transformers from scratch:

1.) https://e2eml.school/transformers.html

https://news.ycombinator.com/item?id=35697627 (46 comments, 1 month ago)

https://news.ycombinator.com/item?id=29315107 (17 comments, 2 years ago)

2.) http://www.peterbloem.nl/blog/transformers

https://news.ycombinator.com/item?id=20773992 (28 comments, 4 years ago)

https://news.ycombinator.com/item?id=29280909 (9 comments, 2 years ago)

Source: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=Transformers%20from%20scratch%20&sort=byPopularity&type=story

you can catch up very quickly.

great resource on transformers, which underpin the openai breakthroughs (the "T" in GPT-3 stands for transformer): http://peterbloem.nl/blog/transformers

great reddit sub for tracking ml research and trends: https://www.reddit.com/r/machinelearning

great email newsletter on ml from andrew ng: https://www.deeplearning.ai/thebatch/

also, someone must maintain old code or things break. thanks for your contributions.

In case you find Attention (and specially transformers) interesting I have some saved links introducing the topic:

http://www.peterbloem.nl/blog/transformers

https://nostalgebraist.tumblr.com/post/185326092369/the-transformer-explained

https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf

http://jalammar.github.io/illustrated-transformer/

https://arxiv.org/pdf/1807.03819.pdf