Name: Natural Language Processing with Transformers
ISBN: 9781098136796

What Hacker News says

gmwhitebox_devApr 2025

Thank you!

To approach it from first principles, I didn’t really follow any one specific tutorial or course. I really just started from the bottom up. Starting with my Tensor module, I familiarized myself with the math operations, backprop, computation graphs, autodiff… etc. That was the hardest part honestly, but it set the foundation for everything else in the system.

Once I had that working, the rest (activations, loss functions, optimizers, layers, transformers) started to make a lot more sense. Writing it all myself gave me full control, removed the abstraction, and helped me to really internalize how each part of the system fits together, and why it works - not just how.

Here’s some resources I found helpful, I also have some links to additional resources in the project readme:

- Deep Learning Foundations and concepts: Book by Christopher Bishop. Mainly covers theory and statistical ML.

- Natural Language Processing with Transformers: Book by Lewis Tunstall, Leandro von Werra, and Thomas Wolf (Hugging Face). Good for understanding real world NLP/ LLMs.

- UvA Deep Learning Tutorials: Website for building and understanding DL modules, has a lot of project-based notebooks (tutorial 7 on GNNs was very helpful).

- Deep Learning: Book by Goodfellow, Bengio & Courville. Covers a lot of foundational theory and math.

- Stanford’s CS231 Course: This is fully available online, with lecture videos and coding walk-throughs. Super helpful for learning about backprop, CNNs, and deep nets, etc.

- The Annotated Transformer: Website by Harvard NLP. This goes over the ‘Attention is All You Need’ paper in an understandable way with example code

- codingvidya.com: A good aggregator for finding ML books and learning resources.

- Andrej Karpathy’s YouTube channel, and his micrograd and tinygrad Repos are super helpful resources, especially when learning by building from scratch.

I’m still learning as I go, but I’m happy to share what’s worked for me so far! Hope this helps!

thread

moyixApr 2022

I have been enjoying Natural Language Processing with Transformers [1]. It's largely focused on the Huggingface library, but Chapter 3 has a very nice walkthrough that builds up the encoder portion of an encoder-decoder Transformer from "scratch" (it still uses some primitives found in PyTorch like nn.Embedding). The decoder portion is covered in less depth and they instead refer folks to Karpathy's awesome minGPT [2], which implements a decoder-only (GPT-style) Transformer in ~300 lines of nicely-commented Python+PyTorch code.

For a higher-level conceptual view of how Transformers work, you can check out the now-classic "Illustrated Transformer" series [3] and this programmer-oriented explanation (with code in Rust) from someone at Anthropic [4].

[1] https://www.oreilly.com/library/view/natural-language-proces...

[2] https://github.com/karpathy/minGPT

[3] https://jalammar.github.io/illustrated-transformer/

[4] https://blog.nelhage.com/post/transformers-for-software-engi...

thread

mindcrimeOct 2024

There are a couple of new books on the topic that are slated to drop any day now, IIRC. Of what's already published, a few I'm familiar with include:

Transformers for Natural Language Processing and Computer Vision: Explore Generative AI and Large Language Models with Hugging Face, ChatGPT, GPT-4V, and DALL-E 3

https://www.amazon.com/gp/product/1805128728/

Transformer, BERT, and GPT: Including ChatGPT and Prompt Engineering

https://www.amazon.com/gp/product/1683928989/

Introduction to Transformers for NLP: With the Hugging Face Library and Models to Solve Problems

https://www.amazon.com/gp/product/1484288432

Transformers for Machine Learning: A Deep Dive

https://www.amazon.com/gp/product/0367767341/

Natural Language Processing with Transformers, Revised Edition

https://www.amazon.com/gp/product/1098136799

EDIT:

a couple of the ones that I thought were still pending have now been released. I haven't read any of these, but they are ones that caught my eye and that I was planning to get:

Large Language Models: A Deep Dive: Bridging Theory and Practice

https://www.amazon.com/gp/product/3031656466

Building LLMs for Production: Enhancing LLM Abilities and Reliability with Prompting, Fine-Tuning, and RAG

https://www.amazon.com/gp/product/B0D4FFPFW8

Building LLM Powered Applications: Create intelligent apps and agents with large language models

https://www.amazon.com/gp/product/1835462316

The "not yet released" group still includes:

Build a Large Language Model (From Scratch) (ships Oct. 29th)

https://www.amazon.com/gp/product/1633437167

LLM Engineer's Handbook: Master the art of engineering Large Language Models from concept to production (ships Nov. 11th)

https://www.amazon.com/gp/product/1836200072

Hands-On Large Language Models: Language Understanding and Generation (ships ???)