Name: Build a Large Language Model
ISBN: 9781633437166

Build a Large Language Model

From the back cover: Build a Large Language Model (From Scratch) is a practical and eminently-satisfying hands-on journey into the foundations of generative AI. Without relying on any existing LLM libraries, you'll code a base model, evolve it into a text classifier, and ultimately create a chatbot that can follow your conversational instructions. And you'll really understand it because you built it yourself! About the reader: Readers need intermediate Python skills and some knowledge of machine learning. The LLM you create will run on any modern laptop and can optionally utilize GPUs.

What Hacker News says

antonvsOct 2026

Glad it was helpful. If you have time for some extracurricular study, I'd highly recommend implementing an LLM. Implementing a toy version is really not that difficult, and can be very illuminating.

There are quite a few resources on the web that can help with this. One good one is Andrej Karpathy's "Let's build GPT: from scratch, in code, spelled out.": https://www.youtube.com/watch?v=kCc8FmEb1nY

A somewhat more intensive one is "Build a Large Language Model (From Scratch)" by Sebastian Raschka: https://www.sebastianraschka.com/llms-from-scratch/

And if you really want to go crazy, Stanford's "CS336: Language Modeling from Scratch" is online: https://cs336.stanford.edu/

(The last one is a full graduate-level course, so don't let it detract from your actual master's program!)

thread

malsheSep 2026

Sebastian is the author of two excellent books related to LLMs

Build a Large Language Model (From Scratch): https://sebastianraschka.com/llms-from-scratch/

Build a Reasoning Model (From Scratch): https://sebastianraschka.com/reasoning-from-scratch/

thread

dan-gAug 2026

Similarly, I’ve been working through Raschka’s Build a Large Language Model (From Scratch) https://sebastianraschka.com/llms-from-scratch/ and it’s been well worth it (particularly paired with his explainer videos)

thread

LatencyKillsJun 2026

Not OP but I worked through Sebastian Raschka's "Build a Large Language Model (From Scratch)" [0] and Raj Abhijit Dandekar's "Build a DeepSeek Model (From Scratch)" [1] books.

I don't think there is anything in a transformer I couldn't explain in the smallest detail now.

[0]: https://www.amazon.com/Build-Large-Language-Model-Scratch/dp...

[1]: https://www.amazon.com/Build-DeepSeek-Scratch-Abhijit-Dandek...

thread

aurahamFeb 2026

I highly recommend Build a large language model from scratch [1] by Sebastian Raschka. It provides a clear explanation of the building blocks used in the first versions of ChatGPT (GPT 2 if I recall correctly). The output of the model is a huge vector of n elements, where n is the number of tokens in the vocabulary. We use that huge vector as a probability distribution to sample the next token given an input sequence (i.e., a prompt). Under the hood, the model has several building blocks like tokenization, skip connections, self attention, masking, etc. The author makes a great job explaining all the concepts. It is very useful to understand how LLMs works.

[1] https://www.manning.com/books/build-a-large-language-model-f...

thread

libraryofbabelJan 2026

This is ok (could use some diagrams!), but I don't think anyone coming to this for the first time will be able to use it to really teach themselves the LLM attention mechanism. It's a hard topic and requires two or three book chapters at least if you really want to start grokking it!

For anyone serious about coming to grips with this stuff, I would strongly recommend Sebastian Raschka's excellent book Build a Large Language Model (From Scratch), which I just finished reading. It's approachable and also detailed.

As an aside, does anyone else find the whole "database lookup" motivation for QKV kind of confusing? (in the article, "Query (Q): What am I looking for? Key (K): What do I contain? Value (V): What information do I actually hold?"). I've never really got it and I just switched to thinking of QKV as a way to construct a fairly general series of linear algebra transformations on the input of a sequence of token embedding vectors x that is quadratic in x and ensures that every token can relate to every other token in the NxN attention matrix. After all, the actual contents and "meaning" of QKV are very opaque: the weights that are used to construct them are learned during training. Furthermore, there is a lot of symmetry between Q and K in the algebra, which gets broken only by the causal mask. Or do people find this motivation useful and meaningful in some deeper way? What am I missing?

[edit: on this last question, the article on "Attention is just Kernel Smoothing" that roadside_picnic posted below looks really interesting in terms of giving a clean generalized mathematical approach to this, and also affirms that I'm not completely off the mark by being a bit suspicious about the whole hand-wavy "database lookup" Queries/Keys/Values interpretation]

thread