Sebastian Raschka

Here are some high level answers:

"7B" refers to the number of parameters or weights for a model. For a specific model, the versions with more parameters take more compute power to train and perform better.

A foundational model is the part of a ML model that is "pretrained" on a massive data set (and usually is the bulk of the compute cost). This is usually considered the "raw" model after which it is fine-tuned for specific tasks (turned into a chatbot).

"8K length" refers to the Context Window length (in tokens). This is basically an LLM's short term memory - you can think of it as its attention span and what it can generate reasonable output for.

"1.5T tokens" refers to the size of the corpus of the training set.

In general Wikipedia (or I suppose ChatGPT 4/Bing Chat with Web Browsing) is a decent enough place to start reading/asking basic questions. I'd recommend starting here: https://en.wikipedia.org/wiki/Large_language_model and finding the related concepts.

For those going deeper, there are lot of general resources lists like https://github.com/Hannibal046/Awesome-LLM or https://github.com/Mooler0410/LLMsPracticalGuide or one I like, https://sebastianraschka.com/blog/2023/llm-reading-list.html (there are a bajillion of these and you'll find more once you get a grasp on the terms you want to surf for). Almost everything is published on arXiv, and most is fairly readable even as a layman.

For non-ML programmers looking to get up to speed, I feel like Karpathy's…

This is a very good question and the answers are different for different models.

---

Deepseek v4 Pro has three modes: no thinking, thinking ("high"), very good thinking ("max"). Max is implemented as follows:

https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/encoding/README.md#reasoning-effort

> When reasoning_effort="max" is set, a special prefix is prepended at the very beginning of the prompt (before the system message) to instruct the model to maximize its reasoning depth:
>
> Reasoning Effort: Absolute maximum with no shortcuts permitted.
You MUST be very thorough in your thinking and comprehensively decompose the problem to resolve the root cause, rigorously stress-testing your logic against all potential paths, edge cases, and adversarial scenarios.
Explicitly write out your entire deliberation process, documenting every intermediate step, considered alternative, and rejected hypothesis to ensure absolutely no assumption is left unchecked.

---

Inkling, apparently, trained their model to treat effort level as "how cheap is one reasoning token":

https://magazine.sebastianraschka.com/p/controlling-reasoning-effort-in-llms

> During large-scale RL, they did two things for each sample:
>
> 1. Specified the desired effort level in the system message.
> 2. Adjusted the cost assigned to each generated token.
> [...]
>
> Then, at inference time, Inkling receives a system message such as Thinking effort level: 0.8, and adjusts its token usage accordingly.

I'm not sure what is the state-of-the-art in sentiment analysis. However, GPT-3.5 and GPT-4 work great for it with few shot learning. You give a representative set of examples in the prompt and it works pretty well.

On the topic of classification, large language models such as BERT are the state-of-the-art. But, there was a paper recently at the ACL [1] which showed that a simple approach of using gzip compression + k-nearest neighbour achieved similar level of performance. Later on, their KNN evaluation methodology came under question because instead of doing KNN with k=2 they reported accuracy based on top-2. Even then, what it shows is that we don't need LLMs for simple tasks such as classification, where traditional techniques still work well and are much cheaper to run. I'm sure some people might say that we should just discard that paper because it had that error. But Sebastian Raschka ran it on the IMDB movie reviews dataset and reported 71% accuracy [3]. I wrote an explainer article on it [2].

[1] https://aclanthology.org/2023.findings-acl.426.pdf [PDF]

[2] https://codeconfessions.substack.com/p/decoding-the-acl-paper-gzip-and-knn

[3] https://magazine.sebastianraschka.com/p/large-language-models-and-nearest

This is amazing, I just spent some time scrolling through these, most of the evolution is about inference cost not capability. GQA, MoE routing, sliding window attention, all trading theoretical capacity for practical efficiency.

Tbh might be the last generation of architectures designed entirely by humans. I dug into that (1) and might add another paragraph based on this if I find the time. The Big LLM Architecture Comparison (2) by Sebastian Raschka already inspired my ograph image for the blog -thanks again!

(1) https://philippdubach.com/posts/the-last-architecture-designed-by-hand/

(2) https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison

It's quadratic, because of the dot product in the attention mechanism.

You can use K-V Caching to get rid of a lot of the quadratic runtime that comes from redundant matrix multiplications, but after you have cached everything, you still need to calculate the dot product k_i * q_j with i,j being index of the tokens. With n tokens, you will get O(n*n).

But you have to remember that this is only n^2 multiplications. It's not exactly the end of the world at context sizes of 32k, for example. It only gets nasty in the hundred thousands to millions.

Here is the source I used: https://sebastianraschka.com/blog/2023/self-attention-from-scratch.html

Why didn't this author compare Llama 3 with GLM 5.2 (released 1 week ago) which is a more standard attention based LLM? To compare 2 separate families of LLMs and then pointing out that they are different is not a surprising result and detracts from the point the author is trying to make.

https://sebastianraschka.com/llm-architecture-gallery/?compare=llama-3-8b%2Cglm-5-2#architecture-diff-tool

If you look at it, the diagrams are very similar, but the main differences are that the feedforward is replaced with a MoE (router to multiple feedforwards) and the model has a different attention implementation.