Interconnects

There are multiple stages of training, and the data/compute mix at each are quite different and produce different "layers" of intelligence.

The pretraining stage is the first stage which consists of "next token prediction" on the entire internet, PB of tokens, etc. This is what most people think of when they think of training LLMs, however it produces a "base model" which is not really "intelligent", but rather much like a blurry JPEG of all human language and knowledge. You cannot really talk to such a model; it will simply complete your prompt by producing both sides of the conversation. Note however at some level the training has encoded enough structure through compression that it is able to simulate all sorts of phenomena, from human conversations to code.

The next few stages are collectively called post-training, and typically consist of supervised fine-tuning, then reinforcement learning.

In supervised fine-tuning, the model is further trained to predict the next token, but on a much more focused data set of natural language conversations where the "assistant" and "user" turns are explicitly delineated with special tokens. The output of this stage is a model which is capable of carrying on proper conversations, but typically with no ability to creatively problem-solve, and less of a personality. The data and compute are many orders of magnitude smaller than in pretraining.

The reinforcement learning stage used to be a small part of model training, but ever since…

I am fairly optimistic about LLMs as a human math -> theorem-prover translator, and as a fan of Idris I am glad that the AI community is investing in Lean. As the author shows, the answer to "Can AI be useful for automated mathematical work?" is clearly "yes."

But I am confident the answer to the question in the headline is "no, not for several decades." It's not just the underwhelming benchmark results discussed in the post, or the general concern about hard undergraduate math using different skillsets than ordinary research math. IMO the deeper problem still seems to be a basic gap where LLMs can seemingly do formal math at the level of a smart graduate student but fail at quantitative/geometric reasoning problems designed for fish. I suspect this holds for O3, based on one of the ARC problems it wasn't able to solve: https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90759a16-3602-4aa8-ba8c-69f1d67c31f1_1147x638.webp (via https://www.interconnects.ai/p/openais-o3-the-2024-finale-of-ai) ANNs are simply not able to form abstractions, they can only imitate them via enormous amounts of data and compute. I would say there has been zero progress on "common sense" math in computers since the invention of Lisp: we are still faking it with expert systems, even if LLM expert systems are easier to build at scale with raw data.

It is the same old problem where an ANN can attain…

That quote is straight up wrong to claim Deepseek has "loathed to reveal their secret sauce".

The source of all the excitement is exactly how much they revealed, and I feel like that thread as a whole emphasizes why people who aren't deeply familiar with the pipeline should not get to define these things.

"Nick" claims the secret sauce is in something not provided, then posts an article that demonstrates the exact "secret sauce", seemingly not making the connection: https://www.interconnects.ai/p/deepseek-r1-recipe-for-o1

-

There is a lot of detail about the nature of the data used and the exact steps needed to reproduce their findings with your own data. They even provide R1-Zero to demonstrate things that might be dead ends just in case someone can continue them. That should be enough to satisfy any useful definition of open source.

Even in the same thread you linked:

> Just a curiosity, according to the Model Openness Framework from the Linux Foundation, DeepSeek-R1 classifies as an Open Model:

> https://mot.isitopen.ai/model/1143

At the end of the day this is as good as it needs to be for LLMs: By their nature a lot of data being used to train them cannot or should not be openly shared, but the shape and motivations behind the data used are able to push others very far along the way to reproduction and iteration.

This is one area where reinforcement learning can help.

The way you should think of RL (both RLVR and RLHF) is the "elicitation hypothesis[1]." In pretraining, models learn their capabilities by consuming large amounts of web text. Those capabilities include producing both low and high quality outputs (as both low and high quality outputs are present in their pretraining corpora). In post training, RL doesn't teach them new skills (see E.G. the "Limits of RLVR"[2] paper). Instead, it "teaches" the models to produce the more desirable, higher-quality outputs, while suppressing the undesirable, low-quality ones.

I'm pretty sure you could design an RL task that specifically teaches models to use modern idioms, either as an explicit dataset of chosen/rejected completions (where the chosen is the new way and the rejected is the old), or as a verifiable task where the reward goes down as the number of linter errors goes up.

I wouldn't be surprised if frontier labs have datasets for this for some of the major languages and packages.

[1] https://www.interconnects.ai/p/elicitation-theory-of-post-training

[2] https://limit-of-rlvr.github.io

"we have information" says a US Government official who almost certainly has had Anthropic and/or OpenAI on the phone spinning him stories.

See also, don't trust anyone in Trump's government who says "we have information".

"they distilled us" is fast becoming standard US FUD.

The same as people telling me with a serious face that the Chinese models are distilled just because it says "I am Claude".

I am not the only one, look at this post on interconnects about Kimi K3 for example:[1]

It should be clear looking at this model that if adversarial distillation from the closed frontier models in the U.S. contributed, it is at most to a relatively small degree. AI observers who followed the distillation panic and came away with the wrong conclusion that Chinese AI labs are only producing good models due to IP theft are in for an awakening – that Chinese companies are extremely good at building models in the same way the leading American companies are.

[1] https://www.interconnects.ai/p/kimi-k3-the-open-weights-escalation

Something that's really hitting me is something brought up in this piece:

https://www.interconnects.ai/p/gpt-5-and-bending-the-arc-of-progress

When a model comes out, I usually think about it in terms of my own use. This is largely agentic tooling, and I mostly us Claude Code. All the hallucination and eval talk doesn't really catch me because I feel like I'm getting value of these tools today.

However, this model is not _for_ me in the same way models normally are. This is for the 800m or whatever people that open up chatgpt every day and type stuff in. All of them have been stuck on GPT-4o unbeknwst to them. They had no idea SOTA was far beyond that. They probably dont even know that there is a "model" at all. But for all these people, they just got a MAJOR upgrade. It will probably feel like turning the lights on for these people, who have been using a subpar model for the past year.

That said I'm also giving GPT-5 a run in Codex and it's doing a pretty good job!