David Noel Ng writes about hacking LLMs, building home AI hardware, and experiments on how language models work internally.
Commenters recommend his RYS layer-repeating post as an easy way into LLM hacking, and discuss his Sapir-Whorf experiments on model layers.
You're right that my quote was misleading, I overlooked "the weird part" in the post because it didn't seem new to me either.
Here's the section in the original post that covers it: https://dnhkng.github.io/posts/rys/#the-brain-scanner All heatmaps are split by tasks and show an optimal point for each. The resulting routing he chose is a trade-off for both tasks, there isn't much else to do unless you intend to train a router anyway.
> So the ‘math organ’ has boundaries on both sides. Too few layers and you get nothing — you’ve cut into the circuit and it can’t complete its operation. Too many layers and you also get nothing — you’ve included tissue from a neighbouring circuit that doesn’t belong. Pre-training carved these structures out of the layer stack, and they only work whole. It also doesn’t translate to other tasks, as the heatmap for EQ scores doesn’t have this patch.
Yes you can! Well, mostly, depends on how pedantic you are with definitions: you can transplant layers but not weights, which in common parlance are conceptually similar. But usually it isn’t a good idea for a few reasons.
There’s a really fascinating example[1] where a guy identifies a particular set of layers and transplants them. Overgeneralizing, early layers are encoders and the later layers are decoders and in the middle some blocks seem to do specific things or tasks related “reasoning”. So you can actually create a FrankenLLM and it sometimes works.
This needs architectures to be roughly similar however and internal representations to be consistent-ish so for “stealing” it’s not really a thing (other practical concerns aside)
[1] https://dnhkng.github.io/posts/rys/
Found it: https://news.ycombinator.com/item?id=47500709
Part 3 might be the best introduction: https://dnhkng.github.io/posts/sapir-whorf/
tl;dr: Based on experiments with similar prompts translated to different languages LLM layers group into three phases: the first decodes from the source language into an abstract space, the middle does something, then there's a last part where the abstract result gets transformed back to the target language. And you can repeat the middle to get a stronger model. Which neatly fits Anthropic's findings here that something similar to CoT is happening in those middle layers
Three months ago. I wonder if Anthropic's J-Space research was actually inspired by those blog posts
Conceptually, this seems a good direction.
The other piece that has always struck me as a huge inefficiency with current usage of LLMs is the hoops they have to jump through to make sense of existing file formats - especially making sense of (or writing) complicated semi-proprietary formats like PDF, DOC(X), PPT(X), etc.
Long-term prediction: for text, we'll move away from these formats and towards alternatives that are designed to be optimal for LLMs to interact with. (This could look like variants of markdown or JSON, but could also be Base64 [0] or something we've not even imagined yet.)
[0] https://dnhkng.github.io/posts/rys/
You can also edit binary distributions of models with means besides changing their weights. See "LLM Neuroanatomy: How I Topped the LLM Leaderboard Without Changing a Single Weight."
On the spectrum of:
careful engineering--hacking--mad science
This kind of thing falls far towards the mad science end of the scale, but has proven effective.
https://dnhkng.github.io/posts/rys/
I keep thinking of the RYS (Repeat Yourself) experiment of simply looping some of the inner layers of LLMs for better results and wonder if any progress was made on it.
https://dnhkng.github.io/posts/rys/
Feels it should be straightforward to integrate in LLMs a network to control the looping. Or just duplicate entire blocks of layers after the initial training.