Theia Vogel

Great article! Agree with everything mentioned. Good to share with all tech people who don't work in the AI space and "hallucinate" about working of LLMs.

Having said that, although LLMs don't directly learn from what we say to them, they can for sure create a representation or persona using Representation Engineering.

Quoting from an excellent article by Theia Vogel [1]: "In October 2023, a group of authors from the Center for AI Safety, among others, published Representation Engineering: A Top-Down Approach to AI Transparency. That paper looks at a few methods of doing what they call 'Representation Engineering': calculating a 'control vector' that can be read from or added to model activations during inference to interpret or control the model's behavior, without prompt engineering or finetuning. (There was also some similar work published in May 2023 on steering GPT-2-XL.)

Previously on HN 3 months ago: [2]

[1] https://vgel.me/posts/representation-engineering/ [2] https://news.ycombinator.com/item?id=39414532

Fine-tuning a particular pattern of speech or bias seems self evident. The personality verbiage and focus is incredibly contrived as well. I prefer how this type of idea was approached by Theia (https://vgel.me/posts/representation-engineering/) and Anthropic (https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html), in both cases the "personality" piece is just a fun little aside to the fact that you can actively bias the LLM in a way that doesn't require retraining, allows for active adjustment, and is more immutable than just using prompts. Also one other thing, the use case in video games while not untrue seems like overkill, character prompts work really well on their own.

In this token-mania frenzy that has taken hold of the industry, I guess solutions like "soul document" and "system prompts" will continue for a while, and once the industry matures a bit we'll go back to things like LoRA[1] and control vectors[2][3].

The other explanation may be that these AI labs may be expecting more government scrutiny, and "here's a document" would probably go better than "here's some vector representation of our values" when talking to politicians.

[1] https://arxiv.org/abs/2106.09685

[2] https://vgel.me/posts/representation-engineering/

[3] https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html

There's another one that intrigued me greatly when i read about it years back. This was back when GPT-3 was state of the art. I had a lot of trouble finding it again but i did!

It's not an exact fit because the output is that of a tool rather than the model itself, but it was the first time I began to realize that just like the brain, these models have an expectation of reality that they work around. They don't necessarily 'trust' an output if it diverges significantly from this 'reality'. And that this disregard may be silent indeed.

GPT-3 will ignore tools when it disagrees with them -
https://vgel.me/posts/tools-not-needed/

thanks! we asked the model to generate some synonyms and antonyms (in this case, we have "cold" and "impassive" vs "affectionate" and "sensitive")

Then, we ask the model to behave that way (with a prompt), and store the difference in activations for each pair. Then, a PCA can be used to extract the principal component, giving use the steering vector. We do most of this using the repeng library, and the author goes into a bit more detail on how it's done on her [blog](https://vgel.me/posts/representation-engineering/#How_do_we_make_one?_Is_it_hard?)

This reminds me of control vectors, especially this line in the linked DwarfStar repo:

> y = y - scale * direction[layer] * dot(direction[layer], y)

From https://vgel.me/posts/representation-engineering/

> A control vector is a vector (technically a list of vectors, one per layer) that you can apply to model activations during inference to control the model's behavior without additional prompting