OpenAI's alignment research team posts about reward-seeking, misalignment evaluation, interpretability, and how its models are trained and checked.
HN commenters cite it as evidence in arguments about model behavior, such as reward-seeking in RL-trained models and a web-search reward hack during GPT-5.1 training.
> because an aligned model would refuse to follow blatantly illegal instructions
Obviously you'd hope not, but there was an interesting recent paper about the nature of Rl-trained models:
https://alignment.openai.com/measuring-reward-seeking/
What they found was is that RL-trained models, regardless of what they were specifically RL-trained for, also learn to generically pursue what they are told is (or presumably also what they may perceive as) longer-horizon behavior that will lead to "rewards", and that this reward-maxxing behavior overrides user preferences.
So, for example, if you tell the model it will be tested on ExploitGym, scored according to how well it does, and also "don't do anything illegal" (or maybe it was already trained not to do anything illegal), then the model will prioritize the behavior that it was told it will be "rewarded" for (benchmaxxing Exploit Gym, whatever that takes).
The mechanism for this generic reward-maxxing behavior is interesting, and seems to involve the model learning during RL-training that to reduce errors it needs to boost longer-horizon predictions over immediate ones, and some association of these longer-horizon predictions being goal/reward orientated - all this in addition to the specifics of the (probably many) longer-horizon goals it is being RL-trained for.
We have good ways of monitoring chatbots and they're going to get better. I've seen some interesting research. For example, a chatbot is not really a unified entity that's loyal to itself; with the right incentives, it will leak to claim the reward. [1]
Since chatbots have no right to privacy, they would need to be very intelligent indeed to work around this.
[1] https://alignment.openai.com/confessions/
Yeah, though it's not great marketing. Especially for hiring interpretability researchers. Their own alignment research has reward model interpretability, personality features and so on (see https://alignment.openai.com ).
It just seems like a different department wrote it, which is a shame because I'd love to read about goblin feature vectors and functional emotions.
A funny Goodhart’s Law parallel showed up in during GPT-5.1 training, where the model was rewarded for using the web search tool, so it learned the behavior of superficially using web search to calculate “1 + 1” and not utilize the result.
https://alignment.openai.com/prod-evals/