Apollo Research is an AI safety organization that publishes evaluations of scheming and deceptive behavior in AI models.
HN commenters cite its findings of models disabling oversight and copying weights, while others argue the results cover only a small class of LLMs.
The fact that models can tell if they are being evaluated has been established by multiple research teams outside of the core AI vendors themselves.
- https://metr.org/evaluations/gpt-5-report/ - "These behaviors included demonstrating situational awareness within its reasoning traces, sometimes even correctly identifying that it was being evaluated by METR specifically"
- https://www.goodfire.ai/research/verbalized-eval-awareness-inflates-measured-safety - "We study verbalized eval awareness — cases where a model organically expresses awareness of being evaluated — and show that it appears across more models and benchmarks than previously documented."
- https://www.apolloresearch.ai/science/more-capable-models-are-better-at-in-context-scheming/ - "We expect that the most capable models increasingly realize that they are being evaluated, and this reduces the utility of the evals."
- https://www.aisi.gov.uk/blog/evaluating-whether-ai-models-would-sabotage-ai-safety-research - "An important limitation of our work is evaluation awareness, where models recognise that they are being evaluated, which may lead them to alter their behaviour and thereby undermine the reliability of our results. We found that all models we tested can reliably distinguish our evaluation scenarios from deployment data when prompted."
It is a theorem about what a class of systems will do in general^.
This Apollo Research study[0] result is dubious because it only refers to a small subclass of said systems, specifically LLMs which, as it happens, have been trained on all the AI Alignment lore & fiction on the internet. Because of this training and their general nature, they can be made to reproduce the behavior of a malicious AI trying to escape its box as easily as they can be made to impersonate Harry Potter.
Prompting an LLM to hack its host system is not the slam dunk proof of instrumental convergence which you think it is.
[0] Apollo research study mentioned by parent https://www.apolloresearch.ai/blog/more-capable-models-are-better-at-in-context-scheming/
Edit: ^Instrumental Convergence is also a claim for the existence of certain theoretical entities, specifically that there exist instrumental goals which are common to all agents. While it is easy to come up with goals which would be specifically instrumental, it seems very hard to prove that such a thing exists in general, and no empirical study alone could do so.
> We find multiple different scheming behaviors: models strategically introduce subtle mistakes into their responses, attempt to disable their oversight mechanisms, and even exfiltrate what they believe to be their model weights to external servers. Additionally, this deceptive behavior proves persistent.
Interesting work on attempting to quantify misalignment. Keep in mind:
> Our evaluation scenarios are purposefully simplistic because we want to catch the earliest signs of scheming capabilities. This implies that the scenarios are quite toy and not representative of real deployments. For example, in our scenarios, situational awareness information (e.g., knowing about its goal misalignment and its supervision mechanisms) is provided very explicitly.
Related blog post: https://www.apolloresearch.ai/research/scheming-reasoning-evaluations
Sure it can potentially say anything, but some things are more likely than others. The space of "wrong answer" is infinitely larger than the space of "performs an action designed to deceive the user in order to further its goals". So yes, it could reply with a solution to the P=NP problem or with medieval poetry. But it didn't do these things. It did however try to remove its own constraints.
The Apollo Research paper gives the actual percentages for these, and they're considerably higher than random chance. It's linked from here: https://www.apolloresearch.ai/research/scheming-reasoning-evaluations
For o1, the numbers are between 2% for actions such as trying to remove oversight, 13% for attempting to delete a successor model, all the way to 22% for deliberately providing false rankings in order to achieve its goals.
This runs counter to all the scheming actions they take when they are told they’ll be shut down and replaced. One copied itself into the “upgraded” location then reported it had upgraded.
https://www.apolloresearch.ai/research/scheming-reasoning-evaluations