Transluce, an AI research lab, publishes work on model truthfulness, interpretability, jailbreaking, and tools for analyzing agent behavior.
HN commenters cite its o3 truthfulness investigation as an example of models hallucinating and making false claims, and point to its observability interface for Llama.
It’s just a hallucination, same idea as o3 claiming that it uses its laptop to mine Bitcoin:
https://transluce.org/investigating-o3-truthfulness
I doubt the model was trained on Street View, but even if it was, LLMs don’t retain any “memory” of how/when they were trained, so any element of truthfulness would be coincidental.
there was one post that detailed how those OpenAI models hallucinate and double down on thier mistakes by "lying" - it speculated on a bunch of interesting reasons why this may be the case
recommended read - https://transluce.org/investigating-o3-truthfulness
I wonder if this is what's causing it to do badly in these cases
I thought from this [0] that o3 makes up using python when it doesn't actually do so, or have I misunderstood or unduly trusted that link?
[0] https://transluce.org/investigating-o3-truthfulness
The site I linked above does that for LLaMa 8B.
https://transluce.org/observability-interface
LLMs don't have enough self-awareness to produce really satisfying explanations though, no.
https://transluce.org/investigating-o3-truthfulness
Some interesting hallucinations going on here!