Toby Ord, an Oxford philosopher, writes about AI scaling, reinforcement learning costs, agent performance, and AI governance.
HN commenters cite his analyses of RL scaling inefficiency and rising hourly costs of AI agents as underappreciated, drawn from limited public data.
The premise of this post and the one cited near the start (https://www.tobyord.com/writing/inefficiency-of-reinforcement-learning) is that RL involves just 1 bit of learning for a rollout, rewarding success/failure.
However, the way I'm seeing this is that a RL rollout may involve, say, 100 small decisions out of a pool of 1,000 possible decisions. Each training step, will slightly upregulate/downregulate a given training step in the step's condition. There will be uncertainty about which decision was helpful/harmful -- we only have 1 bit of information after all -- but this setup where many steps are slowly learned across many examples seems like it would lend itself well to generalization (e.g., instead of 1 bit in one context, you get a hundred 0.01 bit insights across 100 contexts). There may be some benefits not captured by comparing the number of bits relative to pretraining.
As the blog says, "Fewer bits, sure, but very valuable bits", this also seems like a different factor that would also be true. Learning these small decisions may be vastly more valuable for producing accurate outputs than learning through pretraining.
The author of the posts you linked also wrote https://www.tobyord.com/writing/inference-scaling-reshapes-ai-governance which posits:
> AI labs may also be able to reap tremendous benefit from these inference-scaled models by using them as part of the training process. If so, the large scale-up of compute resources could go into post-training rather than deployment. This would have very different implications for AI governance.
> ...
> So iterated distillation and amplification provides a plausible pathway for scaling inference-during-training to rapidly create much more powerful AI systems. Arguably this would constitute a form of ‘recursive self-improvement’ where AI systems are applied to the task of improving their own capabilities, leading to a rapid escalation.
So "inference scaling is required to scale capabilities" doesn't mean that we're reaching the top of the S-curve in intelligence. If anything, it could mean a shorter timeline and more unpredictable landscape for governance (e.g. due to securing weights no longer as effectively preventing escalation, more in the article).
I'm not sure I agree with you. The only widely cited quantitative guess at this scenario I could find in a brief trawl puts the chance of any "climate-driven existential catastrophe" this century at about 0.1 percent [1].
"Extinction" is a stricter outcome than “existential catastrophe", so I would imagine the odds this century are even lower than 0.1 percent. In other words: don't resign yourself to the extinction of humanity in your lifetime - it's highly unlikely, and you're just going to be feeding the defeatism that is so tempting when external events are so seemingly negative.
As sibling commenter to this post notes though, I would imagine that mass unrest, famines, death etc. in less-developed parts of the world will occur, with knock-on effects for the developed world.
[1] https://www.tobyord.com/writing/the-precipice-revisited
Yes. Anyone who doesn't acknowledge the efficiency difference between pretraining vs RL and assume that since we've run out of data for the former, we have to do the latter, is not making a serious attempt at modelling the future:
https://www.tobyord.com/writing/inefficiency-of-reinforcement-learning
This is similar to that other exponential, which happened with CPUs - we ran out of true geometric scaling in the mid 2000s, and everything else supporting Moore's Law has been cleverness that arrived in the nick of time, supported by a bit of marketing, and very optimizable benchmarks, far from guaranteed gains coming from making a single physical metric better.
Scaling law is a power law , requiring orders of magnitude more compute and data for better accuracy from pre-training. Most companies have maxed it out.
For RL, we are arriving at a similar point
https://www.tobyord.com/writing/how-well-does-rl-scale
Next stop is inference scaling with longer context window and longer reasoning. But instead of it being a one-off training cost, it becomes a running cost.
In essence we are chasing ever smaller gains in exchange for exponentially increasing costs. This energy will run out. There needs to be something completely different than LLMs for meaningful further progress.
> just as humans shouldn’t serve as calculators
But they definitely could and were [0]. You just employ multiple, and cross check - with the ability of every single one to also double check and correct errors.
LLMs cannot double check, and multiples won't really help (I suspect ultimately for the same reason - exponential multiplication of errors [1])
[0] https://en.wikipedia.org/wiki/Computer_(occupation)
[1] https://www.tobyord.com/writing/half-life