Yoshua Bengio

I knew there was trouble as soon as I saw "Nobel Prize [winner]" and "Adjunct professor" in the same sentence. What's it take to get tenure track these days? But she mentions the rate of publications. Kps noted that Peter Higgs said something similar as well. There's many others too! Turing prize winner Hinton had this to say about ML and I couldn't agree more

> One big challenge the community faces is that if you want to get a paper
published in machine learning now it's got to have a table in it, with all
these different data sets across the top, and all these different methods
along the side, and your method has to look like the best one. If it doesn’t
look like that, it’s hard to get published. I don't think that's encouraging
people to think about radically new ideas.

> Now if you send in a paper that has a radically new idea, there's no chance
in hell it will get accepted, because it's going to get some junior reviewer
who doesn't understand it. Or it’s going to get a senior reviewer who's
trying to review too many papers and doesn't understand it first time round
and assumes it must be nonsense. Anything that makes the brain hurt is not
going to get accepted. And I think that's really bad.

Or from Bengio

> In the rush preceding a conference deadline, many papers are produced, but there is not enough time to check things properly and the race to put out more papers (especially as first or equal-first author) is humanly crushing. On the other…

The concern is that near future systems will be much more capable than current systems, and by the time they arrive, it may be too late to react. Many people from the large frontier AI companies believe that world-changing AGI is 5 years or less away; see Situational Awareness by Aschbrenner, for example. There's also a parallel concern that AIs could make terrorism easier[1].

Yoshua Bengio has written in detail about his views on AI safety recently[2][3][4]. He seems to put less weight on human level AI being very soon, but says superhuman intelligence is plausible in 5-20 years and says:

> Faced with that uncertainty, the magnitude of the risk of catastrophes or worse, extinction, and the fact that we did not anticipate the rapid progress in AI capabilities of recent years, agnostic prudence seems to me to be a much wiser path.

Hinton also has a detailed lecture he's been giving recently about the loss of control risk.

In general, proponents see this as narrowly tailored bill to somewhat address the worst case worries about loss of control and misuse.

[1] https://www.theregister.com/2023/07/28/ai_senate_bioweapon/

[2] https://yoshuabengio.org/2023/05/22/how-rogue-ais-may-arise/

[3] https://yoshuabengio.org/2023/06/24/faq-on-catastrophic-ai-risks/

[4] https://yoshuabengio.org/2024/07/09/reasoning-through-arguments-against-taking-ai-safety-seriously/

That's what is usually confusing about mathematicians. They don't think using a depth-first tree going straight to the goal like someone from IT would, but rather go breadth-first, painting plenty of interesting concepts along the way, and chaining them seemingly aimlessly together until they crystalize right toward the right solution seemingly magically.

In his Physics Project Wolfram when talking about "Branchial space" is toying around the idea of recombining trees. Where one configuration of the state-space can be reached by various ways. This general concept of recombining trees has for example been used in generative flow networks https://yoshuabengio.org/2022/03/05/generative-flow-networks/ where they use it to directly generate molecule graphs that have useful properties among the sea of useless ones.

Take your pick?

https://yoshuabengio.org/2023/05/22/how-rogue-ais-may-arise/

https://intelligence.org/files/AlignmentHardStart.pdf

https://www.youtube.com/watch?v=pYXy-A4siMw

https://europepmc.org/article/med/26185241

https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies

I could keep going but the reading on this spans for tens of thousands of pages of detailed reasoning and analysis, including mathematical proofs and lab-scale experiments.