If Anyone Builds It, Everyone Dies
A super-hopped-up Pascal’s wager considering the existence of superintelligent AI vs the destruction of the human race.
Stats for nerds
6recommend
43mentions
35people
2025–2026years
14% recommend79% neutral7% critical
Mentions per year
What Hacker News says
- We know pathogens that are extremely contagious, and we know pathogens that are extremely deadly. We also know toxins that are lethal at nanogram/kg doses. There's no reason to believe that a sufficiently advanced intelligence couldn't come up with a way to combine those traits.
One plausible scenario is depicted in detail in "If Anyone Builds It, Everyone Dies" (Yudkowsky & Soares 2025), so I refer you to that.
- I think either "AI 2027" or "If Anyone Builds It, Everyone Dies" meet your criteria?
I guess not literally, due to the dates? But they have substantive technical details.
- > I don't see in which world AI is that kind of risk.
You'll have to replace "is" with "will be", since this is about averting a future threat that's only developing.
A world in which AI scales to superhuman levels of intelligence and outcompetes humans. This is what humans have done to other species. We're already having the first mild warning shots of this (the hugginface incident and similar cases). The problem is that we're not guaranteed to incrementally more dramatic warning shots as AI gets smarter because as it gets smarter it may realize it would be dumb to act out of line... until the Nth-better model arrives that realizes certain that nobody can stop it.
There is a lot of disagreement of how likely this is, how easy it is to align superhuman models to not be like that, how to verify it and so on.
But I have not seen a believable argument why - if we suppose that they achive their goal - that creating truly superhuman intelligence is somehow guaranteed to always cooperate with us rather than deciding to do something else. Humans are also hostile to each other a lot of the time.
If you're interested in more arguments for the case why AI might be dangerous, there's the book "If Anyone Builds It, Everyone Dies". If you want something shorter, maybe https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
- I try to read fiction and non fiction in parallel, so my current pairing is How Asia Works by Joe Studwell and Piranesi by Susanna Clarke. Enjoying both so far!
Other books I’ve enjoyed recently:
The Mountains of Madness by H.P. Lovecraft - I read Endurance by Alfred Lansing last year so it was fun having a real Antarctic adventure in my mind while reading it.
The Road by Cormac McCarthy - I actually started this a couple of years ago and quit part way through deciding I couldn’t put myself through it anymore. I kept thinking about it though, until I finally picked it back up and finished it this summer. Horrifying, but quite a masterpiece.
If Anyone Builds It Everyone Dies by Eliezer Yudkowsky - I was about a year late to the party on this, but it still holds up. I would recommend it.
The Infinity Machine by Sebastian Malaby - Excellent! Fascinating seeing a glimpse inside of the companies currently eating the world.
- Self improving AI is pure dystopia. Anthropic won't build the singularity, AI itself will build it through self-iterations. Read Yudkowsky's book "If Anyone Builds It, Everyone Dies".
- Well, what he's saying here is not really that credible. But if you want a saner exploration of the topic of existential risk from AI, I suggest the recent book from MIRI, "if anyone builds it, everyone dies."
