Superintelligence
The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our species owes its dominant position. Other animals have stronger muscles or sharper claws, but we have cleverer brains. If machine brains one day come to surpass human brains in general intelligence, then this new superintelligence could become very powerful. As the fate of the gorillas now depends more on us humans than on the gorillas themselves, so the fate of our species then would come to depend on the actions of the machine superintelligence. But we have one advantage: we get to make the first move. Will it be possible to construct a seed AI or otherwise to engineer initial conditions so as to make an intelligence explosion survivable? How could one achieve a controlled detonation? To get closer to an answer to this question, we must make our way through a fascinating landscape of topics and considerations. Read the book and learn about oracles, genies, singletons; about boxing methods, tripwires, and mind crime; about humanity's cosmic endowment and differential technological development; indirect normativity, instrumental convergence, whole brain emulation and technology couplings; Malthusian economics and dystopian evolution; artificial intelligence, and biological cognitive enhancement, and collective intelligence.
Stats for nerds
92recommend
234mentions
178people
2014–2026years
46% recommend47% neutral7% critical
Mentions per year
What Hacker News says
- Neuromancer came out in 1984, over 40 years ago, so that covers the broad concept. Or Neo in The Matrix: “Programs hacking programs… why?”
For non-fiction you could look at Nick Bostrom’s Superintelligence: Paths, Dangers, Strategies (2014).
More recently, see Cade Metz’s NYT profile of Geoffrey Hinton, “The Godfather of A.I. Leaves Google and Warns of Danger Ahead” (May 1, 2023).
- Superintelligence by Bostrom came out in 2014, I recommend reading it, at the time I thought it was some of the best sci fi I'd ever consumed, and did not think it would be a public debate within my lifetime. What I find interesting is that respected/knowledgable people have been on TV/Mainstream media saying this for a few years now, Jack Clark has been in front of Congress multiple times. What changed about this particular message that made it so gripping? My little sister msg'd me last night and asked me "Can AI really kill us all? Why can't we just unplug it?". Interesting times....!
- Musk was told by Demis Hassabis in 2012, see https://time.com/6310076/elon-musk-ai-walter-isaacson-biogra...
Altman read Nick Bostrom's book Superintelligence, he mentions it in this post https://blog.samaltman.com/machine-intelligence-part-1
I also read that book in 2014, and it is how I came to understand this problem early.
- Superintelligence by Nick Bostrom is about this topic, quite a prescient read when it was published a decade ago.
- None of this guys arguments are persuasive, but I do recommend everyone read Nick Bostrom's Superintelligence.
- That was a “fun” read. Like Nick Bostrom’s Superintelligence [1].
Related
On intelligence
114 recommend · 228
Artificial Intelligence
60 recommend · 153
The Master Algorithm
23 recommend · 54
Deep Learning
44 recommend · 86
Machine Learning
16 recommend · 29
Programming Collective Intelligence
86 recommend · 135
AI superpowers
14 recommend · 31
