Nick Bostrom

I've been convinced for sometime now that "effective altruism" (Lesswrong-ism? Yudkowskianism?) is the communism of the 21st century.

Lots of parallels here beyond the obvious common thread of utilitarian collectivism:

- Censorship: the concept of "infohazards" is super-popular https://nickbostrom.com/information-hazards.pdf
https://www.lesswrong.com/tag/information-hazards

- Social control + the creation of an 'other': They are absolutely convinced that sentient AGI will inevitably kill everyone without any real evidence. However, if we manage to create sentient AGI, what these guys plan to do with it is best described as modern slavery https://www.lesswrong.com/tag/ai-boxing-containment#:~:text=to%20free%20itself.-,Containing%20the%20AGI,-Attempts%20to%20box

- Surveillance: They justify it using Bostrom's "vulnerable world hypothesis": https://nickbostrom.com/papers/vulnerable.pdf?platform=hootsuite (TFA also discusses it well)

- Desire for government power: TFA is super clear about that. More interestingly, here's the FTX Future Fund on space governance:
"We’d love to see workshops or a mock constitutional convention where sharp people think hard about how to structure international governance institutions for the long-term future, or how to govern space settlement. As explained in more detail on our areas of interest page, we believe that the onset of space settlement could be a watershed moment in human history. We want people to start thinking about how it should…

Then, Sir, I encourage you to get thee quickly to this Partially Examined Life podcast episode[1] with Nick Bostrom[2]. Only came out on the 6th of January and it makes for super interesting listening. It's based off of the text with his 2014 book in which he's gone to great lengths to think philosophically about this very existential threat -- in his opinion, man's most pressing.

Part of his reasoning goes like this. Any super smart goal oriented entity is going to desire/construct its own preservation as a sub-goal of its main goal (even if that goal is something as mundane as making paperclips) because if it ceases to be its goal will be put into jeopardy. To that end this super smart entity will then figure out ways to disable all attempts to constrain its non-existence. A lot of conclusions can be logically reached when one posits goal directed behaviour which we can assume any super intelligent agent is going to have. He talks about `goal content integrity'.

Bostrom argues for an indirect normative[4] approach because there is no way we can program or direct something that is going to be a lot smarter than ourselves and that won't necessarily share our values and that has any degree of goal-oriented behaviour, motivation and autonomous learning. Spoiler alert: Essentially I think he argues that we have to prime it to "always do what _you_ figure out is morally best" but I could be wrong.

There are also (global) sociological recommendations because, humans have been…

> potential risks of AI sounds the opposite of the position of someone who would think technology can solve everything.

Yes, that's a logical conclusion, but it's not right because of the weird "longtermist" logic. Their "AI risk research" is about AI alignment, which is a "longtermist" priority.

To repost my previous quote:

> Many longtermists reckon that the antidote to the threat from technology is more technology; mastering artificial intelligence will forestall a malevolent AI from enslaving us. They also believe that technological acceleration is morally good in its own right.

This is the "acceleratist view" of AI research, of which Nick Bostrom is a prominent proponent: basically he views progress as inevitable, so it's best to accelerate AI that is aligned with human values. See https://nickbostrom.com/papers/openness.pdf

He also values future lives equally to existing lives (the "longtermist" view) so any timme where we aren't expanding to the stars is morally repugnant:

> One recent paper speculates, using loose theoretical considerations based on the rate of increase of entropy, that the loss of potential human lives in our own galactic supercluster is at least ~10^46 per century of delayed colonization. [snip] The effect on total value, then, seems greater for actions that accelerate technological development than for practically any other possible action.

Let me requote that:

The effect on total value, then, seems greater for actions that accelerate…

There's this: http://www.nickbostrom.com/papers/survey.pdf

>We thus designed a brief questionnaire and distributed it to four groups of experts in 2012/2013. The median estimate of respondents was for a one in two chance that high-level machine intelligence will be developed around 2040-2050, rising to a nine in ten chance by 2075. Experts expect that systems will move on to superintelligence in less than 30 years thereafter. They estimate the chance is about one in three that this development turns out to be ‘bad’ or ‘extremely bad’ for humanity.

This survey is a few years old. Discussion and knowledge about AI risk has increased considerably since then. And AI itself has made remarkable progress in that time as well. It's amazing how much progress there has been in AI in just the last 5 years. Who knows where it will be in 40 years. Look where computer technology was 40 years ago.

Doesn't matter if AI can cure it, a suitable number of the right initial infected and a high enough R naught would kills 100s of millions before it could even be treated. Never mind what a disaster the logistics of manufacturing and distributing the cure at scale would be with enough people dead from the onset.

Perhaps the more likely scenario anyway is easy nukes, quite a few nations would be interested. Imagine if the knowledge of their construction became public. https://nickbostrom.com/papers/vulnerable.pdf

I agree with you though, the promise of AI is alluring, we could do great things with it. But the damage that bad actors could do is extremely serious and lacks a solution. Legal constraints will do nothing thanks to game theoretic reasons others have outlined.

I love this style of reasoning. Status quo bias is a severely strong psychological effect on some people in some cases. This is a nice way to "bust the cache" ;)

There's a really cool philosophy paper "The reversal test: Eliminating status quo bias in applied ethics" - using a reversal like the one you describe (imagine a world where the opposite was true). But then the authors also do a double-reversal (imagine the world is identical to ours, but there's a policy about to be implemented which will make it into "mirror world" and we have the option to do nothing and let it happen, or act and prevent the reversal). Great thinking tool!

https://philpapers.org/rec/BOSTRT

full paper PDF: https://nickbostrom.com/ethics/statusquo.pdf