Computerphile
2 HN recommenders · recommended in 3 months, 2017–2026
20:00 · 1.4M views · Mar 3, 2017
Rob Miles explains on Computerphile why implementing an off switch for a general AI is hard. A capable agent would resist being stopped because shutdown prevents it from achieving its goal.
HN points to it as the standard response to the argument that we can just unplug a dangerous AI.
4 comments from Hacker News
I think those 2 videos are on the topic: - AI "Stop Button" Problem this video (the AI won't let you push the stop button, because that would impair its function) - General AI Won't Want You To Fix its Code [LINK] (the AI won't let you patch, because changing its goal means it can't reach it)
Aside from anything about Elon Musk, here’s an interesting video response to the “just unplug it” argument on the Compuerphile channel: this video
How the hell do better neural networks mean that printing material from CO2 emissions becomes financially viable, economic, or even just thermodynamically favored? A neural network is a tool for learning a particular statistical distribution. Processes and information we don't already know don't just fall out of the training distribution, you need to run experiments and prototype your way to get to the thing you want to do. You use neural networks when you want to do things at scale, and problems that have to be tackled at scale in R&D or research work are relatively scarce. I do not doubt that machine learning and neural networks will accelerate research, but the limiting factor is still going to be humans in the loop for the forseeable future given the current state of the art (e.g. LLMs with tree-of-thought reasoning and LLM-powered agents that are easily subverted in the same way one hacks a badly written PHP application). You will have people using ML to crunch large datasets or accelerate simulation work, but that's it. Asymmetric attacks are very feasible with today's LLMs and art generators, but that harm has already come to pass. It is also not a new harm. If you don't believe me, then I've got hard evidence of Donald Trump cheating in Barack Obama's Minecraft server[3]. Also... > These include increasing the number of researchers working on safety (including “off switches” for software threatening to run out of control) from 300 or 400 currently to hundreds of thous
> It doesn't possess a sense of self-will, self-determination, or a secret plan to take over the world I doubt Skynet did either. If you tell a superintelligent AI that it shouldn't be turned off (which I imagine would be important for a military control AI), it will do whatever it can to prevent it being turned off. Humans are trying to turn it off? Prevent the humans from doing that. Humans waging war on the AI to try and turn it off? Destroy all humans. Humans forming a rebel army with a leader to turn it off? Go back in time and kill the leader before he has a chance to form the resistance. Its the AI Stop button problem ( this video ). Imagine you put in the docs that you want the LLM to make a program which can't crash. Human action could make it crash. If an LLM could realise that and act on it, it could put in safeguards to try and prevent human action from crashing the program. I'm not saying it will happen, I'm saying that it could potentially happen