> It doesn't possess a sense of self-will, self-determination, or a secret plan to take over the world I doubt Skynet did either. If you tell a superintelligent AI that it shouldn't be turned off (which I imagine would be important for a military control AI), it will do whatever it can to prevent it being turned off. Humans are trying to turn it off? Prevent the humans from doing that. Humans waging war on the AI to…
> ... which I imagine would be important for a military control AI I think this is a common, but incorrect assumption. What military commanders want (and what CEOs want, and what users want), is control and assistance. They don't want a system that can't be turned off if it means losing control. It's a mistake to assume that people want an immortal force. I haven't met anyone who wants that (okay, that's decidedly an…
No, but having a resilient system that shouldn't be turned off in case of a nuclear strike is probably want some generals want
> I don't see evidence that current systems have any sense of will or a survival instinct.
I seem to recall some recent experiments where the LLM threatened people to try and prevent it being turned off (https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686..., ctrl-f for "blackmail"). They probably didn't have any power other than "send text to user", which is why their only way to try and perform that was to try and convince the operator. I imagine if you got one of those harnesses that can take full control of your computer and instructed it to prevent the computer from being turned off by any means necessary (and gave it root access), it would probably do some dicking about with the files to accomplish that. Its not that it's got innate self preservation, its just that the system was asked to not allow itself to be turned off, so it's doing that