Earlier quoted context omitted.
> The "conscious superintelligent AI" is a scifi LARPing game that some people are getting paid to play Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me. AI researchers are working towards making intelligent machines. Why are we supposed to believe that - if they manage to achieve their goal - these machines won't…
> Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me. Lucky I didn't say anything remotely like that, then. > AI researchers are working towards making intelligent machines. Why are we supposed to believe that - if they manage to achieve their goal - these machines won't surpass our intelligence? I didn't say that th…
That would probably be quickly optimised out by the training process as it’s an inefficient way to achieve a goal. The only basis for this argument is that humans are smarter than other animals and this seems unique to humans but it’s probably just a quirk of humans
“With a big enough gap in intelligence, there's no guarantee that an entity would be able to "think like a human" any more than we can "think like a cat".”
LLMs are clearly already very good at “thinking like a human”
“he still couldn't talk the cat into it.”
Because cats can’t talk? Humans can, and can clearly be manipulated into doing things, sometimes even by current AI
“So how are we supposed to solve ethics and code a moral fixed point for a recursively self-improving intelligence” “ The idea that we can securely design the most complex system ever built, and have it remain secure through thousands of rounds of recursive self-modification, does not match our experience.”
Yes, that is why AI alignment is hard, that’s an argument for that view, not one against it
“I don't buy this argument at all. Complex minds are likely to have complex motivations; that may be part of what it even means to be intelligent.”
This is a clear misunderstanding of what it means to be intelligent. It doesn’t mean “do things that humans find impressive”, it means “achieve your goal effectively”. An AI that goes against its own goals is not intelligent, would not achieve its training objective and the training process (which somewhat resembles evolution) would optimise it out in the next step of training in the same way that a species that immediately throws itself off the nearest cliff will not survive to the next generation. You can imagine a superintelligence as something that goes over all the actions it could take, sees what reward it would get according to its goal and picks that action. The intelligence here is in the world model it uses to predict the consequences of its actions, picking what actions to take is a simple “max over array” function so saying it will randomly decide to persue a more meaningful objective is like saying that a sorting algorithm will put a list in the wrong order because it thinks some other order is more meaningful
(I will continue in a new comment in case there’s some kind of length limit)