I'm not talking about perfection, I'm talking about not being covered in blood and shit so much of the time is all, about not being scared and driven.
> You state the decoupling of psychological problems with some other undefined idealized personhood.
Can you rephrase that? English isn't my first language, and "stating decoupling of" doesn't parse at all for me. It's like there's a sentence fragment missing.
I can't easily define personhood. I know I'm one, that other people are, and I even know animals I recognize as persons, too. With super small animals and plants it gets tricky, but I don't need to fully define first for it to exist, there is something there. Any theory is an abstraction that never fully matches a reality which would remain unchanged if that theory didn't exist. Any intelligence we see and can make theories about results from actual living entitites. There is absolutely no reason to outright assume it can make sense in a vacuum. What would the AGI be intelligent for? Why wouldn't it just calculate whether the heat death of the universe will happen, and shut off if it comes to the conclusion it will? Nothing it would do would make a difference then. Humans are different, more irrational if you will. I don't believe immortality will ever be achieved, I don't even find it desirable, and still I get out of bed in the morning. Everyone I love will die, and everything I do will turn to dust. I still love and make things.
So, what would that "spark" be for AGI? I just had the dumbest association, in the movie "The Fifth Element" there is this scene where the [I forgot what the name of her role was, the female protagonist anyway] is taking in all the atrocities of humanity and gets kind of frustrated, and doesn't feel like saving them or something like that. Then Bruce Willis kisses her and it's alright. As I said, it's dumb, but still, if all we give an AI is problems, why would it come up with solutions, and not more problems?
> I would argue, and the science generally agrees, that those "psychological problems" however defined are the other edge on the sword of intelligence and relative autonomy.
Obviously a piece of rock doesn't have any of the problems or joys we do, but that doesn't just mean anything and everything just "goes with the territory".
> If we make the progress we want I expect heavy opposition.
And you're just going to steamroll over it? Fantastic. You'll go to the friendly baker next and they'll happily sell you bread because they don't know. And you can't face them one on one, so you have to "achieve significant powers" in the shadows. And you're still not seeing the woods for all the trees. No, it's their fault for not understanding what you're not even telling them.