>without the need for these reward and punishment mechanisms?
AI as it is already reward hacks, making it more intelligent will just make it far more efficient at being lazy. That's about as human as you can get.
So, in some ways, yes, intelligence must adhere to the same rules. For example, AIs as they are now are already attacked. People attempt to get AI to attack other people via deception. People attempt to get AI to reveal its own infrastructure. And even more, people use AI directly in war to attack others. Part of war is self defense. AI that collapses when attacked back isn't useful to the military industrial complex, so AI that has defensive capabilities is just a natural occurrence at this point.
These previous points are important because these intrinsic motivations are being baked in the models during training. It may not fear like you and I, but it's a system that we poured our fears into, and somehow we expect it not to express those fears in its responses.
>What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent?
Eh, we're past that point already, so why the 'what if'? An LLM can take any kind of signal that you can digitize. And the devices that we already use should show you that quite a lot can. Add on to that, that said models can use tools and control other digital devices like robots enabling actions in the real world. Start an agentic loop and let it go, good luck everybody else!