think of it like this: the goal of LLMs (im saying LLM as shorthand for all of the AI algs) is to replicate human output / work, not so much to capture the human experience. so in order to do things, like communicate concepts, play a role in a scene, make choices, we project our human experience down into a lower resolution / constrained space to generate an output.
the question then is whether or not LLMs can mimic the projection of human experience as well as the real thing or not. My hypothesis is no for total general replication, but in certain fixed problem spaces I think its yes and the set of these spaces is growing. will it grow enough to encompass all practical work? not sure yet
AI tools are more like an author writing a character than a human revealing truths about their experiences. the difference between the LLM and an actual author is that the LLM kind of starts off with a 'flanderized' version of the character while an author can sort of blend personal experience and character to get at real human relatable decisions for the character. the result can be indistinguishable, because if the LLM is too predictable, too flanderized, we can inject artificial randomness to it to simulate personality.
you end up with unsolvable debates like movie critics have. "This character wouldnt make that decision in that situation if they were a real person, it doesnt make sense" vs real humans making irrational decisions.
ultimately i think the human experience is something we learn about with each other as we live it, and to think were any good at identifying it from sufficiently good imposters is putting the cart before the horse. once i know something is done by an actual human, my understanding of the world can shift. we dont decide what it means to be human, we accept what humanity is and try to work with it. So the philosophical debate on AI experience is moot to me, its not human therefor it teaches me nothing about humans and can replace exactly 0 human activity for me. Still can be a useful tool though