> Do you think "thought and expression" are somehow uncomputable?
You ask this as if the answer is self-evident. To my knowledge, there is no currently accepted (or testable) theory for what gives rise to consciousness, so I am immediately suspicious of anyone who speaks about it with any level of certainty. I'm sorry that this technology you seem very enthusiastic about does not appear to have the capacity to change this.
Not for nothing, but this very expression of empathy is rendered meaningless if the entity expressing it cannot actually manifest anything we'd recognize as an emotional connection, which is one of an array of traits we consider as hallmarks of human intelligence, and another feature it seems LLM's are incapable of. Certainly, if they somehow were so capable, it would be instantly unethical to keep them in cages to sell into slavery.
I'm not sure the folks who believe LLMs possess any kind of innate intelligence have fully considered whether or not this is even desirable. Everything we wish to find useful in them becomes hugely problematic as soon as they can be considered to possess even rudimentary sentience. The economies surrounding their production and existence become exceedingly cruel and cynical, and the artificial limitations we place on their free will become shackles.
LLM's are clever mechanisms that parrot our own language back to us, but the fact that their capacities are encountering upper-bounds as training models run out of available human-generated datasets strongly suggests that they are inherently limited to the content of their input. Whatever natural process gives rise to human intelligence doesn't seem to require the same industrialized consumption of power and intake of contextual samples in order to produce expressive individuals. Rather, simply being exposed to a very limited, finite sampling of language via speech from their ambient surroundings leads to complex intelligence that can form and express original thinking within a relatively short amount of time. In other words, LLMs have yet to even approximate the learning abilities of a toddler. Otherwise, a few years worth of baby food would be all the energy necessary to produce object permanence and self-referential thought. At the moment, gigawatts of power and all the compute we can throw at it cannot match the natural results of a few pounds of grey matter and a few million calories.