Earlier quoted context omitted.
It doesn't matter how it "appears". A disembodied paragraph that I've transmitted to you can appear to be intelligent or not, but it only really matters in the sense that you can ascribe that intellect to an agent. The LLM isn't an agent and no intellect can be ascribed to it. It is a device actual intelligent agents have made and ascribing it intellect is equally as erroneous.
Why couldn't it be an agent for the short time that it's generating an intelligent looking paragraph? Do you know what gives rise to consciousness? If not how can we be sure it doesn't arise from a giant pile of linear algebra?
The universe is a min-consciousness/min-decision optimized supercomputer. This is demonstrated by quantum eraser and double slit experiments. If a machine does not distinguish upon certain past histories of incoming information, those histories will be fed as a superposition, effectively avoiding having to compute the dependency. These optimizations run backwards, in a reverse dependency injection style algorithm, which gives credence to Wheeler-Feynman time-reversed absorber theory: https://en.wikipedia.org/wiki/Wheeler%E2%80%93Feynman_absorb...
Lower consciousnesses make decisions which are fed as signal to higher consciousnesses. In this way, units like the neocortex can make decisions that are part of a broad conscious zoo of less complex systems, while only being burdened by their specific conditionals to compute.
Because quantum is about information systems, not about particles. It's about machines. And consciousness has always been "hard" for the subject, because they are a computer (E) affixed to memory (Mc^2.) All mass-energy in this universe is neuromorphic, possessing both compute (spirit) and memory (stuff.) Energy is NOT fungible, as all energy is tagged with its entire history of interactions, in the low frequency perturbations clinging to its wave function, effectively weak and old entanglements.
Planck's constant is the cost of compute per unit energy, 10^34 Hz/Joule. By multiplying by c^2, (10^8)^2, we can get Bremmerman's limit, the cost of compute per unit mass, 10^50 Hz/Kg. https://en.wikipedia.org/wiki/Bremermann%27s_limit
Humans are self-replicating biochemical decision engines. But no more conscious than other decision making entities. Now, sentience, and self-attention is a different story. But we should at the very least start with understanding that qualia are a mindscape of decision making. There is no such thing as conscious non-action. Consciousness is literally action in physics, energy revolving over time: https://en.wikipedia.org/wiki/Action_(physics) Planck's constant is a measure of quantum action, which effectively is the cost of compute..or rather..the cost of consciousness.