> The "how" doesn't seem to me to be as important as you're suggesting.
When the question is understanding the true nature of what is occurring (eg "is an LLM conscious"), the "how it works" is critical. For example, the 1700s "Mechanical Turk" automaton which appeared to play chess (https://en.wikipedia.org/wiki/Mechanical_Turk). Royal courts and their advisors accepted that it played chess after glancing at the complex gearing inside the cabinet. Had they taken the time to examine how the internal gearing worked in greater detail, they would have arrived at a more accurate understanding of the device's true nature.
> That they do it at all is the point
True in some cases but not others, especially when external appearances can be deceiving. The Mechanical Turk was: 1. Designed to deceive, and 2. Not able to mechanistically play chess. Conversely, LLMs were not intentionally designed to deceive but they can still be misleading because they're a novel system which: 1. Manipulates linguistic symbols in highly complex ways, and 2. Can instantly access vast quantities of detailed information pre-trained into it's relational database that's been indexed across thousands of dimensions. And these abilities are not only novel but can be highly useful for some real-world tasks. This makes LLMs uniquely challenging for humans to reason about because LLMs are specifically tuned to generate output which closely mirrors the exact ways humans assess intelligence (and consciousness). We couldn't have designed a system to be more ideal at playing Turing's 'Imitation Game' and convincing humans they are human-like if we'd intentionally tried to.
In fact, I've previously described LLMs as accidentally being "the most perfectly deceptive magic trick ever" (while I'm a technologist professionally, I've spent quite few years designing actual magic tricks as a hobby). Designers of magical illusions joke that "the perfect floating lady trick" would actually be able to do useful things like replace a forklift, since it could float anything, anywhere instead of just appearing to violate physics. LLMs actually can really do useful things and replace some human labor but that fact doesn't mean they have all the abilities and traits of humans nor that they internally function in similar ways.
> You asked a hypothetical above...
That wasn't me, it was another poster.
> What is experience beyond taking input from the world around you and holding an understanding of it?
In the view of many leading philosophers of mind (Dennett, Chalmers, Nagle, etc) "Experiencing" is quite a bit more than just sensing, processing and recording. They use the term "Qualia" (https://plato.stanford.edu/entries/qualia/) which is what they're talking about when they ask "what is it like to be a..." (wikipedia.org/wiki/What_Is_It_Like_to_Be_a_Bat)? While the philosophical debate around why material reductionism can't explain human consciousness is fascinating, we don't need to go there to understand "what it is like to be an LLM" because we already know the answer: it's not like anything - there is no there there.
First, it's obvious we can't trust what the LLM's textual output says when it's asked "what is it like to be you" because it's an 'imitation machine' trained on 100% human sample text. The algorithms were designed, tuned and tested to generate text output which most plausibly simulates how a composite human would respond to the input (including the invisible system prompt instructing: "You are a Large Language Model, not a human"). We even added a tiny degree of random variability to the processing of the statistical weights because we found that makes the simulation seem a bit more plausibly like what a composite human would say. In short, the 'self-reported' textual output of a system purpose-built to generate plausible human-like textual output can't be trusted any more than a study of pathological liars can trust self-reported data from their study population.
Fortunately, with LLMs we can directly look under the hood at how it works and the entire specialty of Mechanistic Interpretability exists to do exactly that (https://towardsdatascience.com/mechanistic-interpretability-...). So we know with certainty that, despite what they may say, LLMs do not experience qualia in the way that humans and even other mammals do (which we have insight on from 'looking under the biological hood' with fMRI, surgical and brain injury studies).
Then the only question left is whether to redefine "consciousness" in some new way very different from "human consciousness" or "consciousness in mammals" (the only examples we've had until very recently). Personally, I think it makes little sense to radically redefine consciousness to include statistical algorithms running billions of matrix multiplications on a massive database of human-generated text. The term "consciousness", as vague and poorly defined as it is, was created to refer to human, or at least biological, consciousness. I'd be fine with creating a new term to refer to whatever novel traits of LLMs someone wants to quantify but they should leave the term "consciousness" out of it because the poor thing's already barely useful and stretching it further will just leave it broken and devoid of any meaning.