What happens when people don't understand how AI works

The question is, what's wrong with that?

At some level there's a very human desire for something genuine and I suspect that no matter the "humanness" of an AI, it will never be able to close that desire for genuine. Or maybe... it is that people don't like the idea of dealing with an intelligence that will almost always have the upper hand because of information disparity.

You call a Doctor 'Doctor' because they're wearing a white coat and are sitting in a doctor's office. The words they say might make vague sense to you, but since you are not a medical professional, you actually have no empirical grounds to judge whether or not they're bullshitting you, hence you have the option to get a second or third opinion. But otherwise, you're just trusting the process that produces doctors, which involves earlier generations of doctors asking this fellow a series of questions with the ability to discern right from wrong, and grading them accordingly.

When someone can't tell if something just sounds about right or is in fact bullshit, they're called a layman in the field at best or gullible at worst. And it's telling that the most hype around AI is to be found in middle management, where bullshit is the coin of the realm.

With that distinction in mind, whether an LLM-based chatbot’s output looks like human output does not answer the question of whether the LLM is actually like a human.

Not even because measuring that similarity by taking text output at a point in time is laughable (it would have to span the time equivalent of human life, and include much more than text), but because LLM-based chatbot is a tool built specifically to mimic human output; if it does so successfully then it functions as intended. In fact, we should deliberately discount the similarity in output as evidence for similarity in nature, because similarity in output is an explicit goal, while similarity in underlying nature is a non-goal, a defect. It is safe to assume the latter: if it turned out that LLMs are similar enough to humans in more ways than output, they would join octopus and the like and qualify to be protected from abuse and torture (and since what is done to those chatbots in order for them to be useful in the way they are would pretty clearly be considered abuse and torture when done to a human-like entity, this would decimate the industry).

That considered, we do not[0] know exactly how an individual human mind functions to assess that from first principles, but we can approximate whether an LLM chatbot is like a human by judging things like whether it is made in a way at all similar to how a human is made. It is fundamentally different, and if you want to claim that human nature is substrate-independent, I’d say it’s you who should provide some evidence—keeping in mind that, as above, similarity in output does not constitute such evidence.

[0] …and most likely never could, because of the self-referential recursive nature of the question. Scientific method hinges on at least some objectivity and thus is of very limited help when initial hypotheses, experiment procedures, etc., are all supplied and interpreted by the very subject being studied.

Why not just start with a straight forward Google search?

And made the relevant point that I need know what you mean by "understanding"?

The only 2 things in the universe that know that 6 is the maximum white flag emojis for jokes, and then might be modified by plus signs is ...

My brain, and that digital instance of Claude AI, in that context.

That's it - 2. And I didn't teach it, it picked it up.

So if that's not "understanding" what is it?

That's why I asked that first, example second.

I don't see how laying out logically like this makes me the "Reddit Expert", sort of the opposite.

It's not about knowing the internals of a transformer, this is a question that relates to a word that means something to humans ... but what is their interpretation?