What happens when people don't understand how AI works

instruction tuning is like imprimating the chat ux into the models weights

its all about the user/assistant flow instead of just a -text generator- after it

and the assistant always tries to please the user.

they built a sychopantic machine either by mistake or malfeasance

I keep hearing anecdotes but the data, like a widely covered BBC study, say they only compress and shorten and routinely fail outside of testing on real world selection of only the most important content or topics.

The article skirts around a central question: what defines humans? Specifically, intelligence and emotions?

The entire article is saying "it looks kinds like a human in some ways, but people are being fooled!"

You can't really say that without at least attempting the admittedly very deep question of what an authentic human is.

To me, it's intelligent because I can't distinguish its output from a person's output, for much of the time.

It's not a human, because I've compartmentalized ChatGPT into its own box and I'm actively disbelieving. The weak form is to say I don't think my ChatGPT messages are being sent to the 3rd world and answered by a human, though I don't think anyone was claiming that.

But it is also abundantly clear to me that if you stripped away the labels, it acts like a person acts a lot of the time. Say you were to go back just a few years, maybe to covid. Let's say OpenAI travels back with me in a time machine, and makes an obscure web chat service where I can write to it.

Back in covid times, I didn't think AI could really do anything outside of a lab, so I would not suspect I was talking to a computer. I would think I was talking to a person. That person would be very knowledgeable and able to answer a lot of questions. What could I possibly ask it that would give away that it wasn't real person? Lots of people can't answer simple questions, so there isn't really a way to ask it something specific that would work. I've had perhaps one interaction with AI that would make it obvious, in thousands of messages. (On that occasion, Claude started speaking Chinese with me, super weird.)

Another thing that I hear from time to time is an argument along the line of "it just predicts the next word, it doesn't actually understand it". Rather than an argument against AI being intelligent, isn't this also telling us what "understanding" is? Before we all had computers, how did people judge whether another person understood something? Well, they would ask the person something and the person would respond. One word at a time. If the words were satisfactory, the interviewer would conclude that you understood the topic and call you Doctor.

> value within their right answers

The thing is that LLMs provide plenty of answers where "right" is not a verifiable metric. Even in coding the idea of a "right" answer quickly gets fuzzy- should I use CSS grid or flexbox here? should these tables be normalized or not?

People simply have an unconscious bias towards the output just like they have an unconscious bias towards the same answer given by two real people they feel differently about- That is, the sort of thing all humans do (even if you swear that in all cases you are 100% impartial and logical).

I think the impulse of ascribing intent and meaning to the output is there in almost all questions, it's just a matter of degrees (CSS question vs. meaning of life type question)

> People without language - never learned, or sometimes temporarily disabled due to drugs, or permanently due to injury, transient or permanent aphasia - are still consciously thinking people.

There are 40 definitions of the word "consciousness".

For the definitions pertaining to inner world, nobody can tell if anyone besides themselves (regardless of if they speak or move) is conscious, and none of us can prove to anyone else the validity of our own claims to posess it.

When I dream, am I conscious in that moment, or do I create a memory that my consciousness replays when I wake?

> Words from an AI have nothing behind them but word statistics, devoid of any real world, just words based on words.

> […]

> When a fresh out of college person writes this it's based on some shallow real world experience, and lots of hearsay.

My required reading at school included "Dulce Et Decorum Est" by Wilfred Owen.

The horrors of being gassed during trench warfare were alien to us in the peaceful south coast of the UK in 1999/2000.

AI are limited, but what you're describing here is the "book learning" vs. "street smart" dichotomoy rather than their actual weaknesses.