This is great, but it needs examples.
I'll give you one: ChatGPT.
Here's what the main page has to say:
> We’ve trained a model called ChatGPT which interacts in a conversational way. The dialogue format makes it possible for ChatGPT to answer followup questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests.
This is some careful writing. Most important are the weasel words, "possible for", and "in a...way". These make the context of the above paragraph ambiguous enough that it doesn't provide confidence in its own claims.
But is that enough? In court, probably. To the market? It doesn't seem to be. It's clear that most of the conversation about ChatGPT is (ironically) a continuation of the same foundational misconceptions that are expressed (albeit without confidence) here.
What is that misconception exactly? Identity. Here's one last claim that gets straight to the point:
> While we’ve made efforts to make the model refuse inappropriate requests, it will sometimes respond to harmful instructions or exhibit biased behavior.
I want to focus on some very enlightening weasel words: "exhibit behavior".
The more you read about what ChatGPT does, and more importantly, how it works, the more you are likely to see those magic words. That's because ChatGPT, from its most exciting results all the way down to its core behavior, doesn't actually behave.
But isn't that its core feature? Aren't carefully trained behaviors the main thing we expect to see in its output? Yes! And we see them! How exciting.
But there's a really important distinction that's really easy to miss: these behaviors are not ChatGPT's behaviors. They come from somewhere else entirely, and ChatGPT delivers them to us, usually at the right place and the right time. This is a case of mistaken identity.
So what, then, does ChatGPT do exactly? It finds semantic continuations. Where? Inside the neutral network that models a suite of example semantics.
It's only a model!
So whose behavior is being exhibited? Language.
Every exciting feature ChatGPT claims to provide is a feature of language itself.
Interacts in a conversational way? Language. Answer follow-up questions? Language. Admit its mistakes? Language. Challenge incorrect premises, and reject inappropriate requests? Language.
It's not just the exciting ones, either! It's the frustrating ones, too!
Writes plausible-sounding but incorrect or nonsensical answers? The model is often excessively verbose and overuses certain phrases?
All valid and common features of language.
And one more limitation that really illustrates the misunderstanding about what ChatGPT does:
> given one phrasing of a question, the model can claim to not know the answer, but given a slight rephrase, can answer correctly.
That's 100% semantically valid language. Language does nothing to stop us from telling lies! When ChatGPT pulls a continuation out of its model, there may be multiple options that are both valid, and contradictory! It might pull the reactionary response, "I don't know" (a very common continuation in real speech to any sentence structured as a question) or it might pull a domain-specific answer that responds to the subjects in that sentence.
It would be really useful if we could just teach it to do the second one, right? But we can't, because ChatGPT never draws a distinction between grammar and subjects. In fact, ChatGPT doesn't even distinguish words or punctuation!
ChatGPT can never understand. There's no place in its behavior to introduce logic. The best it can do is practice.
Instead of teaching ChatGPT, its authors give it carefully filtered data to model, and then put their fingers on the scale to make continuations they prefer more likely to be chosen. In a word: training. That's the best that can be done. Anything more requires a different kind of tool entirely.