I want people to understand what the thing is, and what it isn't. By people, I especially mean the people involved in making and selling it.
The overwhelming majority of talk I see on the subject misunderstands this very important distinction. Because it is missed, people are taking about ChatGPT personified. They are talking about the version that lives in their hopes and dreams, as if that version is just around the corner. It isn't. It just appears to be. Our hopes and dreams are the content on exhibition. We see them echoed in semantic reconstruction, but what we are looking at is not the work of a new writer: it's the old words of many, successfully repurposed in new semantic space.
It's as if we took all the stories we love, and split them into puzzle pieces, then fit the pieces together in a new order. Because language is so flexible, many of them fit, and even look good together. Language is so good that it pre-sorts subjects, objects, phrases, responses, and even logic. The puzzle-maker doesn't even have to know what they are working with to cut the pieces in a useful way!
> It absolute can and does favor giving truthful answers even though a lie is equally grammatical English.
That depends on the context. In cases that it has the answer, that is mostly true. This covers a lot of cases, especially those that are likely to be tested; because many such cases have "failed" and been resolved. That's literally what testing looks like.
But it's still enough of an open problem that they write about it on the main page.
The very fact that OpenAI has managed to put the sheer quantity and quality of behavior into their model that they have, is indeed impressive. But it is those people who deserve the credit, no matter how much they want to attribute that credit to ChatGPT itself.
My point here is that every "desirable answer" - that ChatGPT provides as a continuation - comes from the already-existing language in its training data. It may be semantically constructed as a result of being pulled from a semantic model, but we need to stop pretending it is symbolically constructed. That work happens entirely in the construction and application of the training data itself. ChatGPT is never doing that part of the job.
> One only need spend a minute looking at HN though to see that "I don't know" is really not part of the language used in most places online
Exactly! Every time any of the "places online" (that end up used as training data) exhibit this behavior, that is an instance of the work being done. The finished product of that very work is freely available. It only needs to be found in one of several semantically correct and semantically related contexts. That's what ChatGPT does, and nothing more.
By virtue of the average given persons' interest in detailed answers to questions, and the average persons' dislike for conversations that end in non-answers, we are already doing the work. Just by participating in the most common internet dialogues, we are creating a wealth of data that captures the same behavior we hope to see in ChatGPT's continuations.
> they're fixed depth circuits and can only engage in longer thoughts by 'thinking out loud', as an example.
That's a great summary of my point. Why are you arguing if you agree with me?
The phrase, "thinking out loud" is an excellent illustration for the power of language itself to capture and encapsulate logical behavior.
We could be spending a lot of time constructing semantically valid nonsense like, "colorless green ideas sleep furiously". But any time you see that sentence, you are likely to see it surrounded in thoughtful context. This sentence is a popular example, not only of the ability for language to contain nonsense, but also for our tendency as language users to avoid doing just that.
A story isn't just a jumble of semantically valid symbols: there is meaning expressed. We don't write a lot of nonsense. We do write a lot of meaning.
That meaning is neatly packaged into the semantics of language. To find it, one only needs to unwrap the semantics themselves.
But that only works when the meaning has been written somewhere. And there is no way to distinguish between two conflicting meanings that live in the same semantic space. Even if neither one is a lie, the semantics must disambiguate, or one of the options has to be preferred as a continuation to its expected context; during the training step.
> LLMs model the world through language.
They don't model the world: people do. They just model the language. Because we have packaged our models of the world (and every interaction with those models) all together into language, we can see language models echo all those behaviors back to us.