GPT-3 vs. 3M free-text trivia responses

A larger corpus and more training seemed to have helped reduce the issue, and I think they tweaked the temperature algorithms as well (I think top-p is an option now, whereas most language models used top-k historically).

In my experience GPT repeats itself more often when it doesn't understand the prompt. I mean, if you don't understand _anything_ about an input sequence, what is the most likely next word? One of the previously used words.

It's likely that the following occurs: Given the phrase "1+1=? ", the first layer assigns a baseline probability to "1", "+", "=", and "?", because those occur in the input and its a good fallback. But then the layers after recognize the input as a math equation and quickly assign a high probability to "2". Since the probability on "2" is so high the temperature algorithm always picks "2" as the output token.

But if we imagine GPT doesn't understand math, then all the layers after the first have no clue what the input is. They can't assign higher probability to anything. They might make a few weak guesses, say "duck". But it isn't confident in them, so really "duck" is as likely as "1". The temperature algorithm then has a fairly decent chance of selecting "1" as the output token. Now GPT is given "1+1=? 1" and it seems like "+" is the next logical token. "1+1=? 1+". And so forth until it GPT is stuck in self reinforcing a loop saying "1+1=? 1+1=? 1+1=? 1+1=?" etc.

Funny thing is, it's not like humans don't do this. When you get asked a hard question that you don't have an answer for, don't you find yourself repeating the question to yourself over and over again, hoping you'll better understand it? I'm not necessarily creating a parallel between that behavior and GPT's behavior. Just food for thought, I suppose.

Since most language models work by using its outputs as its inputs, this means that the emitted text becomes worse the more words that are emitted. Suppose you want your model to emit a novel. The first 30 words might come out ok. The next 30 are so-so. Eventually, after a few hundred words, the model will begin to ramble and will get stuck in loops: "then she went to visit her mother and she went to visit her mother and she vent to visit her mother and she ..."

All language models suffer from this problem; Markov models, RNNs, Transformers, GPT-2/3... In fact, one could argue that GPT-3 has "only" decreased the deterioration speed. Markov models deteriorate after a dozen words while GPT-3 can keep going for perhaps a hundred words before it begins to ramble. However, the statistical properties of the text GPT-3 generates are different from what a human emits (and what those "statistical properties" are is unknown - it appears that capturing what makes human prose human-like in a mathematical formula is very difficult) from the very first word so you can't use GPT-3-generated text to train GPT-4. The paper "The Curious Case of Neural Text Degeneration" describes the problem in more detail and with math.

A major breakthrough in language modeling may come if someone is able to invent a model that doesn't degenerate. One whose thousandth word is as human-like as its first word.

For example, I was just reading this excellent long piece from Rebecca Traister: https://nymag.com/intelligencer/article/andrew-cuomo-miscond...

Even if a language model had all the raw facts needed to write a long piece, there's a ton of structure here. She develops ideas, builds themes, assembles a thesis. She is not just going somewhere, but carefully taking us along with her. That requires modeling not just textual relationships, but taking conceptual relationships and expressing them in ways that require deep awareness of how the audience processes information.

So I'd be willing to bet a true non-degenerative model requires an increasingly complex and accurate theory of mind.