What Is Happening with ChatGPT?

ChatGPT went crazy on me today on more than one occasion. Has this been happening to anyone else?

Yeah I’ve had some strange instances where it ends up restarting a given prompt (this is with the API). Didn’t see the delimiter though.

(edit: i'm a dumbass, but i can't delete this comment, please ignore)

Not what you asked, but I prefer https://www.wordhippo.com/ when I need a thesaurus. Its never gone crazy on me yet.

https://www.wordhippo.com/what-is/another-word-for/deadly.ht...

It's fascinating that it's outputting `` and then some unrelated sentences. When training these LLMs, you typically append a special "end-of-message" token to each sentence and pad the rest of the prompt embeddings with zeroes. The model then learns to add the EOM token to its own output because it's seen that before in the training data, but it's not like that's a strict requirement; there's nothing that forces the model to do this IIRC. During inference, the decoder knows to stop decoding once it sees the model's EOM token, but if the model doesn't output EOM, the inference pipeline will just carry on generating new tokens.

Perhaps when training, OpenAI is trying to feed in multiple unrelated sentences per sample within each batch to increase efficiency? It's a fascinating and clever idea, and I could see how that might help ChatGPT scale up, but if the inference pipeline doesn't understand how to decode these special sentences then the model is just outputting what it's seen during training: the intended result and then a random unrelated sentence.

Or perhaps OpenAI might be batching multiple prompts together from multiple users and separating them out again on the decoder side, for efficiency? This feels like a terrible idea -- I wonder if that might reveal other people's prompts. I'd be surprised if they tried this, but they're certainly incentivized to invest heavily into this sort of engineering to keep costs per token down.

Just speculation about one scenario that could lead to output like that. I have no idea what OpenAI is really doing.

Maybe someday we'll have an entire class of "dirty context" LLM vulnerabilities, similar to how web developers have to worry about XSS/SQL injection attacks. Won't that be exciting! Perhaps attackers might consider how to affect the input prompt directly, or they might figure out how to trick the decoder into spitting out too much text that somehow reveals some hidden state or other.