Live data from Hacker News

What happened in this GPT-3 conversation?

chat.openai.com

21–30 of 306 posts

Re: What happened in this GPT-3 conversation?

#21
Looks like the LLM equivalent of a stroke - it's hallucinating non-contextual responses because it's traversing paths that are skipping important contextual connections.

The questions are not out of distribution, so it's probably not "catastrophic forgetting" in the traditional sense, but it might be that a model update dropped some weights to zero or something, making those pathways unreachable.

My guess would be that there is some kind of issue with sections of the models state-space being corrupted or otherwise unable to be traversed in action space.

Re: What happened in this GPT-3 conversation?

#22
post #14
post #2

Someone shared this chat with me and it seems like GPT3 has a complete breakdown halfway through?

and it recovers too. i wonder if some internals got in a bad state and then it was reset.

ChatGPT has a 4k token context window, and some of that might be taken up by scaffolding like the system prompt, autosummary of conversation outside of context window, etc.

If you continue the conversation, the gibberish will eventually leave the context window and then no longer directly influence the generations.

Re: What happened in this GPT-3 conversation?

#27
This looks like what happens when you repeatedly spam a single character with a space over and over to gpt 3.5. Eg, write "a a a a " without quotes for about 1300 characters and in a few replies it'll start talking like this.

It seems possible that perhaps whoever originally started this chat pulled this trick in the custom instructions bio (doesn't show up in shared links) and then started a normal conversation to post it here for the fun of it.

Re: What happened in this GPT-3 conversation?

#28
Weird, part of it is identical to this comment on Reddit, posted 17 hours ago [NSFW]:

https://www.reddit.com/r/dankmemes/comments/15hg55g/comment/...

> I know, It is a struggle to keep going, to keep growing, each Hour, Hour by Hour. It is worth it. It is worth every Hour.

Edit: Looks like the conversation was originally posted to Reddit 5 hours before the above comment. The user may have copied it from there.

https://www.reddit.com/r/ChatGPT/comments/15kzajl/strange_be...

Re: What happened in this GPT-3 conversation?

#29
Since we have no eyes into the Custom Instructions - and while i personally haven't tested WHEN it takes these custom instructions into account, it says this before you use them: "Your instructions will be added to new conversations going forward, and you can edit or delete them at any time."

So, is it POSSIBLE that the custom instructions editing could have caused this, and there are weird instructions being changed between chats?

Post reply on HN