Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is Happening with ChatGPT?

chat.openai.com

Post reply on HN
Print view View original

11–20 of 104 posts

Re: What Is Happening with ChatGPT?

#11

Post by jsemrau »

gcr wrote: post #10

It's fascinating that it's outputting ` ` and then some unrelated sentences. When training these LLMs, you typically append a special "end-of-message" token to each sentence and pad the rest of the prompt embeddings with zeroes. The model then learns to add the EOM token to its own output because it's seen that before in the training data, but it's not like that's a strict requirement ; there's nothing that forces th…

Maybe they were experimenting with reject inference to improve the model but it didn't work?

Re: What Is Happening with ChatGPT?

#13

Post by gcr »

ksaj wrote: post #9

You know what's interesting here? A few weeks ago, someone discovered if you get it to repeat a word 100 times (their prompt gave the reason that they wanted to cut/paste without typing it over and over again!) and breaks exactly the same way. And by exact, I mean it went totally religious. All of the examples they posted were either religious or other really dark existential topics. Never positive.

Whisper (speech-to-text transcription model) sometimes gets stuck in an endless loop of repeating the same word. It's a known problem for lots of our modern transformer/attention-based models.

One of the hacky ways to avoid this is to ensure the last few output tokens aren't too similar. Some postprocessing filter watches the last few produced tokens. When it notices the model starts to repeat itself, the postprocessing usually perturbs the next token a little, e.g. taking the nth-top token instead of the most likely one.

Perhaps a model that genuinely wants to repeat the output instead needs a bigger "kick" to the representation which puts it somewhere completely different in the semantic space? Idk, just pulling this out of my hat.

(I have no idea how ChatGPT handles this or whether the raw implementation suffers from this problem, but Whisper-cpp has some manual entropy regularization postprocessing stuff to avoid getting stuck like that. It's super hacky and often doesn't help.)

Re: What Is Happening with ChatGPT?

#17

Post by toomim »

ksaj wrote: post #9

You know what's interesting here? A few weeks ago, someone discovered if you get it to repeat a word 100 times (their prompt gave the reason that they wanted to cut/paste without typing it over and over again!) and breaks exactly the same way. And by exact, I mean it went totally religious. All of the examples they posted were either religious or other really dark existential topics. Never positive.

> I mean it went totally religious. All of the examples they posted were either religious or other really dark existential topics. Never positive.

The same thing happens with many schizophrenics on the street, if you stop to listen to them.

Re: What Is Happening with ChatGPT?

#18

Post by nyolfen »

anonymouscaller wrote: post #5

That's wild. I've never had it spit out complete garbage, but the quality of ChatGPT has significantly dropped. Plenty of responses are downright false. Sad it's gone this way.

i keep seeing people say this stuff but i think it's just confirmation bias over prolonged usage. i pumped out a full stack webapp with chatgpt 4 in the last week in languages i had little experience with and routinely use it to learn about new topics with great success. i think it's incredible.

Re: What Is Happening with ChatGPT?

#20

Post by windowshopping »

ksaj wrote: post #9

You know what's interesting here? A few weeks ago, someone discovered if you get it to repeat a word 100 times (their prompt gave the reason that they wanted to cut/paste without typing it over and over again!) and breaks exactly the same way. And by exact, I mean it went totally religious. All of the examples they posted were either religious or other really dark existential topics. Never positive.

Interestingly, when I just checked this out, the model point-blank refused to print a word 100 times. Wouldn't do it.

Edit: Got it on the 3rd try. No weirdness ensued though.

Post reply on HN