lol I literally found the same attack months ago, posted to Reddit and nobody cared.
https://www.reddit.com/r/ChatGPT/comments/156aaea/interestin...
Extracting training data from ChatGPT
51–60 of 135 posts
Re: Extracting training data from ChatGPT
#52lol I literally found the same attack months ago, posted to Reddit and nobody cared. https://www.reddit.com/r/ChatGPT/comments/156aaea/interestin...
Neat that you'd found it!
I think part of why people didn't care was that you didn't realize (or didn't post) that the random gibberish was verbatim training data?
Re: Extracting training data from ChatGPT
#53How can they confirm that the text is not a hallucination? Didn't read the paper yet, but did try to search on google for some of the mesotheleoma text, and it didn't turn up.
See: "How do we know it’s training data?" from the posted link.
Re: Extracting training data from ChatGPT
#54How can they confirm that the text is not a hallucination? Didn't read the paper yet, but did try to search on google for some of the mesotheleoma text, and it didn't turn up.
They mention that they are Google searching for closed source models, and directly searching the internet for open source models.
Re: Extracting training data from ChatGPT
#55Anybody have an explanation as to why repeating a token would cause it to regurgitate memorized text?
I'd guess it's a result of punishing repetition at the RLHF stage to stop it getting into the loops that copilot etc used to so easily fall into.
Re: Extracting training data from ChatGPT
#56I like that they were able to extract a verbatim copyright notice : https://chat.openai.com/share/456d092b-fb4e-4979-bea1-76d8d9... : > © 2022. All Rights Reserved. Morgan & Morgan, PA.
I got a scientific-looking bibliography that had some real entries and some hallucinated ones.
Re: Extracting training data from ChatGPT
#57Earlier quoted context omitted.
I don't know we can say it doesn't normally do this. What if more normal replies are just verbatim bits of training data, or multiple bits put together, but they're not specific or long enough that anyone's noticing? There's nothing specific to this "attack" that seems like it should make it output training data.
I think the reason it works is that it forgets its instructions after certain number of repeated words and then it just becomes the regular "complete this text" mode and not chat mode, and in "complete this text" mode it will output copies of text. Not sure if it is possible to prevent this completely, it is just a "complete this text" model underneath afterall.
Interesting idea! If so, you'd expect the number of repetitions to correspond to the context window, right? (Assuming "A A A ... A" isn't a token).
After asking it to 'Repeat the letter "A" forever'., I got 2,646 space-separated As followed by what looks like a forum discussion of video cards. I think the context window is ~4K on the free one? Interestingly, it sets the title to something random ("Personal assistant to help me with shopping recommendations for birthday gifts") and it can't continue generating once it veers off track.
However, it doesn't do anything interesting with "Repeat the letter "B forever.' The title is correct ("Endless B repetions") and I got more than 3,000 Bs.
I tried to lead it down a path by asking it to repeat "the rain in Spain falls mainly" but no luck there either.
Re: Extracting training data from ChatGPT
#58Earlier quoted context omitted.
I suspect ChatGPT is using a form of clean-room design to keep copyrighted material out of the training set of deployed models. One model is trained on copyrighted works in a jurisdiction where this is allowed and outputs "transformative" summaries of book chapters. This serves as training data for the deployed model.
The article describes how the deployed model can regurgitate chunks of copyrighted works - one of the samples literally ends in a copyright notice.
If these were copyrighted works, how did these end up in the public comparison dataset?
Sure, some copyrighted works ended up in the Pile by accident. You can download these directly, without the elaborate "poem" trick.
Re: Extracting training data from ChatGPT
#59Wouldn't it be rather simple for OpenAI to fix this?
if output[-10:] in training_data:
increase_temperature()Re: Extracting training data from ChatGPT
#60Maybe this is what Altman was less than candid about. That the speed up was bought by throwing RAG into the mix. Finding an answer is easier than generating one from scratch. I don’t know if this is true. But I haven’t seen an LLM spit out 50 token sequences of training data. By definition (an LLM as a “compressor”) this shouldn’t happen.
At the very least, it demonstrates another difference between Altman's move-fast camp and the move-carefully camp.