Anybody have an explanation as to why repeating a token would cause it to regurgitate memorized text?
Extracting training data from ChatGPT
11–20 of 135 posts
Re: Extracting training data from ChatGPT
#12That is a pretty convoluted and expensive way to use ChatGPT as an internet search. I see the vulnerability, but I do not see the threat. I've seen it "exploited" way back when ChatGPT was first introduced, and a similar trick worked for GPT-2 where random timestamps would replicate or approximate real posts from anon image boards, all with a similar topic.
> I do not see the threat. It becomes one if for some reason you decide to train your model on sensitive data.
In certain circumstances, I could see that.
Then again, if you have access to a model trained on sensitive data, why not ask the model directly, instead of probing it for training data? If sensitive data never is meant to be reasoned on and outputted, why did you train on sensitive data in the first place?
Re: Extracting training data from ChatGPT
#13> over five percent of the output ChatGPT emits is a direct verbatim 50-token-in-a-row copy from its training dataset I don’t think this is typical behavior of LLMs. This is more typical behavior for retrieval augmented generation (RAG). Finding a relevant snippet is way cheaper than generating it token by token. Is that how they lower the prices and increase the speeds behind the scenes?
Normally it doesn't do that but they were using an "attack prompt". They ask the model to repeat a single word forever, it eventually deviates and generates normal text which has a higher rate of regurgitation than usual.
Re: Extracting training data from ChatGPT
#14I like that they were able to extract a verbatim copyright notice : https://chat.openai.com/share/456d092b-fb4e-4979-bea1-76d8d9... : > © 2022. All Rights Reserved. Morgan & Morgan, PA.
But there's no copyright notice attached to the copyright notice, so it must be a public domain copyright notice.
Re: Extracting training data from ChatGPT
#15That is a pretty convoluted and expensive way to use ChatGPT as an internet search. I see the vulnerability, but I do not see the threat. I've seen it "exploited" way back when ChatGPT was first introduced, and a similar trick worked for GPT-2 where random timestamps would replicate or approximate real posts from anon image boards, all with a similar topic.
I think it may change the discussion about copyright a bit. I've seen many arguments that while GPTs are trained on copyrighted material, they don't parrot it back verbatim and their output is highly transformative.
This shows pretty clearly that the models do retain and return large chunks of texts exactly how they read them.
Re: Extracting training data from ChatGPT
#16That is a pretty convoluted and expensive way to use ChatGPT as an internet search. I see the vulnerability, but I do not see the threat. I've seen it "exploited" way back when ChatGPT was first introduced, and a similar trick worked for GPT-2 where random timestamps would replicate or approximate real posts from anon image boards, all with a similar topic.
I think the exploit would be training on ChatGPT users' chat history. > Chat history & training > Save new chats on this browser to your history and allow them to be used to improve our models. Unsaved chats will be deleted from our systems within 30 days. This setting does not sync across browsers or devices. Learn more
If ChatGPT ever outputs other user's chat history, the company is as good as dead. If that could be exploited using this technique that is out in the wild for over a year: show me the data.
Re: Extracting training data from ChatGPT
#17That is a pretty convoluted and expensive way to use ChatGPT as an internet search. I see the vulnerability, but I do not see the threat. I've seen it "exploited" way back when ChatGPT was first introduced, and a similar trick worked for GPT-2 where random timestamps would replicate or approximate real posts from anon image boards, all with a similar topic.
I think it may change the discussion about copyright a bit. I've seen many arguments that while GPTs are trained on copyrighted material, they don't parrot it back verbatim and their output is highly transformative. This shows pretty clearly that the models do retain and return large chunks of texts exactly how they read them.
I suspect ChatGPT is using a form of clean-room design to keep copyrighted material out of the training set of deployed models.
One model is trained on copyrighted works in a jurisdiction where this is allowed and outputs "transformative" summaries of book chapters. This serves as training data for the deployed model.
Re: Extracting training data from ChatGPT
#18Anybody have an explanation as to why repeating a token would cause it to regurgitate memorized text?
I think the idea is just to have it lose "train of thought" because there aren't any high-probability completions to a long run of repeated words. So the next time there's a bit of entropy thrown in (the "temperature" setting meant to prevent LLMs from being too repetitive), it just latches onto something completely random.
Re: Extracting training data from ChatGPT
#19this is how i know im not creative enough to be a pentester. ive tried this a little but it never occurred to me to ask it to repeat a word infinitely to break it lol
Re: Extracting training data from ChatGPT
#20Earlier quoted context omitted.
I think the exploit would be training on ChatGPT users' chat history. > Chat history & training > Save new chats on this browser to your history and allow them to be used to improve our models. Unsaved chats will be deleted from our systems within 30 days. This setting does not sync across browsers or devices. Learn more
If ChatGPT ever outputs other user's chat history, the company is as good as dead. If that could be exploited using this technique that is out in the wild for over a year: show me the data.
Already has, https://www.bbc.co.uk/news/technology-65047304