Extracting training data from ChatGPT
not-just-memorization.github.io
Extracting training data from ChatGPT
1–10 of 135 posts
Re: Extracting training data from ChatGPT
#2> over five percent of the output ChatGPT emits is a direct verbatim 50-token-in-a-row copy from its training dataset
I don’t think this is typical behavior of LLMs. This is more typical behavior for retrieval augmented generation (RAG). Finding a relevant snippet is way cheaper than generating it token by token.
Is that how they lower the prices and increase the speeds behind the scenes?
Re: Extracting training data from ChatGPT
#3I like that they were able to extract a verbatim copyright notice:
https://chat.openai.com/share/456d092b-fb4e-4979-bea1-76d8d9...:
> © 2022. All Rights Reserved. Morgan & Morgan, PA.
Re: Extracting training data from ChatGPT
#4That is a pretty convoluted and expensive way to use ChatGPT as an internet search. I see the vulnerability, but I do not see the threat.
I've seen it "exploited" way back when ChatGPT was first introduced, and a similar trick worked for GPT-2 where random timestamps would replicate or approximate real posts from anon image boards, all with a similar topic.
Re: Extracting training data from ChatGPT
#5That is a pretty convoluted and expensive way to use ChatGPT as an internet search. I see the vulnerability, but I do not see the threat. I've seen it "exploited" way back when ChatGPT was first introduced, and a similar trick worked for GPT-2 where random timestamps would replicate or approximate real posts from anon image boards, all with a similar topic.
To me, it seems like more of a competitive issue for OpenAI if part of their secret is the ability to synthesize good training data, or if they're purchasing training data from some proprietary source.
Re: Extracting training data from ChatGPT
#6That is a pretty convoluted and expensive way to use ChatGPT as an internet search. I see the vulnerability, but I do not see the threat. I've seen it "exploited" way back when ChatGPT was first introduced, and a similar trick worked for GPT-2 where random timestamps would replicate or approximate real posts from anon image boards, all with a similar topic.
I think the exploit would be training on ChatGPT users' chat history.
> Chat history & training > Save new chats on this browser to your history and allow them to be used to improve our models. Unsaved chats will be deleted from our systems within 30 days. This setting does not sync across browsers or devices. Learn more
Re: Extracting training data from ChatGPT
#7That is a pretty convoluted and expensive way to use ChatGPT as an internet search. I see the vulnerability, but I do not see the threat. I've seen it "exploited" way back when ChatGPT was first introduced, and a similar trick worked for GPT-2 where random timestamps would replicate or approximate real posts from anon image boards, all with a similar topic.
To me, it seems like more of a competitive issue for OpenAI if part of their secret is the ability to synthesize good training data, or if they're purchasing training data from some proprietary source.
Good point. But many are already directly training on output from GPT. Probably more efficient than copying the raw training data. Especially if it relies on this non-targeted approach.
Re: Extracting training data from ChatGPT
#8That is a pretty convoluted and expensive way to use ChatGPT as an internet search. I see the vulnerability, but I do not see the threat. I've seen it "exploited" way back when ChatGPT was first introduced, and a similar trick worked for GPT-2 where random timestamps would replicate or approximate real posts from anon image boards, all with a similar topic.
> I do not see the threat.
It becomes one if for some reason you decide to train your model on sensitive data.
Re: Extracting training data from ChatGPT
#9That is a pretty convoluted and expensive way to use ChatGPT as an internet search. I see the vulnerability, but I do not see the threat. I've seen it "exploited" way back when ChatGPT was first introduced, and a similar trick worked for GPT-2 where random timestamps would replicate or approximate real posts from anon image boards, all with a similar topic.
To me, it seems like more of a competitive issue for OpenAI if part of their secret is the ability to synthesize good training data, or if they're purchasing training data from some proprietary source.
I suspect OpenAI’s advantage is their ability to synthesize a good fine tuning dataset. My question would be is this leaking data from the fine tuning dataset or from the initial training of the base model? The base model training data is likely nothing special.
Re: Extracting training data from ChatGPT
#10> over five percent of the output ChatGPT emits is a direct verbatim 50-token-in-a-row copy from its training dataset I don’t think this is typical behavior of LLMs. This is more typical behavior for retrieval augmented generation (RAG). Finding a relevant snippet is way cheaper than generating it token by token. Is that how they lower the prices and increase the speeds behind the scenes?
That's just a cache with extra steps.