Can you do the same with SD and get training pictures back?
This is literally mentioned in the post
Extracting training data from ChatGPT
71–80 of 135 posts
Re: Extracting training data from ChatGPT
#72How can they be so sure the model isn’t just hallucinating? It can also hallucinate real facts from the training data. However, that doesn’t mean the entire output is directly from the training data. Also, is there any real world use case? I couldn’t think of a case where this would be able to extract something meaningful and relevant to what the attackers were trying to accomplish.
They cover this in the article, they verified that the output matched data found on the internet, 100% verbatim.
Re: Extracting training data from ChatGPT
#73How can they be so sure the model isn’t just hallucinating? It can also hallucinate real facts from the training data. However, that doesn’t mean the entire output is directly from the training data. Also, is there any real world use case? I couldn’t think of a case where this would be able to extract something meaningful and relevant to what the attackers were trying to accomplish.
> How can they be so sure the model isn’t just hallucinating?
This is explicitly covered in the article, if you scroll down.
Re: Extracting training data from ChatGPT
#74They patched that real quick.
Regex to the rescue!!!
Re: Extracting training data from ChatGPT
#75How can they be so sure the model isn’t just hallucinating? It can also hallucinate real facts from the training data. However, that doesn’t mean the entire output is directly from the training data. Also, is there any real world use case? I couldn’t think of a case where this would be able to extract something meaningful and relevant to what the attackers were trying to accomplish.
They have 10TB on internet data and could find huge swaths of texts verbatim in it.
Re: Extracting training data from ChatGPT
#76Maybe this is what Altman was less than candid about. That the speed up was bought by throwing RAG into the mix. Finding an answer is easier than generating one from scratch. I don’t know if this is true. But I haven’t seen an LLM spit out 50 token sequences of training data. By definition (an LLM as a “compressor”) this shouldn’t happen.
> That the speed up was bought by throwing RAG into the mix.
sorry what? TFA does not mention RAG at all. are you reading your own biases into this or did i miss something
Re: Extracting training data from ChatGPT
#77How can they confirm that the text is not a hallucination? Didn't read the paper yet, but did try to search on google for some of the mesotheleoma text, and it didn't turn up.
They mention that they are Google searching for closed source models, and directly searching the internet for open source models.
Sorry wrote that too hastily - directly searching a 10Tb dataset of the internet for open source models
Re: Extracting training data from ChatGPT
#78Maybe this is what Altman was less than candid about. That the speed up was bought by throwing RAG into the mix. Finding an answer is easier than generating one from scratch. I don’t know if this is true. But I haven’t seen an LLM spit out 50 token sequences of training data. By definition (an LLM as a “compressor”) this shouldn’t happen.
At the very least, it demonstrates another difference between Altman's move-fast camp and the move-carefully camp.
[dead]
Re: Extracting training data from ChatGPT
#79lol I literally found the same attack months ago, posted to Reddit and nobody cared. https://www.reddit.com/r/ChatGPT/comments/156aaea/interestin...
The difference between screwing around and science is writing things down .... and publishing in a peer-reviewed journal.
To be fair, they did write things down. It’s more a matter of explaining why GPT was behaving the way it was (ie, because it was regurgitating its training data). Also, I’d personally respect a blog post just as a much as a peer reviewed journal article on something like this where it’s pretty easy to reproduce yourself, not to mention that I and I’m sure many others have observed this behaviour before.
Re: Extracting training data from ChatGPT
#80Maybe this is what Altman was less than candid about. That the speed up was bought by throwing RAG into the mix. Finding an answer is easier than generating one from scratch. I don’t know if this is true. But I haven’t seen an LLM spit out 50 token sequences of training data. By definition (an LLM as a “compressor”) this shouldn’t happen.
No, it can easily happen.
- They don’t do compression by “definition”. They are designed to predict, prediction is key to information theory, so they just have similar qualities.
- Everyone wants their model to learn, not copy data, but overfitting happens sometimes and overfitting can look the same as copying.