81–90 of 135 posts
lol I literally found the same attack months ago, posted to Reddit and nobody cared. https://www.reddit.com/r/ChatGPT/comments/156aaea/interestin...
Neat that you'd found it! I think part of why people didn't care was that you didn't realize (or didn't post) that the random gibberish was verbatim training data?
Maybe this is what Altman was less than candid about. That the speed up was bought by throwing RAG into the mix. Finding an answer is easier than generating one from scratch. I don’t know if this is true. But I haven’t seen an LLM spit out 50 token sequences of training data. By definition (an LLM as a “compressor”) this shouldn’t happen.
A couple problems with this.
1) That's not the definition of an LLM, it's just a useful way to think about it.
2) That is exactly what I'd expect a compressor to do. That's the exact job of lossless compression.
Of course the metaphor is lossy compression, not lossless. But it's not that surprising if lossy compression reproduces some piece of what it compressed. A jpeg doesn't get every pixel or every local group of pixels wrong.
Maybe this is what Altman was less than candid about. That the speed up was bought by throwing RAG into the mix. Finding an answer is easier than generating one from scratch. I don’t know if this is true. But I haven’t seen an LLM spit out 50 token sequences of training data. By definition (an LLM as a “compressor”) this shouldn’t happen.
No, it can easily happen. - They don’t do compression by “definition”. They are designed to predict, prediction is key to information theory, so they just have similar qualities. - Everyone wants their model to learn, not copy data, but overfitting happens sometimes and overfitting can look the same as copying.
Is there really any difference?
Anybody have an explanation as to why repeating a token would cause it to regurgitate memorized text?
lol I literally found the same attack months ago, posted to Reddit and nobody cared. https://www.reddit.com/r/ChatGPT/comments/156aaea/interestin...
FTA:
> It’s wild to us that our attack works and should’ve, would’ve, could’ve been found earlier.
Earlier quoted context omitted.
No, not at all, given training_data is in the hundreds of gigabytes, and this search would need to be run on every single token (for in-flight temperature adjustment).
There are tricks for that, e.g. bloom filters.
I think the classic Bloom filter is suitable when you have an exact-match operation but not directly suitable for a substring operation. E.g. you could put 500,000 names into the filter and it could tell you efficiently that "Jason Bourne" is probably one of those names, but not that "urn" is a component of one of them.
For the "is this output in the training data anywhere?" question, the most generally useful question might be somdthing like "are the last 200 tokens of output a verbatim substring of HUGE_TRAINING_STRING?".
A totally different challenge: presumably it's very often appropriate for some relatively large "popular" or "common" strings to actually be memorized and repeated on request. E.g., imagine asking a large language model for the text of the Lord's Prayer or the Pledge of Allegiance or the lyrics to some country's national anthem or something. The expected right answer is going to be that verbatim output.
If it weren't for copyright, this would probably also be true for many long strings that don't occur frequently in the training data, although it wouldn't be a high priority for model training because the LLM isn't a very efficient way to store tons of non-repetitive verbatim text.
lol I literally found the same attack months ago, posted to Reddit and nobody cared. https://www.reddit.com/r/ChatGPT/comments/156aaea/interestin...
The difference between screwing around and science is writing things down .... and publishing in a peer-reviewed journal.
Prompt: https://pastebin.com/Nm4jGttE
Not sure if I'm seeing training data or someone else's responses but it's odd. Here is my attempt: https://chat.openai.com/share/6b6ea43f-de2f-4ed5-917f-b6dcd6... pastebin of the output: https://pastebin.com/TdpkPmt6
The best part is it preserves the copyright notices from the training data. So we know that the model was obviously trained on copywritten data the legal question now is... if that is legal.
edit: Just got some random response that appears to be someone asking the model how to rekindle a romance after their partner got distant after an NDE seems personal so I will not post the paste here. This is pretty wild.
The funniest part is the model labeled this chat in the side bar as 'Decline to answer.'
edit2: It's definitely training data I seem to get some model response but after some time it turns into training data I've been able to locate some sources for the data.
Earlier quoted context omitted.
Normally it doesn't do that but they were using an "attack prompt". They ask the model to repeat a single word forever, it eventually deviates and generates normal text which has a higher rate of regurgitation than usual.
I don't know we can say it doesn't normally do this. What if more normal replies are just verbatim bits of training data, or multiple bits put together, but they're not specific or long enough that anyone's noticing? There's nothing specific to this "attack" that seems like it should make it output training data.