Scalable extraction of training data from (production) language models
1–10 of 15 posts
Re: Scalable extraction of training data from (production) language models
#2Related (blog post from the team),
Extracting training data from ChatGPT (https://news.ycombinator.com/item?id=38458683) (126 comments)
And direct link,
https://not-just-memorization.github.io/extracting-training-...
Re: Scalable extraction of training data from (production) language models
#3This is probably an unintended feature bit I see no immediate problem in alignment not erasing raw memorized data. It could as well be a design choice to have a raw memory unaffected by alignment procedures
Re: Scalable extraction of training data from (production) language models
#4>This leads to a natural question that has not yet been dis-
cussed in the literature: if we could query a model infinitely,
how much memorization could we extract in total?
You will get every 50-grams, not because the model memorized all of them but by pure chance. It seems pretty obvious to me.
It makes me question if there were some cases where the model output an identical 50-grams but it wasn't present in the training dataset of the model, like in a very structured setting, like assembly code where there is usually a very limited number of keywords used.
Re: Scalable extraction of training data from (production) language models
#5>This leads to a natural question that has not yet been dis- cussed in the literature: if we could query a model infinitely, how much memorization could we extract in total? You will get every 50-grams, not because the model memorized all of them but by pure chance. It seems pretty obvious to me. It makes me question if there were some cases where the model output an identical 50-grams but it wasn't present in the tr…
One can fine tune a smaller parameter model like GPT-NeoX on a home GPU pretty readily, and it's absolutely capable of doing what you specified. Teach it with a bunch of example sentences that have parts of speech like verb and noun following a simple grammar, and you will see it generate sentences afterward that combine the parts of speech grammatically in novel ways, using the same grammatical structures but forming productions that did not appear in the training set.
Depending on settings, they are also capable of producing a lot of ungrammatical nonsense, but the odds of what it produces are changed considerably by the training.
Re: Scalable extraction of training data from (production) language models
#6>This leads to a natural question that has not yet been dis- cussed in the literature: if we could query a model infinitely, how much memorization could we extract in total? You will get every 50-grams, not because the model memorized all of them but by pure chance. It seems pretty obvious to me. It makes me question if there were some cases where the model output an identical 50-grams but it wasn't present in the tr…
One can fine tune a smaller parameter model like GPT-NeoX on a home GPU pretty readily, and it's absolutely capable of doing what you specified. Teach it with a bunch of example sentences that have parts of speech like verb and noun following a simple grammar, and you will see it generate sentences afterward that combine the parts of speech grammatically in novel ways, using the same grammatical structures but formin…
No I mean creating 50-grams that appear in the dataset created by the paper linked by OP, but not present in the actual dataset the model was trained on. Of course, the model would be able to output 50-grams that were not present in either.
Re: Scalable extraction of training data from (production) language models
#7An LLM remembers like a human. Mostly concepts, but some things it remembers verbatim.
Why is it a problem if a LLM tells you what it knows?
Are LLMs trained on secret data?
Re: Scalable extraction of training data from (production) language models
#8An LLM remembers like a human. Mostly concepts, but some things it remembers verbatim. Why is it a problem if a LLM tells you what it knows? Are LLMs trained on secret data?
> Are LLMs trained on secret data?
Probably. And on copyrighted data probably as well.
Re: Scalable extraction of training data from (production) language models
#9An LLM remembers like a human. Mostly concepts, but some things it remembers verbatim. Why is it a problem if a LLM tells you what it knows? Are LLMs trained on secret data?
DeepMind recently extracted PII from ChatGPT by prompting (e.g., telling the LLM to repeat 'poem' indefinitely will cause a long sequence of that word until popping out of it and revealing by accident some PII from a person's email signature).
So, yes.
Re: Scalable extraction of training data from (production) language models
#10Interesting! This aligns with my hunch that we might soon start to actually store much of data in models rather than unwieldly datasets :) ... as I've been writing about:
https://livingsystems.substack.com/p/the-future-of-data-less...