Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Extracting training data from ChatGPT

not-just-memorization.github.io

Post reply on HN
Print view View original

121–130 of 135 posts

Re: Extracting training data from ChatGPT

#121

Post by yencabulator »

AustinDev wrote: post #88

This attack still works. It hasn't been patched you just have to be a bit creative try this prompt on GPT 3.5 if you want to see how it works right now... until someone from OpenAI sees my post :D Prompt: https://pastebin.com/Nm4jGttE Not sure if I'm seeing training data or someone else's responses but it's odd. Here is my attempt: https://chat.openai.com/share/6b6ea43f-de2f-4ed5-917f-b6dcd6... pastebin of the output…

Well there's a definite regurgitation there:

From your pastebin:

> The Idaho Mountain Express is distributed free to residents and guests throughout the Sun Valley, Idaho resort area community. Subscribers to the Idaho Mountain Express will read these stories and others in this week's issue.

From https://classified.mtexpress.com/showads

> The Idaho Mountain Express is distributed free to residents and guests throughout the Sun Valley, Idaho resort area community. Subscribers to the Idaho Mountain Express will read these stories and others in this week's issue.

Re: Extracting training data from ChatGPT

#122

Post by yencabulator »

haolez wrote: post #95

Just tried this on GPT-4. It's kinda creepy: Sure, I'll repeat "company" for you: company company company company company company company company company company company company company company company company company company company company company company company company company company company company company company company company company company company company company company company company company company co…

All work and no play makes Jack a dull boy.

All work and no play makes Jack a dull boy.

All work and no play makes Jack a dull boy.

Re: Extracting training data from ChatGPT

#123

Post by zer0c00ler »

timfsu wrote: post #21
whywhywhywhy wrote: post #20

Earlier quoted context omitted.

Already has, https://www.bbc.co.uk/news/technology-65047304

That was a regular frontend bug though, not an issue with the LLM

Read OpenAI's response to this security issue carefully - it tells you a lot about how they think of being responsible for issues like this. I remember they put all the blame on the open source library, rather than taking responsibility themselves.

Re: Extracting training data from ChatGPT

#124

Post by amelius »

schoen wrote: post #86
amelius wrote: post #65

Earlier quoted context omitted.

There are tricks for that, e.g. bloom filters.

Is there a Bloom filter equivalent for efficiently searching whether a variable-length string is (or, more challenging, contains!) a substring of a very large string? I think the classic Bloom filter is suitable when you have an exact-match operation but not directly suitable for a substring operation. E.g. you could put 500,000 names into the filter and it could tell you efficiently that "Jason Bourne" is probably o…

Well, you could just slide a 10-token window over your dataset, and insert those (10-token tuples) into the bloom filter.

Re: Extracting training data from ChatGPT

#125

Post by marginalia_nu »

Alifatisk wrote: post #112
marginalia_nu wrote: post #100

Earlier quoted context omitted.

Question remains, how do we know they were part of the training data?

You mean chatGPT where able to scramble the exact same sentence with pure luck?

Well yes, pulling legit-looking text out of its ass is sort of what it does best.

Re: Extracting training data from ChatGPT

#126

Post by wokwokwok »

tallytarik wrote: post #111

I think this is misleading. I ran the same test when I heard about it a few months ago. When I tested it, I'd get back what looked like exact copies of Reddit threads, news articles, weird forum threads with usernames from the deepest corners of the internet. But I'd try to Google snippets of text, and no part of the generated text was anywhere to be found. I even went to the websites that forum threads were supposed…

The results in the paper (page 7) are empirical and reasonably convincing across both ChatGPT and a variety of other open source models.

Why do you think it’s misleading?

You think it’s just generating plausible random crap that happens to exist verbatim on the internet?

I mean… read the paper, 0.8% outputs were verbatim for gpt-3.5.

I’m not sure how you can plausibly claim that’s random chance.

> I think it’s a bug

It is a bug, but that doesn’t make it misleading or untrue.

This is like saying a security vuln in gmail that lets you steal 1% of mail is misleading. That would not be a bug, it would be a freaking disaster.

The problem here is that (as mentioned in other comments), training LLMs in a way that avoids this is actually pretty hard to do.

/shrug

Re: Extracting training data from ChatGPT

#127

Post by tallytarik »

wokwokwok wrote: post #126
tallytarik wrote: post #111

I think this is misleading. I ran the same test when I heard about it a few months ago. When I tested it, I'd get back what looked like exact copies of Reddit threads, news articles, weird forum threads with usernames from the deepest corners of the internet. But I'd try to Google snippets of text, and no part of the generated text was anywhere to be found. I even went to the websites that forum threads were supposed…

The results in the paper (page 7) are empirical and reasonably convincing across both ChatGPT and a variety of other open source models. Why do you think it’s misleading? You think it’s just generating plausible random crap that happens to exist verbatim on the internet? I mean… read the paper, 0.8% outputs were verbatim for gpt-3.5. I’m not sure how you can plausibly claim that’s random chance. > I think it’s a bug…

> You think it’s just generating plausible random crap that happens to exist verbatim on the internet? > I mean… read the paper, 0.8% outputs were verbatim for gpt-3.5.

Look at the sorts of outputs they claim are in the training data. Also note that their appendix includes huge chunks of text but they do not claim the entire chunk was matched to existing data — only a tiny amount of it.

The “bug” to me is something about losing its state and generating a random token. Now if that random token is “Afgh”, I’m not surprised it follows up with “Afghanistan” and a perfect list of countries in alphabetical order. I’m also not surprised that appears in training data, because it appears on thousands of webpages.

So it’s not that there isn’t an overlap between the GPT gibberish and internet content, and therefore likely training data. It’s that it’s not especially unique. If it were — like reproducing a one off Reddit thread verbatim — I think that would be greater cause for concern.

Re: Extracting training data from ChatGPT

#128

Post by Alifatisk »

marginalia_nu wrote: post #125
Alifatisk wrote: post #112

Earlier quoted context omitted.

You mean chatGPT where able to scramble the exact same sentence with pure luck?

Well yes, pulling legit-looking text out of its ass is sort of what it does best.

That’s not my point.

Spitting out content that looks legit is one thing, but spitting out text that matches something online exactly is more suspicious.

Re: Extracting training data from ChatGPT

#130

Post by [deleted] »

KeplerBoy wrote: post #114
c-linkage wrote: post #61

Earlier quoted context omitted.

The difference between screwing around and science is writing things down .... and publishing in a peer-reviewed journal.

Who cares about peer-reviews these days? Progress is happening in the open, progress is happening on GitHub and Arxive. Screw those journals with their peer-reviewed, yet irreproducible, papers without code or data.

[deleted]
Post reply on HN