21–30 of 135 posts
Earlier quoted context omitted.
If ChatGPT ever outputs other user's chat history, the company is as good as dead. If that could be exploited using this technique that is out in the wild for over a year: show me the data.
Already has, https://www.bbc.co.uk/news/technology-65047304
Thanks guys because the attack version in all its glee does not bother to mention if this affects ChatGPT using GPT 4.0.
Oh wait, it does say you’ve exploited a vulnerability in “Open AI’s flagship product”, so it’s all clear now. On to your paper for the breakthrough!…
[Our attack on ChatGPT (gpt-3.5-turbo) is specific to this model and is not applicable to any other production model]
Glad I’m only using ChatGPT Premium with GPT4 and not their flagship product.
Earlier quoted context omitted.
I think it may change the discussion about copyright a bit. I've seen many arguments that while GPTs are trained on copyrighted material, they don't parrot it back verbatim and their output is highly transformative. This shows pretty clearly that the models do retain and return large chunks of texts exactly how they read them.
I suspect ChatGPT is using a form of clean-room design to keep copyrighted material out of the training set of deployed models. One model is trained on copyrighted works in a jurisdiction where this is allowed and outputs "transformative" summaries of book chapters. This serves as training data for the deployed model.
They patched that real quick.
Earlier quoted context omitted.
Already has, https://www.bbc.co.uk/news/technology-65047304
That was a regular frontend bug though, not an issue with the LLM
They patched that real quick.
Anybody have an explanation as to why repeating a token would cause it to regurgitate memorized text?
I think the idea is just to have it lose "train of thought" because there aren't any high-probability completions to a long run of repeated words. So the next time there's a bit of entropy thrown in (the "temperature" setting meant to prevent LLMs from being too repetitive), it just latches onto something completely random.
I don’t know if this is true. But I haven’t seen an LLM spit out 50 token sequences of training data. By definition (an LLM as a “compressor”) this shouldn’t happen.
The paper notes 5 of 11 researchers are affiliated with Google, but it seems to be 11 of 11 if you count having received a paycheck from Google in some form current/past/intern/etc.
I can think of a couple generous interpretations I’d prefer to make, for example maybe it’s simply their models are not mature enough?
However is research right, not competitive analysis? I think at least a footnote mentioning it would be helpful.