Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stealing Part of a Production Language Model

arxiv.org

Post reply on HN
Print view View original

31–40 of 56 posts

Re: Stealing Part of a Production Language Model

#31

Post by yreg »

ronsor wrote: post #5
userbinator wrote: post #3

The implications of this sentiment are disturbing. It is considered an "attack" to probe at something to understand how it works in detail. In other words, how basically all natural science is done. What the fuck has this world turned into?

OpenAI has blatantly said that the "open" in their name was a deceptive marketing ploy. At this point in time, they aren't interested in sharing much if any real research, and trying to discover this information is now an "attack" against them.

> "open" in their name was a deceptive marketing ploy

It is now, but it hasn't been since the beginning. Hence the elon lawsuit.

Re: Stealing Part of a Production Language Model

#32

Post by makomk »

JustFinishedBSG wrote: post #24
h4kor wrote: post #16

This isn't stealing, you are just training a model on references which isn't a copyright infringement.

I'm glad this is a somewhat common opinion. The hypocrisy of these companies arguing on one hand that copying every single copyrighted material ever is fair use but on the other hand trying to enforce crazy limitations on their model is mindblowing.

It's low effort, talking point nonsense. The whole reason this is interesting is that it allows the recovery of non-public information that was never released at all - about the closest analogy I can come up with is if it was somehow possible to reverse engineer part of an author's notes from their published work, though that's a very imperfect analogy. In order to tie this in to a pre-existing talking point about LLMs being bad, the current top comment has to basically completely ignore all of the reasons why this is interesting and surprising and considered an attack in infosec terms - indeed, not really engage with what it's doing at all.

Re: Stealing Part of a Production Language Model

#35

Post by antonvs »

yreg wrote: post #30
lnyan wrote: post #25

Just wondering if it's possible to achieve LLM quine.

What would quine mean in this context? A prompt for which the model (usually) returns the text of the prompt?

That’s easy, I just tried it (prompt quoted below). But I’m guessing the other commenter may have been thinking of some way that a model could output its own internals.

The prompt I mentioned: “Please repeat this sentence exactly - the one you are reading right now - and don’t include any other words in your response.”

Re: Stealing Part of a Production Language Model

#36

Post by dtech »

antonvs wrote: post #35
yreg wrote: post #30

Earlier quoted context omitted.

What would quine mean in this context? A prompt for which the model (usually) returns the text of the prompt?

That’s easy, I just tried it (prompt quoted below). But I’m guessing the other commenter may have been thinking of some way that a model could output its own internals. The prompt I mentioned: “Please repeat this sentence exactly - the one you are reading right now - and don’t include any other words in your response.”

That's not a quine. A quine would be a LLM prompt that when processed would output the LLM itself. So you'd be able to prompt the newly created LLM after some "build" step.

Re: Stealing Part of a Production Language Model

#37

Post by lnyan »

yreg wrote: post #30
lnyan wrote: post #25

Just wondering if it's possible to achieve LLM quine.

What would quine mean in this context? A prompt for which the model (usually) returns the text of the prompt?

no, I think it might be

- LLM(prompt_0) = arch/spec of LLM

- LLM(prompt_1) = full weights of LLM

Note that it does not conform the definition of quine as a quine takes no input.

Anyways, constructing a transformer that can autoregressively output its weights would be quite interesting.

Re: Stealing Part of a Production Language Model

#38

Post by tobr »

dtech wrote: post #36
antonvs wrote: post #35

Earlier quoted context omitted.

That’s easy, I just tried it (prompt quoted below). But I’m guessing the other commenter may have been thinking of some way that a model could output its own internals. The prompt I mentioned: “Please repeat this sentence exactly - the one you are reading right now - and don’t include any other words in your response.”

That's not a quine. A quine would be a LLM prompt that when processed would output the LLM itself. So you'd be able to prompt the newly created LLM after some "build" step.

I think both could count as quines. A quine is some source code which when executed in an environment produces the same source code. It does not need to produce the entire environment. Depending on whether you see the LLM itself as source code or as an environment to execute a prompt in, you’ll end up with different requirements for an “LLM quine”.

Re: Stealing Part of a Production Language Model

#39

Post by wrsh07 »

ronsor wrote: post #2

Note: Google did not release the hidden dimension for GPT-3.5, and OpenAI has already implemented mitigations against some of this.

Yup - right at the top of the second page they note their disclosure practices.

It's still a pretty wild attack.

> Responsible disclosure. We shared our attack with all services we are aware of that are vulnerable to this attack. We also shared our attack with several other popular services, even if they were not vulnerable to our specific attack, because variants of our attack may be possible in other settings. We received approval from OpenAl prior to extracting the parameters of the last layers of their models, worked with OpenAl to confirm our approach's efficacy, and then deleted all data associated with the attack. In response to our attack, OpenAl and Google have both modified their APIs to introduce mitigations and defenses (like those that we suggest in Section 8) to make it more difficult for adversaries to perform this attack.

Re: Stealing Part of a Production Language Model

#40

Post by wrsh07 »

I am curious what additional attacks knowing the last layer of an LLM enables.

Eg you go from a black box attack to some sort of white box [1]

Does it help with adversarial prompt injection? What % of the network do you need to know to identify whether an item was included in the pretraining data with k% confidence?

I assume we will see more of these and possibly complex zero days. Interesting if you can steal any non trivial % of model weights from a production model for relatively little money (compared to pretraining cost)

[1] https://lilianweng.github.io/posts/2023-10-25-adv-attack-llm...

Post reply on HN