Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stealing Part of a Production Language Model

arxiv.org

Post reply on HN
Print view View original

51–56 of 56 posts

Re: Stealing Part of a Production Language Model

#51

Post by cs702 »

andy99 wrote: post #50
cs702 wrote: post #49

Earlier quoted context omitted.

It's still significant. When the Softmax head is the transpose of the embedding matrix, the proposed method enables extraction of the entire matrix of pretrained token embeddings from a black-box model at a shockingly low cost. If I understood this right, there's a lot of valuable information in those embeddings!

That's not always true, many models have separate weights for the embeddings and classifier heads

You're right. I edited my comment to qualify it with "when". Thank you!

Re: Stealing Part of a Production Language Model

#52

Post by sdenton4 »

lnyan wrote: post #37
yreg wrote: post #30

Earlier quoted context omitted.

What would quine mean in this context? A prompt for which the model (usually) returns the text of the prompt?

no, I think it might be - LLM(prompt_0) = arch/spec of LLM - LLM(prompt_1) = full weights of LLM Note that it does not conform the definition of quine as a quine takes no input. Anyways, constructing a transformer that can autoregressively output its weights would be quite interesting.

Easy if all the weights are zero...

Re: Stealing Part of a Production Language Model

#54

Post by andy99 »

userbinator wrote: post #3

The implications of this sentiment are disturbing. It is considered an "attack" to probe at something to understand how it works in detail. In other words, how basically all natural science is done. What the fuck has this world turned into?

It's getting information the org that created and hosts the model doesn't want you to have. Just because you think that information should be shared doesn't make it any less of an attack.

Re: Stealing Part of a Production Language Model

#55

Post by antonvs »

dtech wrote: post #36
antonvs wrote: post #35

Earlier quoted context omitted.

That’s easy, I just tried it (prompt quoted below). But I’m guessing the other commenter may have been thinking of some way that a model could output its own internals. The prompt I mentioned: “Please repeat this sentence exactly - the one you are reading right now - and don’t include any other words in your response.”

That's not a quine. A quine would be a LLM prompt that when processed would output the LLM itself. So you'd be able to prompt the newly created LLM after some "build" step.

I covered both options in my comment.

The prompt I gave is a true quine if you consider the prompt to be the "program", and the model to be the interpreter of the program.

The other option that you described isn't really a true quine, although it's quine-like. A quine is supposed to be a "program", which when "run" without any input, produces its own source code as output.

To be considered a quine in the strict sense, a model that outputs itself implies that you're treating the model as the program. In that case, if it needs a prompt in order to output itself, that breaks the quine rules, strictly speaking.

Post reply on HN