Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Language models are injective and hence invertible

arxiv.org

Post reply on HN
Print view View original

71–80 of 164 posts

Re: Language models are injective and hence invertible

#71

Post by simiones »

matt_kantor wrote: post #49
simiones wrote: post #29

Earlier quoted context omitted.

> If I am understanding this paper correctly, they are claiming that the model weights can be inverted in order to produce the original input text. No, that is not the claim at all. They are instead claiming that given an LLM output that is a summary of chapter 18 of Mary Shelley's Frankenstein, you can tell that the input prompt that led to this output was "give me a summary of chapter 18 of Mary Shelley's Frankenst…

> their claim may be something more complex, after reading the paper. I'm not sure that their result applies to the final output, or it's restricted to knowing the internal state at some pre-output layer. It's the internal state; that's what they mean by "hidden activations". If the claim were just about the output it'd be easy to falsify. For example, the prompts "What color is the sky? Answer in one word." and "Wha…

Even that is not necessarily true. The output of the LLM is not "Blue". It is something like "probability of 'Blue' is 0.98131". And it may well be 0.98132 for the other question. Certainly they only talk about the internal state in 1 layer of the LLM, they don't need the entire LLM values.

Re: Language models are injective and hence invertible

#72

Post by TeMPOraL »

AlienRobot wrote: post #66
spuz wrote: post #24

I remember hearing an argument once that said LLMs must be capable of learning abstract ideas because the size of their weight model (typically GBs) is so much smaller than the size of their training data (typically TBs or PBs). So either the models are throwing away most of the training data, they are compressing the data beyond the known limits, or they are abstracting the data into more efficient forms. That's why…

I'm not sure if I would call it "abstracting." Imagine that you have an a spreadsheet that dates from the beginning of the universe to its end. It contains two columns: the date, and how many days it has been since the universe was born. That's very big spreadsheet with lots of data in it. If you plot it, it creates a seemingly infinite diagonal line. But it can be "abstracted" as Y=X. And that's what ML does.

That's literally what generalization is.

Re: Language models are injective and hence invertible

#73

Post by FeepingCreature »

stared wrote: post #51

It reminded me of "Text embeddings reveal almost as much as text" from 2023 ( https://news.ycombinator.com/item?id=37867635 ) - and yes, they do cite it. It has a huge implication for privacy. There is some "mental model" that embedding vectors are like hash - so you can store them in database, even though you would not store plain text. It is an incorrect assumption - as a good embedding stores ALL - not just the ge…

This mental model is also in direct contradiction to the whole purpose of the embedding, which is that the embedding describes the original text in a more interpretable form. If a piece of content in the original can be used for search, comparison etc., p much by definition it has to be stored in the embedding.

Similarly, this result can be rephrased as "Language Models process text." If the LLM wasn't invertible with regards to a piece of input text, it couldn't attend to this text either.

Re: Language models are injective and hence invertible

#74

Post by bonzini »

vitus wrote: post #60
sigmoid10 wrote: post #7

>we confirm this result empirically through billions of collision tests on six state-of-the-art language models, and observe no collisions This sounds like a mistake. They used (among others) GPT2, which has pretty big space vectors. They also kind of arbitrarily define a collision threshold as an l2 distance smaller than 10^-6 for two vectors. Since the outputs are normalized, that corresponds to a ridiculously tiny…

I envy your intuition about high-dimensional spaces, as I have none (other than "here lies dragons"). (I think your intuition is broadly correct, seeing as billions of collision tests feels quite inadequate given the size of the space.) > Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal. What's the intuition here? Law of large numbers? And how is orthogonality related to…

> What's the intuition here? Law of large numbers?

For unit vectors the cosine of the angle between them is a1*b1+a2*b2+...+an*bn.

Each of the terms has mean 0 and when you sum many of them the sum concentrates closer and closer to 0 (intuitively the positive and negative terms will tend to cancel out, and in fact the standard deviation is 1/√n).

Re: Language models are injective and hence invertible

#75

Post by gowld »

gnfargbl wrote: post #58
rowanG077 wrote: post #47

Earlier quoted context omitted.

I think they are saying that there is no proof of being injective. The argument with the hash is essentially saying, doing the same experiment with a hash would yield a similar result, yet hash function are not injective by definition. So from this experimental result you cannot conclude language models are injective. That's not really formally true, there are so called perfect hash functions that are injective over…

Sure, but the paper doesn't claim absolute injectivity. It claims injectivity for practical purposes ("almost surely injective"). That's the same standard to which we hold hash functions -- most of us would consider it reasonable to index an object store with SHA256.

The problem with "almost surely injective" for "practical purposes". Is that when you try to invert something, how do you know the result you get is one of those "practical purposes" ?

We aren't just trying to claim that two inputs are the same, as in hashing. We are trying to recover lost inputs.

Re: Language models are injective and hence invertible

#76

Post by Thorrez »

fn-mote wrote: post #48
Arnt wrote: post #37

Earlier quoted context omitted.

As I read it, what they did there was a sanity-check by trusting the birthday paradox. Kind of: "If you get orthogonal vectors due to mere chance once, that's okay, but you try it billions of times and still get orthogonal vectors every time, mere chance seems a very unlikely explanation."

Edit: there are other clarifications, eg authors on X, so this comment is irrelevant. The birthday paradox relies on there being a small number of possible birthdays (365-366). There are not a small number of dimensions being used in the LLM. The GP argument makes sense to me.

The birthday paradox equation is approximately the square root. You expect to find a collision in 365 possibilities in ~sqrt(365) = ~19 tries.

You expect to find a collision in 2^256 possibilities in ~sqrt(2^256) = ~2^128 tries.

You expect to find a collision in 10^10000 possibilities in ~sqrt(10^10000) = ~10^5000 tries.

Re: Language models are injective and hence invertible

#77

Post by setopt »

vitus wrote: post #60
sigmoid10 wrote: post #7

>we confirm this result empirically through billions of collision tests on six state-of-the-art language models, and observe no collisions This sounds like a mistake. They used (among others) GPT2, which has pretty big space vectors. They also kind of arbitrarily define a collision threshold as an l2 distance smaller than 10^-6 for two vectors. Since the outputs are normalized, that corresponds to a ridiculously tiny…

I envy your intuition about high-dimensional spaces, as I have none (other than "here lies dragons"). (I think your intuition is broadly correct, seeing as billions of collision tests feels quite inadequate given the size of the space.) > Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal. What's the intuition here? Law of large numbers? And how is orthogonality related to…

> > Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal.

> What's the intuition here? Law of large numbers?

Imagine for simplicity that we consider only vectors pointing parallel/antiparallel to coordinate axes.

- In 1D, you have two possibilities: {+e_x, -e_x}. So if you pick two random vectors from this set, the probability of getting something orthogonal is 0.

- In 2D, you have four possibilities: {±e_x, ±e_y}. If we pick one random vector and get e.g. +e_x, then picking another one randomly from the set has a 50% chance of getting something orthogonal (±e_y are 2/4 possibilities). Same for other choices of the first vector.

- In 3D, you have six possibilities: {±e_x, ±e_y, ±e_z}. Repeat the same experiment, and you'll find a 66.7% chance of getting something orthogonal.

- In the limit of ND, you can see that the chance of getting something orthogonal is 1 - 1/N, which tends to 100% as N becomes large.

Now, this discretization is a simplification of course, but I think it gets the intuition right.

Re: Language models are injective and hence invertible

#78

Post by gowld »

stared wrote: post #51

It reminded me of "Text embeddings reveal almost as much as text" from 2023 ( https://news.ycombinator.com/item?id=37867635 ) - and yes, they do cite it. It has a huge implication for privacy. There is some "mental model" that embedding vectors are like hash - so you can store them in database, even though you would not store plain text. It is an incorrect assumption - as a good embedding stores ALL - not just the ge…

> There is an easy fix to that - a random rotation; preserves all distances.

Is that like homomorphic encryption, in a sense, where you can calculate the encryption of a function on the plaintext, without ever seeing the input or calculated function of plaintext.

Re: Language models are injective and hence invertible

#79

Post by gnfargbl »

gowld wrote: post #75
gnfargbl wrote: post #58

Earlier quoted context omitted.

Sure, but the paper doesn't claim absolute injectivity. It claims injectivity for practical purposes ("almost surely injective"). That's the same standard to which we hold hash functions -- most of us would consider it reasonable to index an object store with SHA256.

The problem with "almost surely injective" for "practical purposes". Is that when you try to invert something, how do you know the result you get is one of those "practical purposes" ? We aren't just trying to claim that two inputs are the same, as in hashing. We are trying to recover lost inputs.

You don't, I guess. But again that's just the same as when you insert something into an object store: you can't be absolutely certain that a future retrieval will give you the same object and not a colliding blob. It's just good enough for all practical purposes.

Re: Language models are injective and hence invertible

#80

Post by [deleted] »

sigmoid10 wrote: post #7

>we confirm this result empirically through billions of collision tests on six state-of-the-art language models, and observe no collisions This sounds like a mistake. They used (among others) GPT2, which has pretty big space vectors. They also kind of arbitrarily define a collision threshold as an l2 distance smaller than 10^-6 for two vectors. Since the outputs are normalized, that corresponds to a ridiculously tiny…

[deleted]
Post reply on HN