Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Language models are injective and hence invertible

arxiv.org

Post reply on HN
Print view View original

91–100 of 164 posts

Re: Language models are injective and hence invertible

#91

Post by Arnt »

fn-mote wrote: post #48
Arnt wrote: post #37

Earlier quoted context omitted.

As I read it, what they did there was a sanity-check by trusting the birthday paradox. Kind of: "If you get orthogonal vectors due to mere chance once, that's okay, but you try it billions of times and still get orthogonal vectors every time, mere chance seems a very unlikely explanation."

Edit: there are other clarifications, eg authors on X, so this comment is irrelevant. The birthday paradox relies on there being a small number of possible birthdays (365-366). There are not a small number of dimensions being used in the LLM. The GP argument makes sense to me.

The number of dimensions used is 768, wrote someone, and that isn't really very different from 365. But even if the number were big were were big, it could hardly escape fate: x has to be very big to keep (1-(1/x))¹⁰⁰⁰⁰⁰⁰⁰⁰⁰ near 1.

Re: Language models are injective and hence invertible

#93

Post by NoMoreNicksLeft »

simiones wrote: post #29
spuz wrote: post #24

I remember hearing an argument once that said LLMs must be capable of learning abstract ideas because the size of their weight model (typically GBs) is so much smaller than the size of their training data (typically TBs or PBs). So either the models are throwing away most of the training data, they are compressing the data beyond the known limits, or they are abstracting the data into more efficient forms. That's why…

> If I am understanding this paper correctly, they are claiming that the model weights can be inverted in order to produce the original input text. No, that is not the claim at all. They are instead claiming that given an LLM output that is a summary of chapter 18 of Mary Shelley's Frankenstein, you can tell that the input prompt that led to this output was "give me a summary of chapter 18 of Mary Shelley's Frankenst…

I was under the impression that without also forcing the exact seed (which is randomly chosen and usually obfuscated), even providing the same exact prompt is unlikely to provide the same exact summary. In other words, under normal circumstances you can't even prove that a prompt and response are linked.

Re: Language models are injective and hence invertible

#94

Post by ahartmetz »

etatoby wrote: post #85
sigmoid10 wrote: post #7

>we confirm this result empirically through billions of collision tests on six state-of-the-art language models, and observe no collisions This sounds like a mistake. They used (among others) GPT2, which has pretty big space vectors. They also kind of arbitrarily define a collision threshold as an l2 distance smaller than 10^-6 for two vectors. Since the outputs are normalized, that corresponds to a ridiculously tiny…

> Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal. Which, incidentally, is the main reason why deep learning and LLM are effective in the first place. A vector of a few thousands dimensions would be woefully inadequate to represent all of human knowledge, if not for the fact that it works as the projection of a much higher, potentially infinite-dimensional vector repres…

Two random vectors are almost always neither collinear nor orthogonal. So what you mean is either "not collinear", which is a trivial statement, or something like "their dot product is much smaller than abs(length(vecA) * length(vecB))", which is probably interesting but still not very clear.

Re: Language models are injective and hence invertible

#95

Post by diffeomorphism »

RossBencina wrote: post #62
morcus wrote: post #46

Earlier quoted context omitted.

I don't think they're claiming surjectivity here. They're just saying the mapping is injective, so for a given output there should be a unique input to construct it.

> I don't think they're claiming surjectivity here. What definition of invertible doesn't include surjectivity?

Many? Just add a "by restricting to the image, we may wlog assume surjectivity".

The question is usually more about whether the inverse is also continuous, smooth, easy to compute....etc.

Re: Language models are injective and hence invertible

#96

Post by BenoitP »

vitus wrote: post #60
sigmoid10 wrote: post #7

>we confirm this result empirically through billions of collision tests on six state-of-the-art language models, and observe no collisions This sounds like a mistake. They used (among others) GPT2, which has pretty big space vectors. They also kind of arbitrarily define a collision threshold as an l2 distance smaller than 10^-6 for two vectors. Since the outputs are normalized, that corresponds to a ridiculously tiny…

I envy your intuition about high-dimensional spaces, as I have none (other than "here lies dragons"). (I think your intuition is broadly correct, seeing as billions of collision tests feels quite inadequate given the size of the space.) > Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal. What's the intuition here? Law of large numbers? And how is orthogonality related to…

> > Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal.

> What's the intuition here? Law of large numbers?

Yep, the large number being the number of dimensions.

As you add another dimension to a random point on a unit sphere, you create another new way for this point to be far away from a starting neighbor. Increase the dimensions a lot and then all random neighbors are on the equator from the starting neighbor. The equator being a 'hyperplane' (just like a 2D plane in 3D) of dimension n-1, the normal of which is the starting neighbor, intersected with the unit sphere (thus becoming a n-2 dimensional 'variety', or shape, embedded in the original n dimensional space; like the earth's equator is 1 dimensional object).

The mathematical name for this is 'concentration of measure' [1]

It feels weird to think about it, but there's also a unit change in here. Paris is about 1/8 of the circle far away from the north pole (8 such angle segments of freedom). On a circle. But if that's the definition of location of Paris, on the 3D earth there would be an infinity of Paris. There is only one though. Now if we take into account longitude, we have Montreal, Vancouver, Tokyo, etc ; each 1/8 away (and now we have 64 solid angle segments of freedom)

[1] https://www.johndcook.com/blog/2017/07/13/concentration_of_m...

Re: Language models are injective and hence invertible

#97

Post by CGMthrowaway »

Summary from the authors:

-Different prompts always map to different embeddings, and this property can be used to recover input tokens from individual embeddings in latent space

- Injectivity is not accidental, but a structural property of language models

- Across billions of prompt pairs and several model sizes, we find no collisions: no two prompts are mapped to the same hidden states

- We introduce SipIt, an algorithm that exactly reconstructs the input from hidden states in guaranteed linear time.

- This impacts privacy, deletion, and compliance: once data enters a Transformer, it remains recoverable.

Re: Language models are injective and hence invertible

#98

Post by program_whiz »

Arnt wrote: post #91
fn-mote wrote: post #48

Earlier quoted context omitted.

Edit: there are other clarifications, eg authors on X, so this comment is irrelevant. The birthday paradox relies on there being a small number of possible birthdays (365-366). There are not a small number of dimensions being used in the LLM. The GP argument makes sense to me.

The number of dimensions used is 768, wrote someone, and that isn't really very different from 365. But even if the number were big were were big, it could hardly escape fate: x has to be very big to keep (1-(1/x))¹⁰⁰⁰⁰⁰⁰⁰⁰⁰ near 1.

Just to clarify, the total dimension of birthdays is 365 (Jan 1 through Dec 31), but a 768 dimension continuous vector means there are 768 numbers, each of which can have values from -1 to 1 (at whatever precision floating point can represent). 1 float has about 2B numbers between -1 and 1 iirc, so 2B ^ 768 is a lot more than 365.

Re: Language models are injective and hence invertible

#99

Post by AlienRobot »

TeMPOraL wrote: post #72
AlienRobot wrote: post #66

Earlier quoted context omitted.

I'm not sure if I would call it "abstracting." Imagine that you have an a spreadsheet that dates from the beginning of the universe to its end. It contains two columns: the date, and how many days it has been since the universe was born. That's very big spreadsheet with lots of data in it. If you plot it, it creates a seemingly infinite diagonal line. But it can be "abstracted" as Y=X. And that's what ML does.

That's literally what generalization is.

I don't think it's the same thing because an abstraction is still tangible. For example, "rectangle" is an abstraction for all sorts of actual rectangular shapes you can find in practice. We have a way to define what a rectangle is and to identify one.

A neural network doesn't have any actual conceptual backing for what it is doing. It's pure math. There are no abstracted properties beyond the fact that by coincidence the weights make a curve fit certain points of data.

If there was truly a conceptual backing for these "abstractions" then multiple models trained on the same data should have very similar weights as there aren't multiple ways to define the same concepts, but I doubt that this happens in practice. Instead the weights are just randomly adjusted until they fit the points of data without any respect given to whether there is any sort of cohesion. It's just math.

Re: Language models are injective and hence invertible

#100

Post by [deleted] »

program_whiz wrote: post #98
Arnt wrote: post #91

Earlier quoted context omitted.

The number of dimensions used is 768, wrote someone, and that isn't really very different from 365. But even if the number were big were were big, it could hardly escape fate: x has to be very big to keep (1-(1/x))¹⁰⁰⁰⁰⁰⁰⁰⁰⁰ near 1.

Just to clarify, the total dimension of birthdays is 365 (Jan 1 through Dec 31), but a 768 dimension continuous vector means there are 768 numbers, each of which can have values from -1 to 1 (at whatever precision floating point can represent). 1 float has about 2B numbers between -1 and 1 iirc, so 2B ^ 768 is a lot more than 365.

[deleted]
Post reply on HN