Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Language models are injective and hence invertible

arxiv.org

Post reply on HN
Print view View original

101–110 of 164 posts

Re: Language models are injective and hence invertible

#101

Post by ajkjk »

gowld wrote: post #75
gnfargbl wrote: post #58

Earlier quoted context omitted.

Sure, but the paper doesn't claim absolute injectivity. It claims injectivity for practical purposes ("almost surely injective"). That's the same standard to which we hold hash functions -- most of us would consider it reasonable to index an object store with SHA256.

The problem with "almost surely injective" for "practical purposes". Is that when you try to invert something, how do you know the result you get is one of those "practical purposes" ? We aren't just trying to claim that two inputs are the same, as in hashing. We are trying to recover lost inputs.

Well that's not a problem, that's just a description of what "almost surely" means. The thesis is "contrary to popular opinion, you can more-or-less invert the model". Not exactly invert it--don't use it in court!--but like, mostly. The prevailing wisdom that you cannot is incorrect.

Re: Language models are injective and hence invertible

#102

Post by Arnt »

program_whiz wrote: post #98
Arnt wrote: post #91

Earlier quoted context omitted.

The number of dimensions used is 768, wrote someone, and that isn't really very different from 365. But even if the number were big were were big, it could hardly escape fate: x has to be very big to keep (1-(1/x))¹⁰⁰⁰⁰⁰⁰⁰⁰⁰ near 1.

Just to clarify, the total dimension of birthdays is 365 (Jan 1 through Dec 31), but a 768 dimension continuous vector means there are 768 numbers, each of which can have values from -1 to 1 (at whatever precision floating point can represent). 1 float has about 2B numbers between -1 and 1 iirc, so 2B ^ 768 is a lot more than 365.

I may have misunderstood — don't they test for orthogonality? Orthogonality would seem to drop much of the information in the vectors.

Re: Language models are injective and hence invertible

#103

Post by [deleted] »

gnfargbl wrote: post #25
sigmoid10 wrote: post #7

>we confirm this result empirically through billions of collision tests on six state-of-the-art language models, and observe no collisions This sounds like a mistake. They used (among others) GPT2, which has pretty big space vectors. They also kind of arbitrarily define a collision threshold as an l2 distance smaller than 10^-6 for two vectors. Since the outputs are normalized, that corresponds to a ridiculously tiny…

The nature of high-dimensional spaces kind of intuitively supports the argument for invertability though, no? In the sense that: > I would expect the chance of two inputs to map to the same output under these constraints to be astronomically small.

[deleted]

Re: Language models are injective and hence invertible

#104

Post by empiricus »

ahartmetz wrote: post #94
etatoby wrote: post #85

Earlier quoted context omitted.

> Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal. Which, incidentally, is the main reason why deep learning and LLM are effective in the first place. A vector of a few thousands dimensions would be woefully inadequate to represent all of human knowledge, if not for the fact that it works as the projection of a much higher, potentially infinite-dimensional vector repres…

Two random vectors are almost always neither collinear nor orthogonal. So what you mean is either "not collinear", which is a trivial statement, or something like "their dot product is much smaller than abs(length(vecA) * length(vecB))", which is probably interesting but still not very clear.

Well, the actual interesting part is that when the vector dimension grows then random vectors will become almost orthogonal. smth smth exponential number of almost orthogonal vectors. this is probably the most important reason why text embedding is working. you take some structure from a 10^6 dimension, and project it to 10^3 dimension, and you can still keep the distances between all vectors.

Re: Language models are injective and hence invertible

#106

Post by orbital-decay »

CGMthrowaway wrote: post #97

Summary from the authors: -Different prompts always map to different embeddings, and this property can be used to recover input tokens from individual embeddings in latent space - Injectivity is not accidental, but a structural property of language models - Across billions of prompt pairs and several model sizes, we find no collisions: no two prompts are mapped to the same hidden states - We introduce SipIt, an algor…

> - This impacts privacy, deletion, and compliance

Surely that's a stretch... Typically, the only thing that leaves a transformer is its output text, which cannot be used to recover the input.

Re: Language models are injective and hence invertible

#107

Post by mattlutze »

KeplerBoy wrote: post #18
blackbear_ wrote: post #16

Earlier quoted context omitted.

Tailoring the message to the audience is really a fundamental principle of good communication. Scientists and academics demand an entirely different level of rigor compared to customers of LLM providers.

Sure, but they went slightly overboard with that headline and they knew it. But oh well, they have a lot of eyes and discussion on their paper so it's a success.

I feel like, if the feedback to your paper is "this is over-done / they claim more than they prove / it's kinda hype-ish" you're going to get less references in future papers.

That would seem to be counter to the "impact" goal for research.

Re: Language models are injective and hence invertible

#108

Post by D-Machine »

Authors: Giorgos Nikolaou‡*, Tommaso Mencattini†‡*, Donato Crisostomi†, Andrea Santilli†, Yannis Panagakis§¶, Emanuele Rodolà†

†Sapienza University of Rome

‡EPFL

§University of Athens

¶Archimedes RC

*Equal contribution; author order settled via Mario Kart.

Re: Language models are injective and hence invertible

#109

Post by weinzierl »

fxwin wrote: post #3

I don't like the title of this paper, since most people in this space probably think of language models not as producing a distribution (wrt which they are indeed invertible, which is what the paper claims) but as producing tokens (wrt which they are not invertible [0]). Also the author contribution statement made me laugh. [0] https://x.com/GladiaLab/status/1983812121713418606

Spoiler but for those who do not want to open the paper, the contribution statement is:

"Equal contribution; author order settled via Mario Kart."

If only more conflicts in life would be settled via Mario Kart.

Re: Language models are injective and hence invertible

#110

Post by joaohaas »

simiones wrote: post #71
matt_kantor wrote: post #49

Earlier quoted context omitted.

> their claim may be something more complex, after reading the paper. I'm not sure that their result applies to the final output, or it's restricted to knowing the internal state at some pre-output layer. It's the internal state; that's what they mean by "hidden activations". If the claim were just about the output it'd be easy to falsify. For example, the prompts "What color is the sky? Answer in one word." and "Wha…

Even that is not necessarily true. The output of the LLM is not "Blue". It is something like "probability of 'Blue' is 0.98131". And it may well be 0.98132 for the other question. Certainly they only talk about the internal state in 1 layer of the LLM, they don't need the entire LLM values.

That's exactly what the quoted answer is saying though?
Post reply on HN