Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Language models are injective and hence invertible

arxiv.org

Post reply on HN
Print view View original

81–90 of 164 posts

Re: Language models are injective and hence invertible

#81

Post by pu_pe »

adamddev1 wrote: post #52

Could this be a way to check for AI plagiarism? Given a chunk of text would you be able to (almost) prove that it came from a prompt saying "Write me a short essay on ___" ?

I first had this impression as well, but I can think of several complications:

1) You would have to know the exact model used, and you would need to have access to their weights

2) System prompt and temperature, not sure how this handles those

3) If anyone changes even a word in the output, this method would fall apart

Re: Language models are injective and hence invertible

#82

Post by qarl »

spuz wrote: post #24

I remember hearing an argument once that said LLMs must be capable of learning abstract ideas because the size of their weight model (typically GBs) is so much smaller than the size of their training data (typically TBs or PBs). So either the models are throwing away most of the training data, they are compressing the data beyond the known limits, or they are abstracting the data into more efficient forms. That's why…

> they are compressing the data beyond the known limits, or they are abstracting the data into more efficient forms.

I would argue that this is two ways of saying the same thing.

Compression is literally equivalent to understanding.

Re: Language models are injective and hence invertible

#84

Post by BurningFrog »

matt_kantor wrote: post #49
simiones wrote: post #29

Earlier quoted context omitted.

> If I am understanding this paper correctly, they are claiming that the model weights can be inverted in order to produce the original input text. No, that is not the claim at all. They are instead claiming that given an LLM output that is a summary of chapter 18 of Mary Shelley's Frankenstein, you can tell that the input prompt that led to this output was "give me a summary of chapter 18 of Mary Shelley's Frankenst…

> their claim may be something more complex, after reading the paper. I'm not sure that their result applies to the final output, or it's restricted to knowing the internal state at some pre-output layer. It's the internal state; that's what they mean by "hidden activations". If the claim were just about the output it'd be easy to falsify. For example, the prompts "What color is the sky? Answer in one word." and "Wha…

There are also billions of possible Yes/No questions you can ask that won't get unique answers.

Re: Language models are injective and hence invertible

#85

Post by etatoby »

sigmoid10 wrote: post #7

>we confirm this result empirically through billions of collision tests on six state-of-the-art language models, and observe no collisions This sounds like a mistake. They used (among others) GPT2, which has pretty big space vectors. They also kind of arbitrarily define a collision threshold as an l2 distance smaller than 10^-6 for two vectors. Since the outputs are normalized, that corresponds to a ridiculously tiny…

> Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal.

Which, incidentally, is the main reason why deep learning and LLM are effective in the first place.

A vector of a few thousands dimensions would be woefully inadequate to represent all of human knowledge, if not for the fact that it works as the projection of a much higher, potentially infinite-dimensional vector representing all possible knowledge. The smaller-sized one works in practice as a projection, precisely because any two such vectors are almost always orthogonal.

Re: Language models are injective and hence invertible

#86

Post by pfdietz »

vitus wrote: post #60
sigmoid10 wrote: post #7

>we confirm this result empirically through billions of collision tests on six state-of-the-art language models, and observe no collisions This sounds like a mistake. They used (among others) GPT2, which has pretty big space vectors. They also kind of arbitrarily define a collision threshold as an l2 distance smaller than 10^-6 for two vectors. Since the outputs are normalized, that corresponds to a ridiculously tiny…

I envy your intuition about high-dimensional spaces, as I have none (other than "here lies dragons"). (I think your intuition is broadly correct, seeing as billions of collision tests feels quite inadequate given the size of the space.) > Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal. What's the intuition here? Law of large numbers? And how is orthogonality related to…

> What's the intuition here? Law of large numbers?

"Concentration of measure"

https://en.wikipedia.org/wiki/Concentration_of_measure

Re: Language models are injective and hence invertible

#87

Post by fxwin »

lou1306 wrote: post #9
fxwin wrote: post #3

I don't like the title of this paper, since most people in this space probably think of language models not as producing a distribution (wrt which they are indeed invertible, which is what the paper claims) but as producing tokens (wrt which they are not invertible [0]). Also the author contribution statement made me laugh. [0] https://x.com/GladiaLab/status/1983812121713418606

Still, it is technically correct. The model produces a next-token likelihood distribution, then you apply a sampling strategy to produce a sequence of tokens.

I agree it is technically correct, but I still think it is the research paper equivalent of clickbait (and considering enough people misunderstood this for them to issue a semi-retraction that seems reasonable)

Re: Language models are injective and hence invertible

#88

Post by mjburgess »

Arnt wrote: post #37
sigmoid10 wrote: post #7

>we confirm this result empirically through billions of collision tests on six state-of-the-art language models, and observe no collisions This sounds like a mistake. They used (among others) GPT2, which has pretty big space vectors. They also kind of arbitrarily define a collision threshold as an l2 distance smaller than 10^-6 for two vectors. Since the outputs are normalized, that corresponds to a ridiculously tiny…

As I read it, what they did there was a sanity-check by trusting the birthday paradox. Kind of: "If you get orthogonal vectors due to mere chance once, that's okay, but you try it billions of times and still get orthogonal vectors every time, mere chance seems a very unlikely explanation."

That assumes the random process by which vectors are generated places them at random angles to each other, it doesnt, it places them almost always very very nearly at (high-dim) right angles

The underlying geometry isnt random, to this order, it's determinstic

Re: Language models are injective and hence invertible

#89

Post by fxwin »

usgroup wrote: post #83

Injective doesn’t mean bijective, and that seems obvious. That is, presumably very many inputs will map to the output “Yes”.

Afaict surjectivity was already a given before this paper, their contribution is the injectivity part (and thus invertibility)

Re: Language models are injective and hence invertible

#90

Post by ghm2180 »

There is actually a good analytical result on how vector similarity can easily fail to recover relevant information https://arxiv.org/pdf/2403.05440

> For some linear models the similarities are not even unique, while for others they are implicitly controlled by the regularization.

I am not strong in mathematics but if this paper claims run opposite to each other.

Post reply on HN