Language models are injective and hence invertible
1–10 of 164 posts
Re: Language models are injective and hence invertible
#2I wonder how these pieces of understanding can be applied to neuroscience.
Re: Language models are injective and hence invertible
#3I don't like the title of this paper, since most people in this space probably think of language models not as producing a distribution (wrt which they are indeed invertible, which is what the paper claims) but as producing tokens (wrt which they are not invertible [0]).
Also the author contribution statement made me laugh.
Re: Language models are injective and hence invertible
#4I don't like the title of this paper, since most people in this space probably think of language models not as producing a distribution (wrt which they are indeed invertible, which is what the paper claims) but as producing tokens (wrt which they are not invertible [0]). Also the author contribution statement made me laugh. [0] https://x.com/GladiaLab/status/1983812121713418606
Yeh it would be fun if we could reverse engineer the prompts from auto generated blog posts. But this is not quite the case.
Re: Language models are injective and hence invertible
#5Are the weights invertible, or are the prompts being fed into the model invertible?
Re: Language models are injective and hence invertible
#6I wonder how these pieces of understanding can be applied to neuroscience.
Real brains have a very different structure and composition. Neurons aren't monotonic, there are loops, etc.
Re: Language models are injective and hence invertible
#7>we confirm this result empirically through billions of collision tests on six state-of-the-art language models, and observe no collisions
This sounds like a mistake. They used (among others) GPT2, which has pretty big space vectors. They also kind of arbitrarily define a collision threshold as an l2 distance smaller than 10^-6 for two vectors. Since the outputs are normalized, that corresponds to a ridiculously tiny patch on the surface of the unit sphere. Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal. I would expect the chance of two inputs to map to the same output under these constraints to be astronomically small (like less than one in 10^10000 or something). Even worse than your chances of finding a hash collision in sha256. Their claim certainly does not sound like something you could verify by testing a few billion examples. Although I'd love to see a detailed calculation. The paper is certainly missing one.
Re: Language models are injective and hence invertible
#8I wonder how these pieces of understanding can be applied to neuroscience.
These pieces of "understanding" mostly don't even applied to other LLMs.
Re: Language models are injective and hence invertible
#9I don't like the title of this paper, since most people in this space probably think of language models not as producing a distribution (wrt which they are indeed invertible, which is what the paper claims) but as producing tokens (wrt which they are not invertible [0]). Also the author contribution statement made me laugh. [0] https://x.com/GladiaLab/status/1983812121713418606
Still, it is technically correct. The model produces a next-token likelihood distribution, then you apply a sampling strategy to produce a sequence of tokens.
Re: Language models are injective and hence invertible
#10Am I misunderstanding this?
Any stateful system that exposes state in a flexible way has risk to data exposure.
Does anyone actually think a stateful system wouldn’t release state?
Why not just write a paper “The sky may usually be blue”?