Author of related work here. This is very cool! I was hoping that they would try to invert layer by layer from the output to the input but it seems that they do a search process at the input layer instead. They rightly point out the residual connections make a layer by layer approach difficult. I may point out though that an rmsnorm layer should be invertible due to the epsilon term in the denominator which can be used to recover the input magnitude
Language models are injective and hence invertible
121–130 of 164 posts
Re: Language models are injective and hence invertible
#122A few critiques:
- If you have a feature detector function (f(x) = 0 when feature is not present, f(x) = 1 when feature is present) and you train a network to compute f(x), or some subset of the network "decides on its own during training" to compute f(x), doesn't that create a zero set of non-zero measure if training continues long enough?
- What happens when the middle layers are of much lower dimension than the input?
- Real analyticity means infinitely many derivatives (according to Appendix A). Does this mean the results don't apply to functions with corners (e.g. ReLU)?
Re: Language models are injective and hence invertible
#123Earlier quoted context omitted.
I envy your intuition about high-dimensional spaces, as I have none (other than "here lies dragons"). (I think your intuition is broadly correct, seeing as billions of collision tests feels quite inadequate given the size of the space.) > Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal. What's the intuition here? Law of large numbers? And how is orthogonality related to…
> > Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal. > What's the intuition here? Law of large numbers? Imagine for simplicity that we consider only vectors pointing parallel/antiparallel to coordinate axes. - In 1D, you have two possibilities: {+e_x, -e_x}. So if you pick two random vectors from this set, the probability of getting something orthogonal is 0. - In 2D, y…
I think that's a good answer for practical purposes.
Theoretically, I can claim that N random vectors of zero-mean real numbers (say standard deviation of 1 per element) will "with probability 1" span an N-dimensional space. I can even grind on, subtracting the parallel parts of each vector pair, until I have N orthogonal vectors. ("Gram-Schmidt" from high school.) I believe I can "prove" that.
So then mapping using those vectors is "invertible." Nyeah. But back in numerical reality, I think the resulting inverse will become practically useless as N gets large.
That's without the nonlinear elements. Which are designed to make the system non-invertible. It's not shocking if someone proves mathematically that this doesn't quite technically work. I think it would only be interesting if they can find numerically useful inverses for an LLM that has interesting behavior.
All -- I haven't thought very clearly about this. If I've screwed something up, please correct me gently but firmly. Thanks.
Re: Language models are injective and hence invertible
#124I remember hearing an argument once that said LLMs must be capable of learning abstract ideas because the size of their weight model (typically GBs) is so much smaller than the size of their training data (typically TBs or PBs). So either the models are throwing away most of the training data, they are compressing the data beyond the known limits, or they are abstracting the data into more efficient forms. That's why…
> they are compressing the data beyond the known limits, or they are abstracting the data into more efficient forms. I would argue that this is two ways of saying the same thing. Compression is literally equivalent to understanding.
If we use gzip to compress a calculus textbook does that mean that gzip understands calculus?
Re: Language models are injective and hence invertible
#125Earlier quoted context omitted.
> > Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal. > What's the intuition here? Law of large numbers? Imagine for simplicity that we consider only vectors pointing parallel/antiparallel to coordinate axes. - In 1D, you have two possibilities: {+e_x, -e_x}. So if you pick two random vectors from this set, the probability of getting something orthogonal is 0. - In 2D, y…
for 768 dimensions, you'd still expect to hit (1-1/N) with a few billion samples though. Like that's a 1/N of 0.13%, which quite frankly isn't that rare at all? Of course are vectors are not only points in one coordinate axes, but it still isn't that small compared to billions of samples.
[deleted]
Re: Language models are injective and hence invertible
#126Does that mean that all these embeddings in all those vector databases can be used to extract all these secret original documents?
Re: Language models are injective and hence invertible
#127Earlier quoted context omitted.
Still, it is technically correct. The model produces a next-token likelihood distribution, then you apply a sampling strategy to produce a sequence of tokens.
I agree it is technically correct, but I still think it is the research paper equivalent of clickbait (and considering enough people misunderstood this for them to issue a semi-retraction that seems reasonable)
I disagree. Within the research community (which is the target of the paper), that title means something very precise and not at all clickbaity. It's irrelevant that the rest of the Internet has an inaccurate notion of "model" and other very specific terms.
Re: Language models are injective and hence invertible
#128Isn't a requirement for injectivity that each input must map to 1 output? Where LLMs can result in the same output given multiple different inputs?
Re: Language models are injective and hence invertible
#129Earlier quoted context omitted.
> their claim may be something more complex, after reading the paper. I'm not sure that their result applies to the final output, or it's restricted to knowing the internal state at some pre-output layer. It's the internal state; that's what they mean by "hidden activations". If the claim were just about the output it'd be easy to falsify. For example, the prompts "What color is the sky? Answer in one word." and "Wha…
There are also billions of possible Yes/No questions you can ask that won't get unique answers.
If the authors are correct (I'm not equipped to judge) then there must be additional output which is thrown away before the user is presented with their yes/no, which can be used to recover the prompt.
It would be pretty cool if this were true. One could annotate results with this metadata as a way of citing sources.
Re: Language models are injective and hence invertible
#130Does that mean that all these embeddings in all those vector databases can be used to extract all these secret original documents?
I don't think vector databases are intended to be secure, encrypted forms of data storage in the first place.