111–120 of 164 posts
Earlier quoted context omitted.
I don't think they're claiming surjectivity here. They're just saying the mapping is injective, so for a given output there should be a unique input to construct it.
> I don't think they're claiming surjectivity here. What definition of invertible doesn't include surjectivity?
Summary from the authors: -Different prompts always map to different embeddings, and this property can be used to recover input tokens from individual embeddings in latent space - Injectivity is not accidental, but a structural property of language models - Across billions of prompt pairs and several model sizes, we find no collisions: no two prompts are mapped to the same hidden states - We introduce SipIt, an algor…
> - This impacts privacy, deletion, and compliance Surely that's a stretch... Typically, the only thing that leaves a transformer is its output text, which cannot be used to recover the input.
Summary from the authors: -Different prompts always map to different embeddings, and this property can be used to recover input tokens from individual embeddings in latent space - Injectivity is not accidental, but a structural property of language models - Across billions of prompt pairs and several model sizes, we find no collisions: no two prompts are mapped to the same hidden states - We introduce SipIt, an algor…
> - This impacts privacy, deletion, and compliance Surely that's a stretch... Typically, the only thing that leaves a transformer is its output text, which cannot be used to recover the input.
Earlier quoted context omitted.
I think they are saying that there is no proof of being injective. The argument with the hash is essentially saying, doing the same experiment with a hash would yield a similar result, yet hash function are not injective by definition. So from this experimental result you cannot conclude language models are injective. That's not really formally true, there are so called perfect hash functions that are injective over…
Sure, but the paper doesn't claim absolute injectivity. It claims injectivity for practical purposes ("almost surely injective"). That's the same standard to which we hold hash functions -- most of us would consider it reasonable to index an object store with SHA256.
Most of the rest of the paper is seemingly actually solid though. They back up their claims with mathematical hand-waving, and their algorithm actually works on their test inputs. That's an interesting result, and a much stronger one than the collision test.
I can't say it's all that surprising in retrospect (you can imagine, e.g., that to get high accuracy on a prompt like you would need to not have lost information in the hidden states when encoding , so at least up to ~1/2 the max context window you would expect the model to be injective), but despite aligning with other LLM thoughts I've had I think if you had previously asked me to consider invertibility then I would have argued against the authors' position.
[0] They only tested billions of samples. Even considering the birthday paradox, and even if they'd used a much coarser epsilon threshold, they'd still need to run over 2^380 simulations to gain any confidence whatsoever in terms of collision resistance.
Earlier quoted context omitted.
> - This impacts privacy, deletion, and compliance Surely that's a stretch... Typically, the only thing that leaves a transformer is its output text, which cannot be used to recover the input.
remains recoverable... for less than a training run of compute .It's a lot, but it is doable
The paper doesn't claim this to be possible either, they prove the reversibility of the mapping between the input and the hidden state, not the output text. Or rather "near-reversibility", i.e. collisions are technically possible but they have to be very precisely engineered during the model training and don't normally happen.
Earlier quoted context omitted.
Sure, but they went slightly overboard with that headline and they knew it. But oh well, they have a lot of eyes and discussion on their paper so it's a success.
I feel like, if the feedback to your paper is "this is over-done / they claim more than they prove / it's kinda hype-ish" you're going to get less references in future papers. That would seem to be counter to the "impact" goal for research.
I remember hearing an argument once that said LLMs must be capable of learning abstract ideas because the size of their weight model (typically GBs) is so much smaller than the size of their training data (typically TBs or PBs). So either the models are throwing away most of the training data, they are compressing the data beyond the known limits, or they are abstracting the data into more efficient forms. That's why…
- we cannot extract training data from the model using our method
- LLMs are not injective w.r.t. the output text, that function is definitely non-injective and collisions occur all the time
- for the same reasons, LLMs are not invertible from the output text
Maybe not surprising if you logged all internal activity, but it can be done from only a single snapshot of hidden activations from the standard forward pass.
Earlier quoted context omitted.
I envy your intuition about high-dimensional spaces, as I have none (other than "here lies dragons"). (I think your intuition is broadly correct, seeing as billions of collision tests feels quite inadequate given the size of the space.) > Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal. What's the intuition here? Law of large numbers? And how is orthogonality related to…
> > Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal. > What's the intuition here? Law of large numbers? Imagine for simplicity that we consider only vectors pointing parallel/antiparallel to coordinate axes. - In 1D, you have two possibilities: {+e_x, -e_x}. So if you pick two random vectors from this set, the probability of getting something orthogonal is 0. - In 2D, y…
Of course are vectors are not only points in one coordinate axes, but it still isn't that small compared to billions of samples.
>we confirm this result empirically through billions of collision tests on six state-of-the-art language models, and observe no collisions This sounds like a mistake. They used (among others) GPT2, which has pretty big space vectors. They also kind of arbitrarily define a collision threshold as an l2 distance smaller than 10^-6 for two vectors. Since the outputs are normalized, that corresponds to a ridiculously tiny…
The nature of high-dimensional spaces kind of intuitively supports the argument for invertability though, no? In the sense that: > I would expect the chance of two inputs to map to the same output under these constraints to be astronomically small.
An intuitive argument against the claim could be made from the observation that people "jinx" eachother IRL every day, despite reality being vast, if you get what I mean.