Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Language models are injective and hence invertible

arxiv.org

Post reply on HN
Print view View original

151–160 of 164 posts

Re: Language models are injective and hence invertible

#151

Post by tonetheman »

setopt wrote: post #77
vitus wrote: post #60

Earlier quoted context omitted.

I envy your intuition about high-dimensional spaces, as I have none (other than "here lies dragons"). (I think your intuition is broadly correct, seeing as billions of collision tests feels quite inadequate given the size of the space.) > Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal. What's the intuition here? Law of large numbers? And how is orthogonality related to…

> > Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal. > What's the intuition here? Law of large numbers? Imagine for simplicity that we consider only vectors pointing parallel/antiparallel to coordinate axes. - In 1D, you have two possibilities: {+e_x, -e_x}. So if you pick two random vectors from this set, the probability of getting something orthogonal is 0. - In 2D, y…

[dead]

Re: Language models are injective and hence invertible

#152

Post by gnfargbl »

boerseth wrote: post #120
gnfargbl wrote: post #25

Earlier quoted context omitted.

The nature of high-dimensional spaces kind of intuitively supports the argument for invertability though, no? In the sense that: > I would expect the chance of two inputs to map to the same output under these constraints to be astronomically small.

I don't think that intuition is entirely trustworthy here. The entire space is high-dimensional, true, but the structure of the subspace encompassing linguistically sensible sequences of tokens will necessarily be restricted and have some sort of structure. And within such subspaces there may occur some sort of sink or attractor. Proving that those don't exist in general seems highly nontrivial to me. An intuitive ar…

I do get what you're saying, and it sounds almost analogous to visualisations of bad PRNGs, e.g. https://www.reddit.com/r/dataisbeautiful/comments/gv4fhr/oc_...

Re: Language models are injective and hence invertible

#153

Post by kouru225 »

Wait so is it possible to pass a message using AI and does this matter?

Like let’s imagine I have an AI model and only me and my friend have it. I write a prompt and get back the vectors only. No actual output. Then I send my friend those vectors and they use this algorithm to reconstruct my message at the endpoint. Does this method of messaging protect against a MITM attack? Can this be used in cryptography?

Re: Language models are injective and hence invertible

#154

Post by sebastianmestre »

sigmoid10 wrote: post #17
sebastianmestre wrote: post #13

Earlier quoted context omitted.

I think that the latent space that GPT-2 uses has 768 dimensions (i.e. embedding vectors have that many components).

It doesn't really matter which vector you are looking at, since they are using a tiny constraint in a high dimensional continuous space. There's gotta be an unfathomable amount of vectors you can fit in there. Certainly more than a few billion.

No, yeah, totally. Even assuming binary vectors 2^768 is a ridiculously huge number. The probability of collision even assuming a bad sampling that discards 75% of dimensions is still vanishingly small.

Re: Language models are injective and hence invertible

#155

Post by qarl »

discreteevent wrote: post #124
qarl wrote: post #82

Earlier quoted context omitted.

> they are compressing the data beyond the known limits, or they are abstracting the data into more efficient forms. I would argue that this is two ways of saying the same thing. Compression is literally equivalent to understanding.

If we use gzip to compress a calculus textbook does that mean that gzip understands calculus?

To a small degree, yes. GZIP knows that some patterns are more common in text than others - that understanding allows it to compress the data.

But that's a poor example of what I'm trying to convey. Instead consider plotting the course of celestial bodies. If you don't understand, you must record all the individual positions. But if you do, say, understand gravity, a whole new level of compression is possible.

Re: Language models are injective and hence invertible

#156

Post by Nydhal »

orbital-decay wrote: post #115
Kostchei wrote: post #112

Earlier quoted context omitted.

remains recoverable... for less than a training run of compute .It's a lot, but it is doable

Here's an output text: "Yes." Recover the exact input that led to it. (you can't, because the hidden state is already irreversibly collapsed during the sampling of each token) The paper doesn't claim this to be possible either, they prove the reversibility of the mapping between the input and the hidden state, not the output text. Or rather "near-reversibility", i.e. collisions are technically possible but they have…

if you generate a lot of output text you can approximate the hidden state.

Re: Language models are injective and hence invertible

#157

Post by [deleted] »

jdhwosnhw wrote: post #149
anvuong wrote: post #146

Earlier quoted context omitted.

This has nothing to do with the birthday paradox. That paradox presumes a small countable state space (365) and a large enough # of observations. In this case, it's a mathematical fact that 2 random vector in high dimensional space is very likely to be near orthogonal.

A slightly stronger (and more relevant) statement is that the number of mutually nearly orthogonal vectors you can simultaneously pack into an N dimensional space is exponential in N. Here “mutually nearly orthogonal” can be formally defined as: choose some threshold epsilon>0 - the set S of unit vectors is nearly mutually orthogonal if the maximum of the pairwise dot products of between all members if S is less than…

[deleted]

Re: Language models are injective and hence invertible

#158

Post by Lord-Jobo »

weinzierl wrote: post #109
fxwin wrote: post #3

I don't like the title of this paper, since most people in this space probably think of language models not as producing a distribution (wrt which they are indeed invertible, which is what the paper claims) but as producing tokens (wrt which they are not invertible [0]). Also the author contribution statement made me laugh. [0] https://x.com/GladiaLab/status/1983812121713418606

Spoiler but for those who do not want to open the paper, the contribution statement is: "Equal contribution; author order settled via Mario Kart." If only more conflicts in life would be settled via Mario Kart.

Envisioning our most-elderly world leaders throwing down in Mario Kart, fighting for 23rd and 24th place as they bump into walls over and over and struggle to hold their controllers properly… well it’s a very pleasant thought.

Re: Language models are injective and hence invertible

#159

Post by krackers »

skybrian wrote: post #134

I'm wondering how this might be summarized in simple terms? It sounds like, after processing some text, the entire prompt is included in the in-memory internal state of the program that's doing inference. But it seems like it would need to remember the prompt to answer questions about it. How does this interact with the attention mechanism?

I wonder why this is such a surprise, this is in fact what you would naively expect given the way the residual stream is structured no?

Each attention block adds to the residual stream. And we already know from logit-lens type work that the residual stream roughly remains in the same "basis" [1], which I vaguely remember is something that resnet architectures explicitly try to achieve.

So maybe it's my armchair naivety but in order for both of these to hold while the LLM being able to do some sort of "abstraction", it seems like it is natural for the initial token embedding to be projected into some high-dimensional subspace and then as it passes through different attention blocks you get added "deltas" on top of that, filling out other parts of that subspace.

And looking at the overall attention network from an information passing perspective, encoding and having access to the input tokens is certainly a nice thing to have.

Now maybe it could be argued that the original input could be lossily "converted" into some other more abstract representation, but if the latent space is large enough to not force this, then there's probably no strict reason to do so. And in fact we do know that traditional BPE token embeddings don't even form a subspace (there's a fixed vocab size and embeddings are just a lookup table, so it's only just a bunch of scattered points).

I wonder if this work is repeated with something like vision tokens, whether you will get the same results.

[1] https://nitter.poast.org/khoomeik/status/1920620258353610896

Re: Language models are injective and hence invertible

#160

Post by [deleted] »

gnfargbl wrote: post #79
gowld wrote: post #75

Earlier quoted context omitted.

The problem with "almost surely injective" for "practical purposes". Is that when you try to invert something, how do you know the result you get is one of those "practical purposes" ? We aren't just trying to claim that two inputs are the same, as in hashing. We are trying to recover lost inputs.

You don't, I guess. But again that's just the same as when you insert something into an object store: you can't be absolutely certain that a future retrieval will give you the same object and not a colliding blob. It's just good enough for all practical purposes.

[deleted]
Post reply on HN