Language models are injective and hence invertible

Post by [deleted] » Wed, Nov 05, 2025, 9:44 AM UTC

gnfargbl wrote: post #58

rowanG077 wrote: post #47

Earlier quoted context omitted.

I think they are saying that there is no proof of being injective. The argument with the hash is essentially saying, doing the same experiment with a hash would yield a similar result, yet hash function are not injective by definition. So from this experimental result you cannot conclude language models are injective. That's not really formally true, there are so called perfect hash functions that are injective over…

Sure, but the paper doesn't claim absolute injectivity. It claims injectivity for practical purposes ("almost surely injective"). That's the same standard to which we hold hash functions -- most of us would consider it reasonable to index an object store with SHA256.

Post by sigmoid10 » Wed, Nov 05, 2025, 1:16 PM UTC

zipy124 wrote: post #119

setopt wrote: post #77

Earlier quoted context omitted.

> > Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal. > What's the intuition here? Law of large numbers? Imagine for simplicity that we consider only vectors pointing parallel/antiparallel to coordinate axes. - In 1D, you have two possibilities: {+e_x, -e_x}. So if you pick two random vectors from this set, the probability of getting something orthogonal is 0. - In 2D, y…

for 768 dimensions, you'd still expect to hit (1-1/N) with a few billion samples though. Like that's a 1/N of 0.13%, which quite frankly isn't that rare at all? Of course are vectors are not only points in one coordinate axes, but it still isn't that small compared to billions of samples.