Harnessing the Universal Geometry of Embeddings
1–10 of 50 posts
Re: Harnessing the Universal Geometry of Embeddings
#2What is the (co)homology of this space?
Re: Harnessing the Universal Geometry of Embeddings
#3Dupe: https://news.ycombinator.com/item?id=44054425
Note this is version 4 of the paper and the original post was version 1 (I think?)
OpenReview (for NeurIPS) for the curious: https://openreview.net/forum?id=jiCLUPq5xv
Re: Harnessing the Universal Geometry of Embeddings
#4Let's assume that monotonocity of pair-wise distances are preserved.
Without knowing the details of how the paper solved the problem, my first attempt would be to find the diametrically distant pair of points in the two different embeddings and assume that the pair is the same pair. Then find the next distant pairs and so on.
After sufficiently many such pairs have been found, or better still, the largest d-simplex is found, find that scaled rigid body transformation that makes the corresponding pairs coincide. Proceeding this way ought to be less work than solving a generic graph isomorphism problem.
Re: Harnessing the Universal Geometry of Embeddings
#5I am not familiar with the standards of publishing in machine learning, but as someone trained in a mathematics background, this paper seems relatively light on details and heavy on exposition. Is that typical? Is this a really novel idea? Not trying to be snarky, just trying to understand how meaningful this is.
Re: Harnessing the Universal Geometry of Embeddings
#6Cyberphrenology. In any two random graphs, you'll find an isomorphic graph which is can be up to log of the size of the graphs.
And if the LLM has been trained up to the limit of what data it can hold, it is going to be random. Proof below if it isn't obvious.
The entire effort of all people who are trying to understand how LLMs work, how they represent their data, its all bound to fail.
Proof: a LLM is a very good approximation of the Solomonov/Levin/Kolmogorov universal probability function on tokens. As such, it will be random--pure white noise--because if you found any patterns in there, you could exploit the regularity and come up with a smaller set of weights for the same LLM.
There are no patterns there to be found. They have all been factored out by training the neural net until it couldn't learn any more.
Re: Harnessing the Universal Geometry of Embeddings
#7I am not familiar with the standards of publishing in machine learning, but as someone trained in a mathematics background, this paper seems relatively light on details and heavy on exposition. Is that typical? Is this a really novel idea? Not trying to be snarky, just trying to understand how meaningful this is.
You are not wrong. But this has by no means proven its up to the standard of being publishable in a machine learning journal. Its on arXiv.org, which, lets face it, at the end of the day is a vanity press.
Re: Harnessing the Universal Geometry of Embeddings
#8[flagged]
Re: Harnessing the Universal Geometry of Embeddings
#9I am not familiar with the standards of publishing in machine learning, but as someone trained in a mathematics background, this paper seems relatively light on details and heavy on exposition. Is that typical? Is this a really novel idea? Not trying to be snarky, just trying to understand how meaningful this is.
You are not wrong. But this has by no means proven its up to the standard of being publishable in a machine learning journal. Its on arXiv.org, which, lets face it, at the end of the day is a vanity press.
At a minimum posting to arxiv gives others a standard way to cite the work.
Re: Harnessing the Universal Geometry of Embeddings
#10I am not familiar with the standards of publishing in machine learning, but as someone trained in a mathematics background, this paper seems relatively light on details and heavy on exposition. Is that typical? Is this a really novel idea? Not trying to be snarky, just trying to understand how meaningful this is.
You are not wrong. But this has by no means proven its up to the standard of being publishable in a machine learning journal. Its on arXiv.org, which, lets face it, at the end of the day is a vanity press.
The pace of things is moving along so rapidly right now, I’m not sure that waiting for peer reviews is always a wise move. Doubly so if there’s a paywall; why limit your article’s impact by placing it where practitioners’ agents might not be able to access it? The rapid progress right now is challenging for conventional academic processes.
If the value of the paper is difficult to independently verify, for example, if it depends on the credibility of the author, then the academic ritual can add something. If it’s a mathematical result, one that can be automatically verified, or a machine learning technique that anyone can try with Claude code reconstructing it for them, this sort of pre-print publishing model is advantageous.