Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

latentdictionary.com

Post reply on HN
Print view View original

11–20 of 43 posts

Re: Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

#11

Post by tikimcfee »

Edit: I think this is fascinating. If you use words, like dog, electric, life, and human, all of them appear in one mass however, the words like greet, chicken, and “a“ appear in a different mass density section. I think it’s interesting that the words have diverged in location, with some seeming relationship in the way, the words are used. If this were truly random, I would expect those words to be mixed into the other ones.

I have this except you can see every single word in any dictionary at once in space, it renders individual glyphs. It can show an entire dictionary of words - definitions and roots - and let you fly around in them. It’s fun. I built a sample that “plays” a sentence and its definitions. GitHub.com/tikimcfee/LookAtThat The more I see stuff like this, the more i want to complete it. It’s heartening to see so many people fancied with seeing words… I just wish I knew where to find these people to like.. befriend and get better. Im getting the feeling I just kinda exist between worlds of lofty ideas and people that are incredibly smart sticking around other people that are incredibly smart.

Re: Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

#13

Post by tikimcfee »

wrsh07 wrote: post #8

I wish there were more context and maybe the ability to do math on the vectors Eg what is the real distance between the two vectors? That should be easy to compute Similarly: what do I get from summing two vectors and what are some nearby vectors? Maybe just generally: what are some nearby vectors? Without any additional context it's just a point cloud with a couple of randomly labeled elements

If I gave you a live GPU shader that let you arbitrarily position any of say a few million words with simple Cartesian coordinates, what would you do with it? Whole words expressed as Individual letters - not symbols, representations, or abstractions. Just letters arranged in a specific order to form words.

Re: Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

#14

Post by tikimcfee »

tetris11 wrote: post #12

Interesting that "cromulent" and "hentai" seem to map right next to each other, as well as the words "decorate" and "spare".

Similarly, words like “I” and “am” appear in a slightly different dense section of the map by default.

Re: Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

#15

Post by tikimcfee »

behnamoh wrote: post #7
kaoD wrote: post #6

Earlier quoted context omitted.

In the bottom left "?" button it says it performs PCA down to 3 dimensions. That's going to lose a ton of information, rendering the space mostly useless.

Yeah, it's a fun useless project.

I don’t know about useless. I think there is some real magic waiting to be discovered in mapping language language to some specific and enlightening visualization. I think it involves something like this. Using high statistics and simple, spatial relationships to create a “mapping” of a single individuals wordspace.

Imagine walking around the world and seeing everyone’s slightly unique relationship space of words. This is something I have envisioned for a very long time.

Re: Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

#17

Post by tudorw »

smrtinsert wrote: post #16

I would love a quickest path between two words. For example between color and colour

I think that's going to be a geodesic in a hyper-dimensional manifold. There was an article here about 'wordlets' on a hyper-sphere and a piece on time and LLM and the relating manifold. Visualising LLM topology (multi-dimensional topological manifolds) is a very rich area for exploration. I'm waiting for someone to use PHATE to do the dimension reduction, it's used in neuroscience to reduce dimensionality providing information not visible using PCA, t-SNE, LDA or UMAP.

Re: Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

#18

Post by refulgentis »

wrsh07 wrote: post #8

I wish there were more context and maybe the ability to do math on the vectors Eg what is the real distance between the two vectors? That should be easy to compute Similarly: what do I get from summing two vectors and what are some nearby vectors? Maybe just generally: what are some nearby vectors? Without any additional context it's just a point cloud with a couple of randomly labeled elements

You're actually kinda hitting the nail on the head. _Generally_, the word2vec woman + king = queen thing was cute but not very real.

People rarely have to get down to the real true metal on the embeddings models, and they're not what people think they are from their memory of word2vec. Ex. there's actually one vector emitted _per token_, the final vector is the mean. And cosine distance for similarity is the only metric anyone is training for.

In summary, there's ~no reason to think a visualization trying to show multiple vectors will ever be meaningful. Even just starting from "they have way way way more dimensions than we can represent visually" is enough to rule it out

Mini LM v2, foundation of most vector dbs, is 384 dims.

n.b. dear reader, if you've heard of that: you should be using v3! V3 is for asymmetric search, aka query => result docs. V2 is for symmetric search, aka chunk of text => similarly worded chunks of texts. It's very very funny how few people read the docs, in this case, the sentence transformers site.

Re: Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

#19

Post by theaussiestew »

tikimcfee wrote: post #13
wrsh07 wrote: post #8

I wish there were more context and maybe the ability to do math on the vectors Eg what is the real distance between the two vectors? That should be easy to compute Similarly: what do I get from summing two vectors and what are some nearby vectors? Maybe just generally: what are some nearby vectors? Without any additional context it's just a point cloud with a couple of randomly labeled elements

If I gave you a live GPU shader that let you arbitrarily position any of say a few million words with simple Cartesian coordinates, what would you do with it? Whole words expressed as Individual letters - not symbols, representations, or abstractions. Just letters arranged in a specific order to form words.

I would want to have multi lingual embeddings so I can learn languages more efficiently. Being able to see clouds of different words in different languages would allow me to contextualize them more easily. Same for phrases and sentences.

Re: Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

#20

Post by tikimcfee »

theaussiestew wrote: post #19
tikimcfee wrote: post #13

Earlier quoted context omitted.

If I gave you a live GPU shader that let you arbitrarily position any of say a few million words with simple Cartesian coordinates, what would you do with it? Whole words expressed as Individual letters - not symbols, representations, or abstractions. Just letters arranged in a specific order to form words.

I would want to have multi lingual embeddings so I can learn languages more efficiently. Being able to see clouds of different words in different languages would allow me to contextualize them more easily. Same for phrases and sentences.

I’m very much with you. Since most languages I know of are written with combinations of ordered glyphs, they all get rendered the same (although I don’t handle about 20k current Unicode characters of the full 400k+, and none of the grouping really works, so RTL languages would be a mess for individual words).

However, this is exactly where I want to go. A dictionary is a cyclic graph of words mapping to words. That means there’s at least one finite way to visit every single node and give it a position, with a direct relationship to the words that define it, and those words that define them, and so on.

This creates an arbitrary and unique geometric structure per language, and if you get fancy and create modifiers for an individual’s vocabulary, you can even create transforms for a “base” dictionary, and the way someone chooses to use certain words differently. You would be able to see, but likely not understand, the “structure” of types of text - poetry, storytelling, instructional writing, etc.

Post reply on HN