Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

latentdictionary.com

Post reply on HN
Print view View original

31–40 of 43 posts

Re: Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

#31

Post by kaoD »

tikimcfee wrote: post #15
behnamoh wrote: post #7

Earlier quoted context omitted.

Yeah, it's a fun useless project.

I don’t know about useless. I think there is some real magic waiting to be discovered in mapping language language to some specific and enlightening visualization. I think it involves something like this. Using high statistics and simple, spatial relationships to create a “mapping” of a single individuals wordspace. Imagine walking around the world and seeing everyone’s slightly unique relationship space of words. Th…

What you're describing exists. It's called "embeddings" and it's one of the first steps ChatGPT does to do its magic and it's indeed very useful.

What renders this useless is reducing the dimensionality from thousands to just 3.

Re: Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

#32

Post by smrtinsert »

kaoD wrote: post #6
larodi wrote: post #3

Is this with some sort of dimensionality reduction of the embedding space?

In the bottom left "?" button it says it performs PCA down to 3 dimensions. That's going to lose a ton of information, rendering the space mostly useless.

It is lossy, but doesn't PCA function as a grouping even when forced like this?

Re: Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

#34

Post by tikimcfee »

kaoD wrote: post #31
tikimcfee wrote: post #15

Earlier quoted context omitted.

I don’t know about useless. I think there is some real magic waiting to be discovered in mapping language language to some specific and enlightening visualization. I think it involves something like this. Using high statistics and simple, spatial relationships to create a “mapping” of a single individuals wordspace. Imagine walking around the world and seeing everyone’s slightly unique relationship space of words. Th…

What you're describing exists. It's called "embeddings" and it's one of the first steps ChatGPT does to do its magic and it's indeed very useful. What renders this useless is reducing the dimensionality from thousands to just 3.

Thank you.. sincerely, thank you.

You just unlocked something in my brain about how this works in a certain way and I.. am incredibly excited to explore this more.

If I’m intuitively understanding that, I really really really feel like I can… get this. Do something with this.

Thank you. Now I need to go understand embedding a more. And use them.

If you have any resources or ideas or keywords to look for (I think i expect a long YouTube trek for myself…) I’m happy to read and share my thoughts.

Re: Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

#35

Post by tikimcfee »

patcon wrote: post #22
tudorw wrote: post #17

Earlier quoted context omitted.

I think that's going to be a geodesic in a hyper-dimensional manifold. There was an article here about 'wordlets' on a hyper-sphere and a piece on time and LLM and the relating manifold. Visualising LLM topology (multi-dimensional topological manifolds) is a very rich area for exploration. I'm waiting for someone to use PHATE to do the dimension reduction, it's used in neuroscience to reduce dimensionality providing…

Yep, been thinking on that paper as well: Traveling words. There's code! https://arxiv.org/abs/2309.07315 https://github.com/santiag0m/traveling-words

This is so damn cool! You just showed me something that may change the course of my ideation, current project, and who I may talk to. Thank you so much for sharing.

Words in space. It’s finally freaking happening!!!

Re: Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

#36

Post by tikimcfee »

cuttysnark wrote: post #24

edge of the galaxy: 'if when that then wherever where while for'

I wonder what patterns we could create from the word glyphs that produce meaningful patterns. You said “edge of the galaxy” based on the position of the words. I wonder what else you’d come up with different embeddings and organizations.

Re: Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

#38

Post by tikimcfee »

granawkins wrote: post #37

Hey guys, I'm the bored SOB who built this. Thanks for the awesome discussion, a lot of you know more about this than I do! I hadn't planned to keep building this but if I do, what should I add/change?

Hey there! Sincerely cool stuff, I’m glad it’s fun for you.

It’s actually quite approachable to play with, and some of the comments about, “wut?” may be best answered by a little more experimentation on the user’s side, haha. I think the content itself is tricky, which may trip people up.

Something I’ve seen before that may be interesting is doing something with the definitions of words. ATM, you’re using a source list of words and using the embedded vectors to visualize. But what if you visualized not just the words themselves, but the ordered list of words that make up the definition(s) of that word visible in some spatial relationship. This would look interesting because around (connected to?) each word is its meaning in this case; changing the definition (the context use of the word) would also change the definition… and also change the connected word nodes in the graph. I envision ordered lines and colored words in this style.

If you end up doing something like that, start with like.. a “sentence player”. At the moment you show the words at once. What would it look like to “animate” the appearance of the words and their relationships by definition?

Anyway. Thanks for getting this far, haha. This is a really fascinating project and I’m glad you shared it. Please do tell if any of this is close or far off from something you might be interested in!

Re: Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

#39

Post by bravura »

granawkins wrote: post #37

Hey guys, I'm the bored SOB who built this. Thanks for the awesome discussion, a lot of you know more about this than I do! I hadn't planned to keep building this but if I do, what should I add/change?

T SNE Not PCA. With t SNE parameters that can be configured

Re: Latent Dictionary: 3D map of Oxford3000+search words via DistilBert embeddings

#40

Post by warkdarrior »

pyinstallwoes wrote: post #30
minimaxir wrote: post #29

Earlier quoted context omitted.

> So “dog” vs image of dog would both translate to a primordial signal : identity representation and in the domain of frequency do the comparison and project a coordinate in the spatial sense and eventually those two nodes would more likely be triggered at the same time due to the likelihood of “dog” being next to image of dog when parsing information across future events. That is how CLIP embeddings work and were tr…

Yeah but it doesn’t use a universal method does it? And it requires labeling. The method I’m describing requires no labeling. Labeling would be a local only translation (alias). Labels emerge based on meaning. But the labels are more of an interface - not the actual nodes themselves which arise off the not identity principle * event proximity * comparisons.

Labeling (which is typically manual and thus not scalable) is a proxy for comparisons. Two things are the same if they have the same label. The question is how else to encode the comparison information.
Post reply on HN