Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Visualizing GPT-OSS-20B embeddings

melonmars.github.io

Post reply on HN
Print view View original

11–20 of 25 posts

Re: Visualizing GPT-OSS-20B embeddings

#11

Post by minimaxir »

graphviz wrote: post #3

What do people learn from visualizations like this? What is the most important problem anyone has solved this way? Speaking as somewhat of a co-defendant.

Not everything has to be directly informative or solve a problem. Sometimes data visualization can look pretty for pretty's sake.

Dimensionality reduction/clustering like this may be less useful for identifying trends in token embeddings, but for other types of embeddings it's extremely useful.

Re: Visualizing GPT-OSS-20B embeddings

#13

Post by minimaxir »

ashvardanian wrote: post #10

Any good comparisons of traditional embedding models against embeddings derived from autoregressive language models?

They are incomparable. Token embeddings generated with something like word2vec worked well because the networks are shallow and therefore the learned semantic data can be contained solely and independently within the embeddings themselves. Token embeddings as a part of an LLM (e.g. gpt-oss-20b) are conditioned on said LLM and do not have fully independent learned data, although as shown here there still can be some relationships preserved.

Embeddings derived from autoregressive language models apply full attention mechanisms to get something different entirely.

Re: Visualizing GPT-OSS-20B embeddings

#14

Post by lawlessone »

what does it mean that some embeddings are close to others in this space?

That they're related or connected or it arbitrary?

Why does it look like a fried egg?

edit: must be related in some way as one of the "droplets" in the bottom left quadrant seems to consist of various versions of the word "parameter"

Re: Visualizing GPT-OSS-20B embeddings

#15

Post by minimaxir »

lawlessone wrote: post #14

what does it mean that some embeddings are close to others in this space? That they're related or connected or it arbitrary? Why does it look like a fried egg? edit: must be related in some way as one of the "droplets" in the bottom left quadrant seems to consist of various versions of the word "parameter"

Typically these algorithms cluster by similarity (either euclidian or cosine).

The density of the clusters tend to have trends. In this case, the "yolk" has a lot of bizarre unicode tokens.

Re: Visualizing GPT-OSS-20B embeddings

#16

Post by TuringNYC »

graphviz wrote: post #3

What do people learn from visualizations like this? What is the most important problem anyone has solved this way? Speaking as somewhat of a co-defendant.

> What do people learn from visualizations like this?

Applying the embeddings model to some dataset of yours of interest, and then a similar visualization, is where it gets cool because you can visually look at clusters and draw conclusions about the closeness of items in your own dataset

Re: Visualizing GPT-OSS-20B embeddings

#17

Post by esafak »

Without a way to tune it, this visualization is as much about the dimensionality reduction algorithm used as the embeddings themselves, because trade-offs are unavoidable when you go from a very high dimensional space to a 2D one. I would not read too much into it.

Re: Visualizing GPT-OSS-20B embeddings

#19

Post by voodooEntity »

@Author i would recommend you to give

https://github.com/vasturiano/3d-force-graph

a try, for the text labels you can use

https://github.com/vasturiano/three-spritetext

its based on Three.js and creates great 3D graph visualisations GPU rendered (webgl). This could make it alot more interresting to watch because it could display actual depth (your gpu is gonne run hot but i guess worth it)

just a suggestion.

Re: Visualizing GPT-OSS-20B embeddings

#20

Post by mkl »

int_19h wrote: post #18
numpad0 wrote: post #9

Is this handling Unicode correctly? Seems like a lot of even Latin alphabets are getting mangled.

It looks like it's not handling UTF-8 at all and displaying it as if it were Latin-1

I don't think it's actually UTF-8. The data is at https://melonmars.github.io/LatentExplorer/embeddings_2d.jso... and contains things like

  "\u00e0\u00a7\u012d\u00e0\u00a6\u013e"
with some characters > 0xff (but none above 0x0143, weirdly).
Post reply on HN