Visualizing GPT-OSS-20B embeddings
melonmars.github.io
Visualizing GPT-OSS-20B embeddings
1–10 of 25 posts
Re: Visualizing GPT-OSS-20B embeddings
#2It's an interesting looking plot I suppose.
My guess is its the 2 largest principle components of the embedding.
But none of the points are labelled? There isn't a writeup on the page or anything?
Re: Visualizing GPT-OSS-20B embeddings
#3What do people learn from visualizations like this?
What is the most important problem anyone has solved this way?
Speaking as somewhat of a co-defendant.
Re: Visualizing GPT-OSS-20B embeddings
#4It's an interesting looking plot I suppose. My guess is its the 2 largest principle components of the embedding. But none of the points are labelled? There isn't a writeup on the page or anything?
Usually PCA doesn't look quite like this so this is likely done using TSNE or UMAP, which are non parametric embeddings (they optimize a loss by modifying the embedded points directly). I can see labels if I mouseover the dots.
Re: Visualizing GPT-OSS-20B embeddings
#5What do people learn from visualizations like this? What is the most important problem anyone has solved this way? Speaking as somewhat of a co-defendant.
I lets you inspect what actually constitutes a given cluster, for example it seems like the outer clusters are variations of individual words and their direct translations, rather than synonyms (the ones I saw at least).
Re: Visualizing GPT-OSS-20B embeddings
#6I have the suspicion that this is how GPT-OSS-20B would generate a visualization of it's embeddings. Happy to learn otherwise.
Re: Visualizing GPT-OSS-20B embeddings
#7Cool ! Would it possible to generate visualizations of any given open weight model out there ?
Re: Visualizing GPT-OSS-20B embeddings
#8It's an interesting looking plot I suppose. My guess is its the 2 largest principle components of the embedding. But none of the points are labelled? There isn't a writeup on the page or anything?
I can see the labels when I hover with the pointer
Re: Visualizing GPT-OSS-20B embeddings
#9Is this handling Unicode correctly? Seems like a lot of even Latin alphabets are getting mangled.
Re: Visualizing GPT-OSS-20B embeddings
#10Any good comparisons of traditional embedding models against embeddings derived from autoregressive language models?