Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Generalized K-Means Clustering

github.com

Post reply on HN
Print view View original

21–30 of 85 posts

Re: Generalized K-Means Clustering

#21

Post by potatoman22 »

minimaxir wrote: post #20

I built a pipeline to automatically cluster and visualize large amounts of text documents in a completely unsupervised manner: - Embed all the text documents. - Project to 2D using UMAP which also creates its own emergent "clusters". - Use k-means clustering with a high cluster count depending on dataset size. - Feed the ChatGPT API ~10 examples from each cluster and ask it to provide a concise label for the cluster.…

Interesting. What do you use the visualization for? Looking at trends in the documents?

Re: Generalized K-Means Clustering

#22

Post by jncfhnb »

staticautomatic wrote: post #3

What are people using k-means for? I can count on one hand the number of times I’ve had a good a priori rationale for the value of k.

K is 3.

It’s honestly fine for just finding key differences like a principal component for light storytelling. They don’t need to be distinct clusters

Re: Generalized K-Means Clustering

#23

Post by jncfhnb »

minimaxir wrote: post #20

I built a pipeline to automatically cluster and visualize large amounts of text documents in a completely unsupervised manner: - Embed all the text documents. - Project to 2D using UMAP which also creates its own emergent "clusters". - Use k-means clustering with a high cluster count depending on dataset size. - Feed the ChatGPT API ~10 examples from each cluster and ask it to provide a concise label for the cluster.…

Why not just use DBSCAN though

Re: Generalized K-Means Clustering

#24

Post by milofeynman »

I applied to a certain scraping fintech in the Bay Area around 5 years ago and was asked to open the Wikipedia page to k-means squared clustering and implement the algorithm with tests from scratch. I was applying for an android position. I still laugh thinking about how they paid to fly me out and ask such a stupid interview question.

Re: Generalized K-Means Clustering

#25

Post by minimaxir »

potatoman22 wrote: post #21
minimaxir wrote: post #20

I built a pipeline to automatically cluster and visualize large amounts of text documents in a completely unsupervised manner: - Embed all the text documents. - Project to 2D using UMAP which also creates its own emergent "clusters". - Use k-means clustering with a high cluster count depending on dataset size. - Feed the ChatGPT API ~10 examples from each cluster and ask it to provide a concise label for the cluster.…

Interesting. What do you use the visualization for? Looking at trends in the documents?

Let's say you want to look at a large dataset of user-submitted reviews for you app. User reviews are written extremely idiosyncratic so all traditional NLP methods will likely fail.

With the pipeline mentioned, it's much easier to look at cluster density to identify patterns and high-level trends.

Re: Generalized K-Means Clustering

#26

Post by minimaxir »

jncfhnb wrote: post #23
minimaxir wrote: post #20

I built a pipeline to automatically cluster and visualize large amounts of text documents in a completely unsupervised manner: - Embed all the text documents. - Project to 2D using UMAP which also creates its own emergent "clusters". - Use k-means clustering with a high cluster count depending on dataset size. - Feed the ChatGPT API ~10 examples from each cluster and ask it to provide a concise label for the cluster.…

Why not just use DBSCAN though

You can use DBSCAN instead of k-means, but DBSCAN has a worst-case memory complexity of O(n^2) so things can get spicy with large datasets, which is why I opt it to only use it for subclusters. k-means also fixes the number of clusters, which is good for visualization sanity.

https://scikit-learn.org/stable/modules/generated/sklearn.cl...

Re: Generalized K-Means Clustering

#27

Post by esafak »

brianleb wrote: post #9
staticautomatic wrote: post #3

What are people using k-means for? I can count on one hand the number of times I’ve had a good a priori rationale for the value of k.

So after re-reading your comment a few times, I am left with this thought: either you don't understand what k-means clustering is, or I don't understand what k-means clustering is. I wouldn't describe myself as a machine learning expert, but I have taken some grad level classes in statistics, analytics, and methods like this/related to this. So my question is... could you elaborate?

The first problem is picking k. The second problem is the definition of distance or, equivalently, the uniformity of the space. Naively using the Euclidean distance in an embedding where similarity is non-uniform leads to bad outcomes. This problem is solved by learning a uniform embedding, and this is much harder than running k-means.

k-means assumes these hard parts of the problem are taken care of and offers a trivial solution to the rest. Thanks for the help, I'll cluster it myself.

Re: Generalized K-Means Clustering

#28

Post by sshumaker »

minimaxir wrote: post #20

I built a pipeline to automatically cluster and visualize large amounts of text documents in a completely unsupervised manner: - Embed all the text documents. - Project to 2D using UMAP which also creates its own emergent "clusters". - Use k-means clustering with a high cluster count depending on dataset size. - Feed the ChatGPT API ~10 examples from each cluster and ask it to provide a concise label for the cluster.…

You can also look at Bertopic which has this functionality as an open source library:

https://maartengr.github.io/BERTopic/index.html

Re: Generalized K-Means Clustering

#29

Post by mike_ivanov »

minimaxir wrote: post #20

I built a pipeline to automatically cluster and visualize large amounts of text documents in a completely unsupervised manner: - Embed all the text documents. - Project to 2D using UMAP which also creates its own emergent "clusters". - Use k-means clustering with a high cluster count depending on dataset size. - Feed the ChatGPT API ~10 examples from each cluster and ask it to provide a concise label for the cluster.…

Why 2D? (edit: just the vis or there is some other reason?)

Re: Generalized K-Means Clustering

#30

Post by girvo »

refulgentis wrote: post #17
derwiki wrote: post #10

Earlier quoted context omitted.

Used it in college to downscale an X color image to Y number of colors. Sure, Photoshop does it, but it was informative to do it manually.

Google's Material You uses this to initiate color theming (n.b. Celebi's, note usage of Lab / notHSL, respect Cartesian / polar nature of inputs / outputs, and ask for high-K, 128 is what we went with but it's arbitrary. Can get away with as few as 32 if you're ex. Doing brand color from favicon)

Ooh this is a much nicer approach than the kind of brute force approach we took at work for theme gen
Post reply on HN