Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Generalized K-Means Clustering

github.com

Post reply on HN
Print view View original

31–40 of 85 posts

Re: Generalized K-Means Clustering

#31

Post by minimaxir »

mike_ivanov wrote: post #29
minimaxir wrote: post #20

I built a pipeline to automatically cluster and visualize large amounts of text documents in a completely unsupervised manner: - Embed all the text documents. - Project to 2D using UMAP which also creates its own emergent "clusters". - Use k-means clustering with a high cluster count depending on dataset size. - Feed the ChatGPT API ~10 examples from each cluster and ask it to provide a concise label for the cluster.…

Why 2D? (edit: just the vis or there is some other reason?)

Both the viz, and that the 2D UMAP projection is actually enough to get accurately delineated topics.

Hence why I think the typical embedding dimensionality is way way too high.

Re: Generalized K-Means Clustering

#32

Post by dhosek »

staticautomatic wrote: post #3

What are people using k-means for? I can count on one hand the number of times I’ve had a good a priori rationale for the value of k.

I did a modified version of this once for a map of auto dealerships, although rather than working with a fixed k, I used a fix threshold for cluster distance. The algorithm I was working with had O(n³) complexity so to keep the pregeneration of clusters manageable, I partitioned data by state. The other fun part was finding the right metric formula for measuring distances. Because clusters needed to correspond to the rectangular view window on the map, rather than a standard Euclidean distance, I used d = max(Δx,Δy) which gives square neighborhoods rather than round ones.

Re: Generalized K-Means Clustering

#33

Post by calvinmorrison »

staticautomatic wrote: post #3

What are people using k-means for? I can count on one hand the number of times I’ve had a good a priori rationale for the value of k.

We used k means clustering on a project used to track fruit fly memory and learning behaviors

http://git.ceux.org/FlyTracking.git/

Re: Generalized K-Means Clustering

#34

Post by web9ed »

staticautomatic wrote: post #3

What are people using k-means for? I can count on one hand the number of times I’ve had a good a priori rationale for the value of k.

I would recommend checking out DBSCAN as it is similar without having to provide a number k https://en.m.wikipedia.org/wiki/DBSCAN

Re: Generalized K-Means Clustering

#35

Post by sonium »

minimaxir wrote: post #20

I built a pipeline to automatically cluster and visualize large amounts of text documents in a completely unsupervised manner: - Embed all the text documents. - Project to 2D using UMAP which also creates its own emergent "clusters". - Use k-means clustering with a high cluster count depending on dataset size. - Feed the ChatGPT API ~10 examples from each cluster and ask it to provide a concise label for the cluster.…

I did something similar (but not for documents) but I’m struggling with selecting the optimal number of clusters.

Re: Generalized K-Means Clustering

#36

Post by [deleted] »

brianleb wrote: post #9
staticautomatic wrote: post #3

What are people using k-means for? I can count on one hand the number of times I’ve had a good a priori rationale for the value of k.

So after re-reading your comment a few times, I am left with this thought: either you don't understand what k-means clustering is, or I don't understand what k-means clustering is. I wouldn't describe myself as a machine learning expert, but I have taken some grad level classes in statistics, analytics, and methods like this/related to this. So my question is... could you elaborate?

[deleted]

Re: Generalized K-Means Clustering

#37

Post by mr_mitm »

minimaxir wrote: post #20

I built a pipeline to automatically cluster and visualize large amounts of text documents in a completely unsupervised manner: - Embed all the text documents. - Project to 2D using UMAP which also creates its own emergent "clusters". - Use k-means clustering with a high cluster count depending on dataset size. - Feed the ChatGPT API ~10 examples from each cluster and ask it to provide a concise label for the cluster.…

Which libraries are you using, in particular for the first step?

Re: Generalized K-Means Clustering

#38

Post by Konnstann »

staticautomatic wrote: post #3

What are people using k-means for? I can count on one hand the number of times I’ve had a good a priori rationale for the value of k.

I've used something similar for tissue segmentation from hyperspectral images of animals where I know there should be K different tissue types I care about.

Re: Generalized K-Means Clustering

#39

Post by minimaxir »

mr_mitm wrote: post #37
minimaxir wrote: post #20

I built a pipeline to automatically cluster and visualize large amounts of text documents in a completely unsupervised manner: - Embed all the text documents. - Project to 2D using UMAP which also creates its own emergent "clusters". - Use k-means clustering with a high cluster count depending on dataset size. - Feed the ChatGPT API ~10 examples from each cluster and ask it to provide a concise label for the cluster.…

Which libraries are you using, in particular for the first step?

Embeddings is just SentenceTransformers: https://www.sbert.net/

I used the bge-large-en-v1.5 model (https://huggingface.co/BAAI/bge-large-en-v1.5) because I could, but the common all-MiniLM-L6-v2 model is sufficient. The trick is to batch generate the embeddings on a GPU, which SentenceTransformers mostly does by default.

Other libraries are the typical ones (umap for UMAP, scikit-learn for k-means/DBSCAN, chatgpt-python for ChatGPT interfacing, plotly for viz, pandas for some ETL). You don't need to use a bespoke AI/ML package for these workflows and they aren't too complicated.

Re: Generalized K-Means Clustering

#40

Post by zoogeny »

There is a Twitch streamer Tsoding who posted a video of himself implementing K-means clustering in C recently [1]. He also does a follow up 3d visualization of the algorithm in progress using raylib [2].

1. https://www.youtube.com/watch?v=kH-hqG34ylA&t=4788s&ab_chann...

2. https://www.youtube.com/watch?v=K7hWqxC_7Mw&ab_channel=Tsodi...

Post reply on HN