Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Semantic clusters and embeddings for 500k Hacker News comments

app.airtrain.ai

Post reply on HN
Print view View original

1–10 of 54 posts

Re: Show HN: Semantic clusters and embeddings for 500k Hacker News comments

#4

Post by josh-sematic »

calebheinzman wrote: post #3

Interesting breakdown of the dataset. Are you guys manually assigning labels to the clusters after the fact or is this using some kind of LLM to create a cluster name?

Everything is automatic and non-manually curated; we basically just uploaded the data without doing anything dataset-specific as far as the clusters are concerned. First we create an embedding of the rows, then we run projection & clustering on it. The first clusters we generate are narrow. After we generate the first round of clusters we label them with an LLM, and then cluster those to create the more generalized clusters. Have an LLM label those, and then we're done.

Re: Show HN: Semantic clusters and embeddings for 500k Hacker News comments

#7

Post by austinjp »

GeoAtreides wrote: post #5

I did not consent on my HN data being used by entities other than HN[1]. Please remove all my comments and data from this dataset. [1] As per HN license: https://www.ycombinator.com/legal/

Yep indeed, same here.

I know "me too" comments aren't generally acceptable here, but I think the wholesale pillaging of the web for LLMs etc is way out of hand.

Re: Show HN: Semantic clusters and embeddings for 500k Hacker News comments

#8

Post by neutralino1 »

politelemon wrote: post #6

I'm not sure how to view the embedding, I clicked on the graph, narrowed down to a comment, but it only shows the row and not the raw array? (Or I've misunderstood)

Oh you cannot actually view the raw embedding vector, only the corresponding row.

Any particular use-case to view the raw embedding?

Re: Show HN: Semantic clusters and embeddings for 500k Hacker News comments

#9

Post by neutralino1 »

GeoAtreides wrote: post #5

I did not consent on my HN data being used by entities other than HN[1]. Please remove all my comments and data from this dataset. [1] As per HN license: https://www.ycombinator.com/legal/

The original dataset is located at [1] (not our HF account). HN data is directly available via the HN API [2]. The privacy policy you point to does not cover HN posts.

[1] https://huggingface.co/datasets/OpenPipe/hacker-news [2] https://github.com/HackerNews/API

Re: Show HN: Semantic clusters and embeddings for 500k Hacker News comments

#10

Post by thenewnewguy »

GeoAtreides wrote: post #5

I did not consent on my HN data being used by entities other than HN[1]. Please remove all my comments and data from this dataset. [1] As per HN license: https://www.ycombinator.com/legal/

You did, although I have no idea if OP is an affiliate of YC (IANAL but maybe you could argue the public API is a form of sublicensing the data?):

> By uploading any User Content you hereby grant and will grant Y Combinator and its affiliated companies a nonexclusive, worldwide, royalty free, fully paid up, transferable, sublicensable, perpetual, irrevocable license to copy, display, upload, perform, distribute, store, modify and otherwise use your User Content for any Y Combinator-related purpose in any form, medium or technology now known or later developed.

Post reply on HN