Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Semantic clusters and embeddings for 500k Hacker News comments

app.airtrain.ai

Post reply on HN
Print view View original

31–40 of 54 posts

Re: Show HN: Semantic clusters and embeddings for 500k Hacker News comments

#31

Post by shadowgovt »

austinjp wrote: post #7
GeoAtreides wrote: post #5

I did not consent on my HN data being used by entities other than HN[1]. Please remove all my comments and data from this dataset. [1] As per HN license: https://www.ycombinator.com/legal/

Yep indeed, same here. I know "me too" comments aren't generally acceptable here, but I think the wholesale pillaging of the web for LLMs etc is way out of hand.

Back in the day, we called that "indexing" and it was fundamental to making the web in any way usable; without search engines, the whole thing was data with no ability to locate it.

I don't know precisely what changed that people decided that analysis is a bridge too far.

Re: Show HN: Semantic clusters and embeddings for 500k Hacker News comments

#32

Post by shadowgovt »

olivierduval wrote: post #25
GeoAtreides wrote: post #5

I did not consent on my HN data being used by entities other than HN[1]. Please remove all my comments and data from this dataset. [1] As per HN license: https://www.ycombinator.com/legal/

Actually, another problem might be GDPR: I have found my username... and that is a clearly a PII because it's directly and univocally bound to me I dont really care (for now) about this... but on the principle, I'm a bit fed up too by companies just crawling anything to train anymodel without any care about the datas, the people that produced them, and the consequences on people's life. Maybe I could use the new Euro…

I look forward to the not-too-distant future where the EU protections grow stronger and places like HN have to respond by banning all European users lest they run afoul of a draconic legal framework.

It'll kill a lot of experiments (Mastodon immediately comes to mind; can't be pulling comments from other people's servers if those comments are attached to personal data like the commenter's username, right?).

Re: Show HN: Semantic clusters and embeddings for 500k Hacker News comments

#33

Post by Alifatisk »

politelemon wrote: post #6

I'm not sure how to view the embedding, I clicked on the graph, narrowed down to a comment, but it only shows the row and not the raw array? (Or I've misunderstood)

It would be very cool if one could view the whole 3D embedding space

Re: Show HN: Semantic clusters and embeddings for 500k Hacker News comments

#34

Post by olivierduval »

shadowgovt wrote: post #32
olivierduval wrote: post #25

Earlier quoted context omitted.

Actually, another problem might be GDPR: I have found my username... and that is a clearly a PII because it's directly and univocally bound to me I dont really care (for now) about this... but on the principle, I'm a bit fed up too by companies just crawling anything to train anymodel without any care about the datas, the people that produced them, and the consequences on people's life. Maybe I could use the new Euro…

I look forward to the not-too-distant future where the EU protections grow stronger and places like HN have to respond by banning all European users lest they run afoul of a draconic legal framework. It'll kill a lot of experiments (Mastodon immediately comes to mind; can't be pulling comments from other people's servers if those comments are attached to personal data like the commenter's username, right?).

Well, maybe you should think about the real responsabilities: Europe make laws in reaction to ABUSES. So dont blame Europe for the legislation, but the abusers that made this legislation mandatory to defend european citizen ;-)

Actually, Europe is so slow that a lot of experiments may take place. And there wont be any legislation if there no abuse...

It took a loooonnnng time for Europe to react to Facebook, Google & co abuse with users datas. Same for OpenAI using a awful lot of copyrighted material without giving anything back... So thank'em for Europe legislation :-)

Re: Show HN: Semantic clusters and embeddings for 500k Hacker News comments

#35

Post by neutralino1 »

Alifatisk wrote: post #33
politelemon wrote: post #6

I'm not sure how to view the embedding, I clicked on the graph, narrowed down to a comment, but it only shows the row and not the raw array? (Or I've misunderstood)

It would be very cool if one could view the whole 3D embedding space

We debated doing 2D vs 3D and 3D brought a bunch of usability issues. We also noticed most SOTA embedding visualizations were 2D and already yielded good insights.

Re: Show HN: Semantic clusters and embeddings for 500k Hacker News comments

#36

Post by GeoAtreides »

shadowgovt wrote: post #30
maleldil wrote: post #21

Earlier quoted context omitted.

From your link: > Hacker News Information: If you create a Hacker News account (ID and profile), we do not collect any Personal Information unless you choose to provide your email address and/or information in the "about" field (“HN Information”). Your submissions to, and comments you make on, the Hacker News site are not Personal Information and are not "HN Information" as defined in this Privacy Policy.

Correct. I'm a little surprised people don't know the ownership story on HN. Didn't it raise questions when they realized they can't delete their posts without mother-may-I'ing the mods? HN is pretty up-front that when you post here you are providing them content for free to more-or-less consume as they please.

I disagree on the up-front... front. Neither of the things you mentioned are made clear when signing up for an account.

Re: Show HN: Semantic clusters and embeddings for 500k Hacker News comments

#37

Post by GeoAtreides »

senordevnyc wrote: post #28
GeoAtreides wrote: post #24

Earlier quoted context omitted.

Welp, there it is then. If they are YC affiliated, nothing I can do. But I do feel saddened and personally betrayed; I thought the licence I gave to YN was just for news.ycombinator.com to store and show my comments, not for any other purposes. Silly, silly me.

All HN did do was store and show your (public) comments. What this other company did with that (public) data seems to me to be a separate issue that you should take up with that company, just like the fact that your public comments (which you explicitly gave permission to HN to show) have been indexed by Google, Bing, and probably thousands of other spiders, bots, scrapers, etc. I'm curious how you expected this to w…

Yes, I expect it to work in the same way instagram works, for example. If a commercial entity started yoinking photos from instagram and using them for commercial purposes, shit will hit the fan.

Again, the fact my user data has been scrapped already doesn't mean it was scrapped legally. I'm ok with HN showing my comments, I'm not ok with anyone else than HN using my user data.

Re: Show HN: Semantic clusters and embeddings for 500k Hacker News comments

#38

Post by averageRoyalty »

GeoAtreides wrote: post #37
senordevnyc wrote: post #28

Earlier quoted context omitted.

All HN did do was store and show your (public) comments. What this other company did with that (public) data seems to me to be a separate issue that you should take up with that company, just like the fact that your public comments (which you explicitly gave permission to HN to show) have been indexed by Google, Bing, and probably thousands of other spiders, bots, scrapers, etc. I'm curious how you expected this to w…

Yes, I expect it to work in the same way instagram works, for example. If a commercial entity started yoinking photos from instagram and using them for commercial purposes, shit will hit the fan. Again, the fact my user data has been scrapped already doesn't mean it was scrapped legally. I'm ok with HN showing my comments, I'm not ok with anyone else than HN using my user data.

> ommercial entity started yoinking photos from instagram and using them for commercial purposes, shit will hit the fan.

Will it though? I would imagine Meta would block them and then posture with a C&D or a frivolous lawsuit, but if they share the phones you gave them on the public internet, they're publicly consumable right? What law do you feel is broken there?

Re: Show HN: Semantic clusters and embeddings for 500k Hacker News comments

#39

Post by averageRoyalty »

olivierduval wrote: post #25
GeoAtreides wrote: post #5

I did not consent on my HN data being used by entities other than HN[1]. Please remove all my comments and data from this dataset. [1] As per HN license: https://www.ycombinator.com/legal/

Actually, another problem might be GDPR: I have found my username... and that is a clearly a PII because it's directly and univocally bound to me I dont really care (for now) about this... but on the principle, I'm a bit fed up too by companies just crawling anything to train anymodel without any care about the datas, the people that produced them, and the consequences on people's life. Maybe I could use the new Euro…

I'm not up on the nuance of the GDPR, but has it been tested that your public profile name - which you set knowing it will be displayed publicly - is PII?

I'd be very surprised if that were the case.

Re: Show HN: Semantic clusters and embeddings for 500k Hacker News comments

#40

Post by shadowgovt »

GeoAtreides wrote: post #37
senordevnyc wrote: post #28

Earlier quoted context omitted.

All HN did do was store and show your (public) comments. What this other company did with that (public) data seems to me to be a separate issue that you should take up with that company, just like the fact that your public comments (which you explicitly gave permission to HN to show) have been indexed by Google, Bing, and probably thousands of other spiders, bots, scrapers, etc. I'm curious how you expected this to w…

Yes, I expect it to work in the same way instagram works, for example. If a commercial entity started yoinking photos from instagram and using them for commercial purposes, shit will hit the fan. Again, the fact my user data has been scrapped already doesn't mean it was scrapped legally. I'm ok with HN showing my comments, I'm not ok with anyone else than HN using my user data.

Shit hits the fan because Instagram considers user content a golden goose, and they have a vested interest in not letting it get outside their control. Not because they feel a particular obligation to protect user privacy. That's generally been status quo for every social network.

HN cares a lot less; they're a tech comment site and don't actively discourage people using the dataset gleanable from the contents of the site for novel experimentation.

(Sidebar: I see "scrapped" coming up a lot in these conversations these days. Where is that neologism coming from? I'm familiar with people calling it "scraping" but it seems like the term has drifted for some reason?)

Post reply on HN