Would be cool to see member similarity. Finding like-minded commentors/posters may help discover content that would be of interest.
Accidental dating app.
Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun
21–30 of 171 posts
Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun
#22Really love the island map! But the automatic zooming on the map doesn't seem very relevant. E.g. try typing "openai" - I can't see anything related to that query in that part of the map
Thanks! Yeah sometimes there are one or two "far" away results which make the auto zoom seem strange. It's something I'd like to tune, perhaps zooming to where most but not all results are.
Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun
#23This is a surprisingly big endeavour for what looks like an exploratory hobby project. Not to minimize the achievement, very cool, I'm just surprised by how much was invested into it. They used 150 GPUs and developed two custom systems (db-rpc and queued) for inter-server communication, and this was just to compute the embeddings, there's a lot of other work and computation surrounding it. I'm curious about the conte…
(1) Definitely you could use a cheaper embedding and still get pretty good results
(2) I apply classical ML (say probability calibrated SVM) to embeddings like that and get good results for classification and clustering at speeds over 100x fine-tuning an LLM.
Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun
#24As a novice, is there a benefit to using custom Node as the downloader? When I did my download of the 40 million Hacker News api items I used "curl --parallel".
What I would like to figure out is the easiest way to go from the API straight into a parquet file.
Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun
#25This is a surprisingly big endeavour for what looks like an exploratory hobby project. Not to minimize the achievement, very cool, I'm just surprised by how much was invested into it. They used 150 GPUs and developed two custom systems (db-rpc and queued) for inter-server communication, and this was just to compute the embeddings, there's a lot of other work and computation surrounding it. I'm curious about the conte…
The author is definitely very skilled. I find it interesting they submit posts on HN but haven’t commented since 2018! And then embarked on this project. As far as funding/time, one possibility is they are between endeavors/employment and it’s self funded as they have had a successful career or business financially. They were very efficient at the GPU utilization so it probably didn’t cost that much.
Thanks! Haha yeah I'm trying to get into the habit of writing about and sharing the random projects I do more often. And yeah the cost was surprisingly low (in the hundreds of dollars), so it was pretty accessible as a hobby project.
Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun
#26Related a month ago:
A Peek inside HN: Analyzing ~40M stories and comments
Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun
#27This is a surprisingly big endeavour for what looks like an exploratory hobby project. Not to minimize the achievement, very cool, I'm just surprised by how much was invested into it. They used 150 GPUs and developed two custom systems (db-rpc and queued) for inter-server communication, and this was just to compute the embeddings, there's a lot of other work and computation surrounding it. I'm curious about the conte…
Hey, thanks for the kind words. I wasn't able to mention the costs in the post (might follow up in the future) but it was in the hundreds of dollars, so was reasonably accessible as a hobby project. The GPUs were surprisingly cheap, and was only scaled up mostly because I was impatient :) --- the entire cluster only ran for a few hours. Do you have any links to your work? They sound interesting and I'd like to read m…
"Hundreds of dollars" sounds a bit painful as an EU engineer and entrepreneur :), but I guess it's all relative. We would think twice about investing this much manpower and compute for such an exploratory project even in a commercial setting if it was not directly funded by a client.
But your technical skill is obvious and very impressive.
If you want to read more, my old bachelor's thesis is somewhat related, from when we only had word embeddings and document embeddings were quite experimental still: https://ad-publications.cs.uni-freiburg.de/theses/Bachelor_J...
I've done a lot follow-up work in my startup Scitodate, which includes large-scale graph and embedding analysis, but we haven't published most of it for now.
Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun
#28Awesome visualisation, and great write-up.
On mobile (in portrait), a lot of longer titles get culled as their origin scrolls off, with half of it still off the other side of the screen - wonder if it'd be worth keeping on rendering them until the entire text field is off screen (especially since you've already got a bounding box for them).
I stumbled upon [1] using it that reflects your comments on comment sentiment.
This also reminded me of [2] (for which the site itself had rotted away, incidentally) - analysing HN users' similarity by writing style.
[1] https://minimaxir.com/2014/10/hn-comments-about-comments/ [2] https://news.ycombinator.com/item?id=33755016
Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun
#29Adding a subscribe feature to get an email with the most recent posts in a topic/community would be really cool. One of my favorite parts of HN is the weekly digest I get in my inbox; it would be awesome if that were tailored to me. What you've built is really impressive. I'm excited to see where this goes!
Thanks! Yeah if there's enough interested users I'd love to turn this into a live service. Would an email subscription to a set of communities you pick be something you'd be interested in?
Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun
#30Earlier quoted context omitted.
Hey, thanks for the kind words. I wasn't able to mention the costs in the post (might follow up in the future) but it was in the hundreds of dollars, so was reasonably accessible as a hobby project. The GPUs were surprisingly cheap, and was only scaled up mostly because I was impatient :) --- the entire cluster only ran for a few hours. Do you have any links to your work? They sound interesting and I'd like to read m…
"Hundreds of dollars" sounds a bit painful as an EU engineer and entrepreneur :), but I guess it's all relative. We would think twice about investing this much manpower and compute for such an exploratory project even in a commercial setting if it was not directly funded by a client. But your technical skill is obvious and very impressive. If you want to read more, my old bachelor's thesis is somewhat related, from w…
Thanks for sharing, I'll have a read, looks very relevant and interesting!