11–20 of 171 posts
What you've built is really impressive. I'm excited to see where this goes!
They used 150 GPUs and developed two custom systems (db-rpc and queued) for inter-server communication, and this was just to compute the embeddings, there's a lot of other work and computation surrounding it.
I'm curious about the context of the project, and how someone gets this kind of funding and time for such research.
PS: Having done a lot of similar work professionally (mapping academic paper and patent landscapes), I'm not sure if 150 GPUs were really needed. If you end up just projecting to 2D and clustering, I think that traditional methods like bag-of-words and/or topic modelling would be much easier and cheaper, and the difference in quality would be unnoticeable. You can also use author and comment-thread graphs for similar results.
Would be cool to see member similarity. Finding like-minded commentors/posters may help discover content that would be of interest.
Really love the island map! But the automatic zooming on the map doesn't seem very relevant. E.g. try typing "openai" - I can't see anything related to that query in that part of the map
It would be cool to have analogous continents, countries, sub-regions, roads, different-sized settlements, and significant landmarks... This version looks great at the highest zoom level, but rapidly becomes hard to interpret as you zoom in, same as most similar large embedding or graph visualizations.
This is a surprisingly big endeavour for what looks like an exploratory hobby project. Not to minimize the achievement, very cool, I'm just surprised by how much was invested into it. They used 150 GPUs and developed two custom systems (db-rpc and queued) for inter-server communication, and this was just to compute the embeddings, there's a lot of other work and computation surrounding it. I'm curious about the conte…
As far as funding/time, one possibility is they are between endeavors/employment and it’s self funded as they have had a successful career or business financially. They were very efficient at the GPU utilization so it probably didn’t cost that much.
Very nice. Since Hn data spawns so many such fun projects, there should be a monthly or weekly updates zip file or torrent with this data, which hackers can just download instead of writing a scraper and starting from scratch all the time.
Feature request : Is it possible to show in the graph how famous the topic / sub topic / article is ?
So that we can do an educated exploration in the graph around what was upvoted and what was not ?
This is a surprisingly big endeavour for what looks like an exploratory hobby project. Not to minimize the achievement, very cool, I'm just surprised by how much was invested into it. They used 150 GPUs and developed two custom systems (db-rpc and queued) for inter-server communication, and this was just to compute the embeddings, there's a lot of other work and computation surrounding it. I'm curious about the conte…
This is a surprisingly big endeavour for what looks like an exploratory hobby project. Not to minimize the achievement, very cool, I'm just surprised by how much was invested into it. They used 150 GPUs and developed two custom systems (db-rpc and queued) for inter-server communication, and this was just to compute the embeddings, there's a lot of other work and computation surrounding it. I'm curious about the conte…
Do you have any links to your work? They sound interesting and I'd like to read more about them.
Obviously the scale of OP's project adds a lot of interesting complexity, this tool cannot handle that, but it's great for medium-sized datasets.