41–50 of 171 posts
It was not obvious at first glance to me, but the actual app is here: https://hn.wilsonl.in/
(That's not a dig, I think it's a good idea.)
This is impressive work, especially for a one man show! One thing that stood out to me was the graph of the sentiment analysis over time, I hadn't seen something like that before and it was interesting to see it for Rust. What were the most positive topics over time? And were there topics that saw very sudden drops? I also found this sentence interesting, as it rings true to me about social media "there seems to be a…
The negative sentiment stood out to me mostly because I was expecting a more "clear-cut" sentiment graph: largely neutral-positive, with spikes in the positive direction around positive posts and negative around negative posts. However, for almost all my queries, the sentiment was almost always negative. Even positive posts apparently attracted a lot of negativity (according to the model and my approach, both of which could be wrong). It's something I'd like to dive deeper into, perhaps in a future blog post.
I trained a model to predict if a given post will reach the front page, get flagged etc, I collected over a 1000 RSS feeds and rank the RSS entries with my ranking models.
I submit the high ranking entries on HN to test out my models and I can reach the front page consistently sometimes having multiple entries on the front page at a given time.
I also experiment with user->content recommendation, for that I use comment data for modeling interactions between users and entries, which seems to work fine.
Only problem I have is that I get a lot of 'out of distribution' content in my RSS feeds which causes my ranking models to get 'confused' for this I trained models to predict if a given entry belongs HN or not. On top of that I have some tagging models trained on data I scraped from lobste.rs and hand annotated.
I had been working on this on and off for the last 2 years or so, this account is not my main, and just one I created for testing.
AMA
A long term side project of mine is to try to build a recommendation algorithm trained on HN data. I trained a model to predict if a given post will reach the front page, get flagged etc, I collected over a 1000 RSS feeds and rank the RSS entries with my ranking models. I submit the high ranking entries on HN to test out my models and I can reach the front page consistently sometimes having multiple entries on the fr…
i.e. do HN users upvote more based on the title of the article or on actually reading them?
Good example of data engineering/MLops for people who aren't familiar. I'd suggest using HDBScan to generate hierarchical clusters for the points, then use a model to generate names for interior clusters. That'll make it easy to explore topics out to the leaves, as you can just pop up refinements based on the connectivity to the current node using the summary names. The groups need more distinct coloring, which I thi…
Related a month ago: A Peek inside HN: Analyzing ~40M stories and comments https://news.ycombinator.com/item?id=39910600
Really neat work
edit: Also had no idea HN went back to 2006. https://news.ycombinator.com/item?id=1
edit2: PG wrote this? https://news.ycombinator.com/item?id=487171
A long term side project of mine is to try to build a recommendation algorithm trained on HN data. I trained a model to predict if a given post will reach the front page, get flagged etc, I collected over a 1000 RSS feeds and rank the RSS entries with my ranking models. I submit the high ranking entries on HN to test out my models and I can reach the front page consistently sometimes having multiple entries on the fr…
This gave me a belly laugh.