Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tagger News takes a subset of HN articles, analyzes using ML, and applies tags

techcrunch.com

Post reply on HN
Print view View original

11–20 of 70 posts

Re: Tagger News takes a subset of HN articles, analyzes using ML, and applies tags

#11

Post by minimaxir »

Direct URL to project details: https://devpost.com/software/tagger-news

A few comments:

1. To other commenters, as with the HN Vue demo a week ago (https://news.ycombinator.com/item?id=14284877), the project is a technical proof-of-concept; the aesthetics aren't the primary focus.

2. The Algolia API is better for scraping because it allows for bulk requests, unlike the official API (my old 2014 script still works I think: https://github.com/minimaxir/get-all-hacker-news-submissions...)

3) How much time did it take to manually label the training/test set before training the RF classifier? Even with topic modeling for extrapolating tags, accurate labeling for 20,000 submissions is a task.

Re: Tagger News takes a subset of HN articles, analyzes using ML, and applies tags

#13

Post by gnicholas »

CaliforniaKarl wrote: post #4
nickpsecurity wrote: post #2

Strength of Hacker News is the network effect of a diverse, intelligent crowd. It would be hard to replace. A supplemental site tagging it and aiding search has value. Biggest problem I have searching HN, though, is Google mixing up stories and comments. The fix might be as simple as two domains that contain stories and comments cloned from HN, one domain for each, followed by Google Searches within those domains. No…

https://hn.algolia.com/ does the job pretty well!

Also, you can easily search HN in Chrome: start typing news.ycombinator.com, it will be ready for autocomplete, and it will say in grey text "Press [tab] to search HN". The results show up from algolia.

If only there were a way to make the default search order recency instead of popularity — most of my searching is before posting something, to make sure it hasn't already been posted.

Re: Tagger News takes a subset of HN articles, analyzes using ML, and applies tags

#14

Post by [deleted] »

nickpsecurity wrote: post #2

Strength of Hacker News is the network effect of a diverse, intelligent crowd. It would be hard to replace. A supplemental site tagging it and aiding search has value. Biggest problem I have searching HN, though, is Google mixing up stories and comments. The fix might be as simple as two domains that contain stories and comments cloned from HN, one domain for each, followed by Google Searches within those domains. No…

[deleted]

Re: Tagger News takes a subset of HN articles, analyzes using ML, and applies tags

#16

Post by saganus »

rocky1138 wrote: post #15

Stop trying to remake Hacker News. It's pretty much perfect the way it is.

Maybe it's not perfect for everyone.

I, for one, do like the idea of tagging stuff, since I might favorite a lot of stories but then years later it's hard to find a particular one, even if you do remember the general topic.

Tags for this would be really helpful for me, ergo it's not perfect for me, ergo it's good someone else is trying to make it better.

Since it's not affecting the original site, why would you want to stop them?

Re: Tagger News takes a subset of HN articles, analyzes using ML, and applies tags

#17

Post by var_explained »

minimaxir wrote: post #11

Direct URL to project details: https://devpost.com/software/tagger-news A few comments: 1. To other commenters, as with the HN Vue demo a week ago ( https://news.ycombinator.com/item?id=14284877 ), the project is a technical proof-of-concept; the aesthetics aren't the primary focus. 2. The Algolia API is better for scraping because it allows for bulk requests, unlike the official API (my old 2014 script still works I…

One of the devs here.

1. That's the way we were thinking about it :)

2. Oh, excellent! We hadn't found that or we'd have used it, and we'll start working with it.

3. Tomorrow I'm going to blog about how we approached the machine learning. Short version; we manually came up with regular expressions to classify a training set based on titles. The idea is that when we experimented with manual annotations on titles, the vast majority of the time we were looking for only a few key words. There's no question that this adds biases and will not be entirely accurate, but manual inspection convinced us it was a good enough approach for our hackathon, and most of the articles we identified with the resulting algorithm would not have been found by the title regex alone.

You can see the table of regular expressions [here](https://github.com/dodger487/analyze_hn/blob/master/topics.c...) and a bunch of (pretty unstructured) analysis code [here](https://github.com/dodger487/analyze_hn/blob/master/hn-analy...).

Re: Tagger News takes a subset of HN articles, analyzes using ML, and applies tags

#18

Post by shawkinaw »

I think this is great, especially being able to click a tag and see a top 30 list of that tag.

Obvious suggestions that would make it usable as a primary HN interface:

• Login and voting (not sure the HN API supports this though)

• Tag suggestions to feed into the model

Re: Tagger News takes a subset of HN articles, analyzes using ML, and applies tags

#19

Post by jibbolo »

var_explained wrote: post #17
minimaxir wrote: post #11

Direct URL to project details: https://devpost.com/software/tagger-news A few comments: 1. To other commenters, as with the HN Vue demo a week ago ( https://news.ycombinator.com/item?id=14284877 ), the project is a technical proof-of-concept; the aesthetics aren't the primary focus. 2. The Algolia API is better for scraping because it allows for bulk requests, unlike the official API (my old 2014 script still works I…

One of the devs here. 1. That's the way we were thinking about it :) 2. Oh, excellent! We hadn't found that or we'd have used it, and we'll start working with it. 3. Tomorrow I'm going to blog about how we approached the machine learning. Short version; we manually came up with regular expressions to classify a training set based on titles . The idea is that when we experimented with manual annotations on titles, the…

Hey mate, you should follow this guide step by step when you deploy a django app: https://docs.djangoproject.com/en/1.11/howto/deployment/chec...

BTW, congrats for the projects, well done!

Re: Tagger News takes a subset of HN articles, analyzes using ML, and applies tags

#20

Post by searchhn »

nickpsecurity wrote: post #2

Strength of Hacker News is the network effect of a diverse, intelligent crowd. It would be hard to replace. A supplemental site tagging it and aiding search has value. Biggest problem I have searching HN, though, is Google mixing up stories and comments. The fix might be as simple as two domains that contain stories and comments cloned from HN, one domain for each, followed by Google Searches within those domains. No…

http://searchhn.com is a small demo we have been building at Searchera (https://searchera.io)
Post reply on HN