Ask HN: Has anyone tried to write a Bayesian classifier for stories? • Hacker News Board

Re: Ask HN: Has anyone tried to write a Bayesian classifier for stories?

#11

Post by StavrosK » Mon, Jan 21, 2013, 2:59 PM UTC

Houshalter wrote: post #10

StavrosK wrote: post #8

Earlier quoted context omitted.

There are various things you can do and various machine learning techniques you can use, but I imagine that the single user version would be enough, to start with. I'll give it a go and see if it works well. If it does, I might release it as a service.

That would be cool. The basic info might be enough. The thing about the previous guy who tried something like this which was posted above is, he was mostly sorting by the words in the title which seems like only a very weak predictor. Better would be the number of comments and votes, and maybe other stuff like how long the article is or whether or not certain words are in the comments. He also trained it on whether h…

I plan to implement votes (maybe), domain, actual raw text of the article, title, submitter (maybe) and show articles that are deemed "important" (i.e. have stayed on the front page for longer than X hours), as well as some random ones, to avoid a bubble. Plus, I've already started training the filter manually, I'll maybe write a simple web UI later on so I can up/downvote articles from there.

I think that should give a good first draft.

Re: Ask HN: Has anyone tried to write a Bayesian classifier for stories?

#12

Post by rjwebb » Tue, Jan 22, 2013, 1:00 PM UTC

StavrosK wrote: post #3

a_macgregor wrote: post #2

Stavrosk, If what you want is to only classify the stories in the front page and classify them based on a preset of categories, that's actually pretty simple to do. I been working on a similar concept for personal project. Here are my recommendations: - Be sure to remove stopwords from the titles before using the classifier. - The ankusa gem will help you greatly https://github.com/bmuller/ankusa Ankusa is a naive ba…

Thanks for your answer! What I'm thinking of making is basically separating posts into two categories, things that interest me and things that don't. Then, I want to receive emails at intervals I specify. This is so I no longer have the urge to check HN frequently, but still stay up t date. The actual classification is probably the easy part, the hard part is training the model, which is why I wanted to ask if anyone…

I suppose you could train the classifier by having it record what you upvote, or which links you click on. Perhaps a Firefox/Chrome extension could do that?

Some people at Reddit were programming a recommender about a year ago: http://www.reddit.com/r/redditdev/comments/lowwf/attempt_2_w... It doesn't use a Naive Bayesian Classifier but it might still interest you.

Re: Ask HN: Has anyone tried to write a Bayesian classifier for stories?

#13

Post by StavrosK » Tue, Jan 22, 2013, 1:07 PM UTC

rjwebb wrote: post #12

StavrosK wrote: post #3

Earlier quoted context omitted.

Thanks for your answer! What I'm thinking of making is basically separating posts into two categories, things that interest me and things that don't. Then, I want to receive emails at intervals I specify. This is so I no longer have the urge to check HN frequently, but still stay up t date. The actual classification is probably the easy part, the hard part is training the model, which is why I wanted to ask if anyone…

I suppose you could train the classifier by having it record what you upvote, or which links you click on. Perhaps a Firefox/Chrome extension could do that? Some people at Reddit were programming a recommender about a year ago: http://www.reddit.com/r/redditdev/comments/lowwf/attempt_2_w... It doesn't use a Naive Bayesian Classifier but it might still interest you.

I'm currently using a very simple bookmarklet scheme, one for upvote and one for downvote. It works very well for collecting data, I'll train it later tonight, I think.

Thank you for the link, it looks very extensive, I'll peruse it later on.