Show HN: Subreddit classifier using scikit-learn and a high-performance Go proxy
1–10 of 18 posts
Re: Show HN: Subreddit classifier using scikit-learn and a high-performance Go proxy
#2This is really cool. I've been working on a search engine for lectures, and I recently set it up so you can filter conference talks by the programming language.
https://www.findlectures.com/?p=1&class1=Technology&category...
For the first iteration I wrote heuristics based on a list of language-specific subreddits. The technique this blog post describes is the next logical step, so I'm thrilled about this write-up.
Re: Show HN: Subreddit classifier using scikit-learn and a high-performance Go proxy
#3This is really cool. I've been working on a search engine for lectures, and I recently set it up so you can filter conference talks by the programming language. https://www.findlectures.com/?p=1&class1=Technology&category... For the first iteration I wrote heuristics based on a list of language-specific subreddits. The technique this blog post describes is the next logical step, so I'm thrilled about this write-up.
Thank you. Post author here. I'm glad you enjoyed it. I'll write many more blog posts over the christmas period on the same topic, so if you enjoyed then do keep checking.
Re: Show HN: Subreddit classifier using scikit-learn and a high-performance Go proxy
#4Wait, you had 15 test cases, and didn't even say how the classifier did? I guess I could run the code myself, and that's nice you provided it on Github, but still, I feel like you gotta post results in a blog post like this.
Re: Show HN: Subreddit classifier using scikit-learn and a high-performance Go proxy
#5This is really cool. I've been working on a search engine for lectures, and I recently set it up so you can filter conference talks by the programming language. https://www.findlectures.com/?p=1&class1=Technology&category... For the first iteration I wrote heuristics based on a list of language-specific subreddits. The technique this blog post describes is the next logical step, so I'm thrilled about this write-up.
Thank you. Post author here. I'm glad you enjoyed it. I'll write many more blog posts over the christmas period on the same topic, so if you enjoyed then do keep checking.
Agree, nice article! Afraid other than your posts making it to front page, I don't see how to keep checking:
-- can't find an RSS feed for your blog -- ioloop.io has a blank homepage -- ioloop.io/blog gives a 404
So where's the best place to keep checking?
Thanks!
Re: Show HN: Subreddit classifier using scikit-learn and a high-performance Go proxy
#6This is a really great resource. Thank you.
I'm just getting started with experimenting in this topic, and it's great to see something that isn't about classifying flower petals that is approachable!
Re: Show HN: Subreddit classifier using scikit-learn and a high-performance Go proxy
#7Wait, you had 15 test cases, and didn't even say how the classifier did? I guess I could run the code myself, and that's nice you provided it on Github, but still, I feel like you gotta post results in a blog post like this.
You are indeed right, and I encourage anyone interested in computing the precision metrics to do so.
The focus of this post is the GoLang proxy used for the caching. It's actually used in a CI / CD environment, but I'm finding it incredibly useful for a whole variety of tasks.
Regarding the precision metrics, you can find all the information required here http://scikit-learn.org/stable/tutorial/text_analytics/worki... to compute the precision.
In the scikit-learn article the classifier scores over 90% precision, so I'd expect it would be possible to do the same.
I'll be posting more about this over the xmas period, so I'll write a part II, where I compute the precision metrics.
Another thing worth noting is that it's not caching the HN comments, while it is using the reddit comments. Despite using tfidf, this still completely skews the results towards reddit as opposed to HN. So that's something else any interested reader can look into.
Thank you for pointing this out.
Re: Show HN: Subreddit classifier using scikit-learn and a high-performance Go proxy
#8Wait, you had 15 test cases, and didn't even say how the classifier did? I guess I could run the code myself, and that's nice you provided it on Github, but still, I feel like you gotta post results in a blog post like this.
You are indeed right, and I encourage anyone interested in computing the precision metrics to do so. The focus of this post is the GoLang proxy used for the caching. It's actually used in a CI / CD environment, but I'm finding it incredibly useful for a whole variety of tasks. Regarding the precision metrics, you can find all the information required here http://scikit-learn.org/stable/tutorial/text_analytics/worki..…
Well sure it's possible, I've done writeups on text classification with Scikit-Learn as well here:
https://bigishdata.com/2016/12/05/classifying-amazon-reviews...
But results and how well the classifier performs really just depends on the quality and amount of training data you have. So would be interesting to see how this does if you can get a bunch more data from each of the subreddits and have some more test examples!
Re: Show HN: Subreddit classifier using scikit-learn and a high-performance Go proxy
#9Earlier quoted context omitted.
Thank you. Post author here. I'm glad you enjoyed it. I'll write many more blog posts over the christmas period on the same topic, so if you enjoyed then do keep checking.
Agree, nice article! Afraid other than your posts making it to front page, I don't see how to keep checking: -- can't find an RSS feed for your blog -- ioloop.io has a blank homepage -- ioloop.io/blog gives a 404 So where's the best place to keep checking? Thanks!
+1
Re: Show HN: Subreddit classifier using scikit-learn and a high-performance Go proxy
#10This is really cool. I've been working on a search engine for lectures, and I recently set it up so you can filter conference talks by the programming language. https://www.findlectures.com/?p=1&class1=Technology&category... For the first iteration I wrote heuristics based on a list of language-specific subreddits. The technique this blog post describes is the next logical step, so I'm thrilled about this write-up.
Thank you. Post author here. I'm glad you enjoyed it. I'll write many more blog posts over the christmas period on the same topic, so if you enjoyed then do keep checking.
I did, I'm going to try to replicate what you did then as well :)