Live data from Hacker News

An algorithm for generating automatic hashtags

blog.swayy.co

1–10 of 35 posts

Re: An algorithm for generating automatic hashtags

#2
Although very basic, this is really nice - the wonders of NLTK! You explain your algorithm in very simple terms so kudos for that.

It might be a good idea to put this in a GitHub repository rather than a simple gist maybe? I am sure there will be plenty of people (including myself) interested in contributing, it's much easier doing it on a repository rather than a gist :)

Re: An algorithm for generating automatic hashtags

#4
Cool stuff. I have been playing with nltk and gensim myself recently in order to solve similar problem. But I want to combine unsupervised topic modeling (LDA) with some supervised learning algorithm (probably naive bayes). Struggling with LDA at this point, but pretty interesting stuff.

Re: An algorithm for generating automatic hashtags

#7
For those people who are interested in topic/article classification and NLP, Twitter can a gold-mine, especially hashtags. If you gather the hashtags for a million articles, you pretty much have a co-location database. Now you can mine that data and see which hashtags are common if you have "Google Panda" in your title for instance, or which hash tags are commonly used with #seo. Hashtags are basically structured semantic data, if you look at them in aggregation. A good tool for doing this is SOLR or ElasticSearch. Simply import all the hashtags for a bunch of articles to the index, and do a faceted search for a specific hashtag, or keyword, and you'll get the top 10 associated hashtags that are highly related to that keyword.

Re: An algorithm for generating automatic hashtags

#8

For those people who are interested in topic/article classification and NLP, Twitter can a gold-mine, especially hashtags. If you gather the hashtags for a million articles, you pretty much have a co-location database. Now you can mine that data and see which hashtags are common if you have "Google Panda" in your title for instance, or which hash tags are commonly used with #seo. Hashtags are basically structured sem…

something similar to - http://hashtagify.me

Re: An algorithm for generating automatic hashtags

#10
post #6

A strange example. Does the author really think hashtagifying the word 'content' has improved the tweet in some way? Do they expect people to be searching Twitter for #content and getting some useful results?

Probably not, but he did disclaim that it was pretty naive and could be improved in many ways. I think it's a pretty darn good first pass. That particular issues comes about from tagging words that are common in the target text without reference to whether or not that's actually significant- i.e., whether it's common in the text just because it's a common word overall, rather than because it's actually an indication of the text subject. That should be pretty easy to fix by comparing with an English word frequency list.
Post reply on HN