Latent Dirichlet Allocation on Tweets • Hacker News Board

Re: Latent Dirichlet Allocation on Tweets

#11

Post by biomimic » Mon, Sep 08, 2014, 2:34 AM UTC

dhotson wrote: post #10

Very cool! I've read that LDA doesn't work well on short documents. Your approach of concatenating all tweets for a user appears to work quite well. One other technique I've seen is to concatenate multiple tweets together that contain the same hashtag. One of our intern students at 99designs did some work on applying LDA to classify graphic design tasks: http://99designs.com.au/tech-blog/blog/2014/01/22/Swiftly-Ma...…

Also used to classify genes related to lifespan - LDA on Statistical modeling of biomedical corpora: mining the Caenorhabditis Genetic Center Bibliography for genes related to life span. http://www.ncbi.nlm.nih.gov/pubmed/16681860

Re: Latent Dirichlet Allocation on Tweets

#12

Post by BIackSwan » Mon, Sep 08, 2014, 3:38 AM UTC

We did something similar about 3 and a half years ago where we detected online communities based on their follower network and text of tweets. Something you might be interested in - http://www.slideshare.net/akshayubhat/twitter-lda

Re: Latent Dirichlet Allocation on Tweets

#13

Post by datahipster » Mon, Sep 08, 2014, 8:58 AM UTC

I did something similar to this with tweets during the Boston Marathon bombing [0]. One of the coolest things that I saw was that the topics themselves were neatly ordered in time. In other words, you can visualize how the distribution of vocabulary evolves with time [1].

It would be interesting to extend the LDA model to include a temporal variable. Never got around to doing it, but it seems like it would work well for social media data.

[0] http://blog.dc.esri.com/2013/04/18/the-evolution-of-discussi...

[1] http://blog.dc.esri.com/files/2013/04/topic-distribution2.pn...

Re: Latent Dirichlet Allocation on Tweets

#14

Post by datahipster » Mon, Sep 08, 2014, 9:00 AM UTC

xtacy wrote: post #8

Daishiman wrote: post #6

It's a fun exercise for a single tweet feed, but it is unfortunately not very useful when applied to larger scale learning, since you start hitting the model's limitations of a fixed number of topics and, its potentially long runtime, and the fact that it's not an online method (although there are online variants of LDA).

Would you like to point us to some progress on how to overcome these limitations with LDA?

Also check out Vowpal Wabbit [0]. Although this is a tool that provides several online versions of classification, regression, etc, there's an online LDA module included as well. It takes advantage of the "hashing trick", meaning that you can build an LDA model with a fixed memory footprint. I've had a lot of success using this tool.

[0] https://github.com/JohnLangford/vowpal_wabbit