A new algorithm predicts which Twitter topics will trend hours in advance

#1

Post by denzil_correa » Wed, Nov 07, 2012, 12:58 PM UTC

A new algorithm predicts which Twitter topics will trend hours in advance
web.mit.edu

Re: A new algorithm predicts which Twitter topics will trend hours in advance

#2

Post by lucvh » Wed, Nov 07, 2012, 1:58 PM UTC

Do I predict this sparking a restriction in data availability in the API?

Re: A new algorithm predicts which Twitter topics will trend hours in advance

#3

Post by mhuffman » Wed, Nov 07, 2012, 2:20 PM UTC

lucvh wrote: post #2

Do I predict this sparking a restriction in data availability in the API?

That was my initial thoughts. Might this be a potential way to monetize twitter finally?

Re: A new algorithm predicts which Twitter topics will trend hours in advance

#4

Post by denzil_correa » Wed, Nov 07, 2012, 2:34 PM UTC

The only concern I have is the way the experiment is conducted. The correct way to collect data would be to consume the Streaming API to collect the random sample of tweets provided by Twitter [0]. They should then run their algorithm to compare how they are doing against Twitter's algorithm. Currently, they are filtering out certain topics of interest and just comparing these topics. Therefore, they are filtering a great degree of noise which is actually observed by Twitter.

[0] https://dev.twitter.com/docs/api/1.1/get/statuses/sample

Re: A new algorithm predicts which Twitter topics will trend hours in advance

#5

Post by denzil_correa » Wed, Nov 07, 2012, 2:35 PM UTC

lucvh wrote: post #2

Do I predict this sparking a restriction in data availability in the API?

They are already monetizing their data. DataSift is one of their key official partners [0].

[0] http://datasift.com/

Re: A new algorithm predicts which Twitter topics will trend hours in advance

#6

Post by klapinat0r » Wed, Nov 07, 2012, 2:39 PM UTC

mhuffman wrote: post #3

lucvh wrote: post #2

Do I predict this sparking a restriction in data availability in the API?

That was my initial thoughts. Might this be a potential way to monetize twitter finally?

Could you elaborate please? Are you refering to the "early access" to trending topics, or restricting limits on API?

Re: A new algorithm predicts which Twitter topics will trend hours in advance

#7

Post by gubatron » Wed, Nov 07, 2012, 2:44 PM UTC

can it be used to predict market trends? instead of topics use stock tickers?

Re: A new algorithm predicts which Twitter topics will trend hours in advance

#8

Post by denzil_correa » Wed, Nov 07, 2012, 2:50 PM UTC

gubatron wrote: post #7

can it be used to predict market trends? instead of topics use stock tickers?

There is already some work done on this front [0]. Ruiz et al basically concluded the stock price of a company is directly proportional to the number of discussion topics of that company viz. if there for a company X there are fewer topics under discussion the stock prices would be poor.

[0] http://www.cs.ucr.edu/~vagelis/publications/wsdm2012-microbl...

Re: A new algorithm predicts which Twitter topics will trend hours in advance

#9

Post by VMG » Wed, Nov 07, 2012, 2:51 PM UTC

Does the algorithm still work when it includes it's own effects?

If an algorithm like this was widely used, the predicted topics will trend just because they have been picked -- the algorithm will fulfill the prophecy itself.

Re: A new algorithm predicts which Twitter topics will trend hours in advance

#10

Post by sanxiyn » Wed, Nov 07, 2012, 3:38 PM UTC

The algorithm itself seems very simple. The paper quotes "The Unreasonable Effectiveness of Data". The surprising thing is the simple algorithm works great given large data.