Live data from Hacker News

Looking At The World Through Twitter Data

arashd.scripts.mit.edu

1–10 of 27 posts

Re: Looking At The World Through Twitter Data

#2
If you're interested in how information diffuses through social networks like Twitter, take a look at Truthy (one of my projects):

http://truthy.indiana.edu

     Truthy is a system to analyze and visualize the
     diffusion of information on Twitter. The Truthy system
     evaluates thousands of tweets an hour to identify new
     and emerging bursts of activity around memes of various 
     flavors. The data and statistics provided by Truthy are
     designed to aid in the study of social epidemics: How do
     memes propagate through the Twittersphere? What causes a
     burst of popularity?

Re: Looking At The World Through Twitter Data

#4
But, this is because non-English tweets that we have discarded are much more frequent during the night in our time zone, and they often don’t contain the word ‘a’ as often as English tweets do.

This doesn't make sense; are they only discarding the non-English tweets during certains times?

Re: Looking At The World Through Twitter Data

#5
Wow... This is awesome. I actually did a project for my high school science fair that focused on analyzing twitter. It was no where near as sophisticated but it really opened my eyes to the massive amount of data and the availability of commodity hardware that can actually handle terabytes of data.

Re: Looking At The World Through Twitter Data

#8
post #6

Out of curiosity, is Twitter data such as this freely available to anyone, or was this specially acquired for this set of students? I can imagine a number of interesting projects that might arise out of such a data set.

You need to get whitelisted to have access to it. The only problem is that since they partnered with Gnip, Twitter no longer gives whitelist access for free (http://www.readwriteweb.com/archives/twitter_to_sell_50_of_a... ).

I was lucky enough to get it more than two years ago and have been accumulating data ever since.

Re: Looking At The World Through Twitter Data

#10
A friend and I started playing around with twitter data back in early 2010. We currently have something close to over 587 million tweets collected (We stopped collecting earlier this year). We only pulled English tweets and those that described what someone was feeling (Im, I am, I feel, I am feeling, etc. along with the negatives I don't feel, I do not feel, etc).

We were able to see some interesting events happen during the time though. This is a graph of the anxiety levels of twitter on March 11th, the bottom axis is the hour of the day EST. The earthquake hit Japan @ 1:46 EST.

http://i.imgur.com/BeBwa.jpg

There is a strange dip around noon that we are unsure of how to account for as our servers did not report any failures.

It was a fun project to play around with.

Post reply on HN