Twitter to Release All Tweets to Scientists • Hacker News Board

Re: Twitter to Release All Tweets to Scientists

#21

Post by dirtyaura » Tue, May 27, 2014, 5:42 AM UTC

flycaliguy wrote: post #16

There is no magic discovery about the nature of man hidden away in that data. Nothing you're average stand up comedian hasn't already written a bit about.

That was a good one.

On a more serious tone, there is one area of research that Twitter data is very valuable for: how information and disinformation is created and spread during major news events: wars, catastrophes, uprisings, school shootings etc.

My gut feeling is that today a lot of quality journalism happens outside of the traditional journalistic organisations. The downside is that also a lot of wild speculation and rumours are spread, but it would be valuable to see how good this modern "crowd" journalism is. A skilled research group can use Twitter data and Internet Archive to track down the original sources of information pretty well.

Re: Twitter to Release All Tweets to Scientists

#22

Post by unclesaamm » Tue, May 27, 2014, 5:44 AM UTC

Can anyone find a primary source?

Re: Twitter to Release All Tweets to Scientists

#23

Post by scalene » Tue, May 27, 2014, 6:55 AM UTC

Not to be skeptical, but I'm pretty sure one of these scientists may happen to work for the NSA.

Re: Twitter to Release All Tweets to Scientists

#24

Post by beejiu » Tue, May 27, 2014, 9:59 AM UTC

apetresc wrote: post #2

This is exciting to me; does anyone know how Twitter will go about this? Will there be a public dataset available for download? A research contract through the recently-acquired GNIP? Or just firehose access for future streams?

Considering there's at least 400GB of data generated per day, I don't think it'll be readily available for the public as a download.

Re: Twitter to Release All Tweets to Scientists

#25

Post by mike415 » Tue, May 27, 2014, 10:02 AM UTC

unclesaamm wrote: post #22

Can anyone find a primary source?

There is a link to apply for a "Data Grant" here: https://engineering.twitter.com/research. Unfortunately, it looks like submissions are closed.

Re: Twitter to Release All Tweets to Scientists

#26

Post by StavrosK » Tue, May 27, 2014, 10:34 AM UTC

beejiu wrote: post #24

apetresc wrote: post #2

This is exciting to me; does anyone know how Twitter will go about this? Will there be a public dataset available for download? A research contract through the recently-acquired GNIP? Or just firehose access for future streams?

Considering there's at least 400GB of data generated per day, I don't think it'll be readily available for the public as a download.

Jeez, 400 GB of text per day? How the hell?

EDIT: That's 11.5k tweets/sec. How do you get eleven thousand people to tweet every second?

Re: Twitter to Release All Tweets to Scientists

#27

Post by Smulv » Tue, May 27, 2014, 10:40 AM UTC

It appears as if the data is only available to those scientists who apply for the data grant and win it. Furthermore, applications for the grant have been closed since midway through March. Yea, I'm not surprised Twitter isn't making its historical data public. That would literally end Gnip, which is a revenue source for Twitter not based on advertising to users.

Re: Twitter to Release All Tweets to Scientists

#28

Post by vschiavoni » Tue, May 27, 2014, 10:57 AM UTC

StavrosK wrote: post #26

beejiu wrote: post #24

Earlier quoted context omitted.

Considering there's at least 400GB of data generated per day, I don't think it'll be readily available for the public as a download.

Jeez, 400 GB of text per day? How the hell? EDIT: That's 11.5k tweets/sec. How do you get eleven thousand people to tweet every second?

Have you ever heard about the 'Human Brain Project' ?

Well, they produce 100 GB of data _per second_

Welcome to the real big data world.

Re: Twitter to Release All Tweets to Scientists

#29

Post by Houshalter » Tue, May 27, 2014, 11:05 AM UTC

StavrosK wrote: post #26

beejiu wrote: post #24

Earlier quoted context omitted.

Considering there's at least 400GB of data generated per day, I don't think it'll be readily available for the public as a download.

Jeez, 400 GB of text per day? How the hell? EDIT: That's 11.5k tweets/sec. How do you get eleven thousand people to tweet every second?

Maybe it includes images.

Re: Twitter to Release All Tweets to Scientists

#30

Post by maxerickson » Tue, May 27, 2014, 11:31 AM UTC

StavrosK wrote: post #26

beejiu wrote: post #24

Earlier quoted context omitted.

Considering there's at least 400GB of data generated per day, I don't think it'll be readily available for the public as a download.

Jeez, 400 GB of text per day? How the hell? EDIT: That's 11.5k tweets/sec. How do you get eleven thousand people to tweet every second?

Pick a topic you are familiar with. Open up a twitter search for it. Wait a while. See the inevitable storm of tweet-spam that sort of looks like social sharing.

Thousands of fake accounts are tweeting out nonsense all the time. Another example, there are multiple accounts that tweet items from HN (and presumably lots of other rss feeds).