Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter to Release All Tweets to Scientists

scientificamerican.com

Post reply on HN
Print view View original

31–40 of 55 posts

Re: Twitter to Release All Tweets to Scientists

#31

Post by Wilya »

StavrosK wrote: post #26
beejiu wrote: post #24

Earlier quoted context omitted.

Considering there's at least 400GB of data generated per day, I don't think it'll be readily available for the public as a download.

Jeez, 400 GB of text per day? How the hell? EDIT: That's 11.5k tweets/sec. How do you get eleven thousand people to tweet every second?

Depends how much metadata there is with every tweet.

I took a random tweet on my timeline, and the JSON representation, with added metadata, weighs 8740 bytes.

With this figure, you would "only" need 500 tweets/sec to get 400GB a day.

There's a lot of people on Twitter, and a lot of bots, so that doesn't seem unreasonable to me.

Re: Twitter to Release All Tweets to Scientists

#32

Post by nevinera »

StavrosK wrote: post #26
beejiu wrote: post #24

Earlier quoted context omitted.

Considering there's at least 400GB of data generated per day, I don't think it'll be readily available for the public as a download.

Jeez, 400 GB of text per day? How the hell? EDIT: That's 11.5k tweets/sec. How do you get eleven thousand people to tweet every second?

Most of that data is not the content of the tweet itself, but the metadata associated with it. When I last checked, we were storing about a kilobyte of data for every tweet.

Also, tweets are limited to 140 characters, not bytes - chinese tweets typically take about 200-250 bytes, for example.

Re: Twitter to Release All Tweets to Scientists

#33

Post by nevinera »

NamTaf wrote: post #13

I can't wait to see someone legitimately design a better sewerage system by using twitter's geolocation.

Their geo-data is utter crap. The vast majority of it is based on 'profile location', which means that there are almost a million people tweeting from the exact center of Atlanta. It's a crowded spot, must be a Starbucks there or something.

Re: Twitter to Release All Tweets to Scientists

#34

Post by TallGuyShort »

coherentpony wrote: post #19
flycaliguy wrote: post #16

There is no magic discovery about the nature of man hidden away in that data. Nothing you're average stand up comedian hasn't already written a bit about.

That's a little cynical. I like to think we can predict big historical social events (regime changes, major protests, climate change?) based on Twitter's data. You never know unless you try.

Are you familiar with Atrocity Watch? If not, you should look them up - that's exactly what they do.

Re: Twitter to Release All Tweets to Scientists

#37

Post by uptown »

One thing I've wondered. Is it possible to follow "everyone" on Twitter? If-not, what type of cap does Twitter enforce on the number of accounts you're allowed to follow? I realize it'd be difficult to know which new accounts to add as people join, but how far could you push a roll-your-own stream of the Twitter firehose?

Re: Twitter to Release All Tweets to Scientists

#38

Post by jonknee »

uptown wrote: post #37

One thing I've wondered. Is it possible to follow "everyone" on Twitter? If-not, what type of cap does Twitter enforce on the number of accounts you're allowed to follow? I realize it'd be difficult to know which new accounts to add as people join, but how far could you push a roll-your-own stream of the Twitter firehose?

http://lmgtfy.com/?q=twitter+maximum+follow+limit&l=1

Re: Twitter to Release All Tweets to Scientists

#39

Post by JamesMcMinn »

School-Cotton wrote: post #4
apetresc wrote: post #2

This is exciting to me; does anyone know how Twitter will go about this? Will there be a public dataset available for download? A research contract through the recently-acquired GNIP? Or just firehose access for future streams?

Doesn't Twitter make a fair bit of money from selling access to various slices of their data? I'd be surprised if they released it all to the general public. I imagine scientists would have to be under some sort of NDA.

The Twitter terms of service prohibit sharing the data in the Tweets. Researchers are allowed to share tweet IDs and User IDs which can be used to identify a Tweet. Currently, Twitter collections are shared using this method -- I recently released a dataset of 120 million Tweet IDs which cover a sample of a months worth of data, and numerous other researchers have used these IDs to crawl Twitter and obtain the same dataset as I used in my experiments.

Re: Twitter to Release All Tweets to Scientists

#40

Post by JamesMcMinn »

StavrosK wrote: post #26
beejiu wrote: post #24

Earlier quoted context omitted.

Considering there's at least 400GB of data generated per day, I don't think it'll be readily available for the public as a download.

Jeez, 400 GB of text per day? How the hell? EDIT: That's 11.5k tweets/sec. How do you get eleven thousand people to tweet every second?

I have access to the Twitter gardenhose (which is equal to slightly less than 10% of the full volume). These are the RX and TX statistics from the machine that I've been using to gather data for a few months now:

RX bytes:18053724505080 (16.4 TiB) TX bytes:2686623557042 (2.4 TiB)

It works out at around 70GB/day, so I'd actually think that the full firehose would use considerably more data than 400GB/day (likely closer to 800GB).

Post reply on HN