How about just loosening the API rate limits, or making a better token request process with resource allocation e.g. I'd like 1500 requests per 15 min window (as opposed to 15, for some things) for 72 hours. I guess this could be limited to those with a academic email address if they insist.
Twitter to Release All Tweets to Scientists
51–55 of 55 posts
Re: Twitter to Release All Tweets to Scientists
#52Earlier quoted context omitted.
Not all tweets are public.
Surely they aren't coughing up private tweets?
That was my interpretation of "all", but reading back it seems there's another difference:
"Although a majority of tweets are public, if scientists want to freely search the lot, they do it through Twitter's application programming interface, which currently scours only 1 percent of the archive. But that is about to change: in February the company announced that it will make all its tweets, dating back to 2006, freely available to researchers."
Re: Twitter to Release All Tweets to Scientists
#53I've heard that before. * Library of Congress: http://blogs.loc.gov/loc/2013/01/update-on-the-twitter-archi... * Twitter Data grants: https://blog.twitter.com/2014/introducing-twitter-data-grant... I'll admit, I haven't applied for access through either one, but neither have I seen any papers cite access through those venues—and I read quite a few NLP + Twitter papers.
from http://www.loc.gov/today/pr/2013/files/twitter_report_2013ja...
"Transfer of Data to the Library
In December, 2010, Twitter named a Colorado-based company, Gnip, as the delivery agent for moving data to the Library. Shortly thereafter, the Library and Gnip began to agree on specifications and processes for the transfer of files - "current" tweets - on an ongoing basis.
In February 2011, transfer of "current" tweets was initiated and began with tweets from December 2010.
On February 28, 2012, the Library received the 2006-2010 archive through Gnip in three compressed files totaling 2.3 terabytes. When uncompressed the files total 20 terabytes. The files contained approximately 21 billion tweets, each with more than 50 accompanying metadata fields, such as place and description.
As of December 1, 2012,the Library has received more than 150 billion additional tweets and corresponding metadata, for a total including the 2006-2010 archive of approximately 170 billion tweets totaling 133.2 terabytes for two compressed copies."
I find the quantities hilarious. But since they haven't been able to cope with providing access yet I get pessimistic about their prospects of doing so at all any time soon.
Can we do something to help them?
I've been thinking maybe GPU-accelerated databases like MapD, could mitigate the cost issue for them, but I'm pretty sure that doesn't go all the way to solving the problem...
Re: Twitter to Release All Tweets to Scientists
#54Earlier quoted context omitted.
That's a little cynical. I like to think we can predict big historical social events (regime changes, major protests, climate change?) based on Twitter's data. You never know unless you try.
Are you familiar with Atrocity Watch? If not, you should look them up - that's exactly what they do.
I am not. Checking them out now; looks pretty cool.
Re: Twitter to Release All Tweets to Scientists
#55Earlier quoted context omitted.
Jeez, 400 GB of text per day? How the hell? EDIT: That's 11.5k tweets/sec. How do you get eleven thousand people to tweet every second?
Most of that data is not the content of the tweet itself, but the metadata associated with it. When I last checked, we were storing about a kilobyte of data for every tweet. Also, tweets are limited to 140 characters , not bytes - chinese tweets typically take about 200-250 bytes, for example.
Yeah, I figured 300 bytes per Tweet, to be generous, but didn't realize it would take 1 kb of metadata. Thanks for that detail.