1–10 of 41 posts
- DB Size: 543 million rows
- Data Size: 173GB (uncompressed)
- Stored in mysql
- 200+ Million tweets from 13+ Million users
- Collected in 1 week
- Operation costs: 100+ dollars
- Rackspace Cloud - 1 CentOS 8GB Ram server
- Java, memcache, mysql and perl for core processing
- js, php for analytics & visualization
* Download the data at this url http://www.archive.org/details/2011-06-calufa-twitter-sql
Twitter search still only goes back 10 days in 2011, so how deep is this data?
Hmm, how many days back does it go? Twitter search still only goes back 10 days in 2011, so how deep is this data?
Thanks! More interested in the scraper.. is it open-source? If yes, where can we download it? If not, can you write about your experience in building it?
Hmm, how many days back does it go? Twitter search still only goes back 10 days in 2011, so how deep is this data?
To be honest I have no idea. It crawled 13MM users, some accounts can be very old with very old tweets... You can look at the CD_data table and look for the tweet html code and parse the timestamp.
So you caught about 2 days worth but randomly in time.
I was a part of the webecology project (and 140kit.com, both of which gave large twitter datasets to researchers.
Hmm, how many days back does it go? Twitter search still only goes back 10 days in 2011, so how deep is this data?
To be honest I have no idea. It crawled 13MM users, some accounts can be very old with very old tweets... You can look at the CD_data table and look for the tweet html code and parse the timestamp.
Twitter changed their ToS to explicitly disallow distributing twitter dumps like this: http://chronicle.com/blogs/profhacker/the-end-of-twapperkeep... I was a part of the webecology project (and 140kit.com, both of which gave large twitter datasets to researchers.
bunzip2 < my_database.sql.bz2 | mysql -h localhost -u root -p my_database