Torrent here, when done:
http://burnbit.com/torrent/170493/twitter_sql_bz2
Free SQL dump with 200 million tweets from 13 million users
11–20 of 41 posts
Re: Free SQL dump with 200 million tweets from 13 million users
#12Thanks! More interested in the scraper.. is it open-source? If yes, where can we download it? If not, can you write about your experience in building it?
Writing a Twitter scraper is pretty trivial and you can find several good examples on Github. I'd put mine online, but the commands I was using in 2009/2010 are changed/deprecated largely and my code wouldn't run.
In either case, as Sethish said, distributing dumps like this is against the new ToS.
Re: Free SQL dump with 200 million tweets from 13 million users
#13Twitter changed their ToS to explicitly disallow distributing twitter dumps like this: http://chronicle.com/blogs/profhacker/the-end-of-twapperkeep... I was a part of the webecology project (and 140kit.com, both of which gave large twitter datasets to researchers.
Additionally, this data probably isn't as useful as many might think. We found that collecting random tweets probably isn't that useful for most research overall, partially because using any of the streaming APIs omit tweets. Even 'full' firehose seems to omit some tweets, so it can't be considered a complete set, nor verified as a completely random set.
Re: Free SQL dump with 200 million tweets from 13 million users
#14import to mysql: bunzip2 < my_database.sql.bz2 | mysql -h localhost -u root -p my_database
[deleted]
Re: Free SQL dump with 200 million tweets from 13 million users
#15Twitter changed their ToS to explicitly disallow distributing twitter dumps like this: http://chronicle.com/blogs/profhacker/the-end-of-twapperkeep... I was a part of the webecology project (and 140kit.com, both of which gave large twitter datasets to researchers.
Additionally, this data probably isn't as useful as many might think. We found that collecting random tweets probably isn't that useful for most research overall, partially because using any of the streaming APIs omit tweets. Even 'full' firehose seems to omit some tweets, so it can't be considered a complete set, nor verified as a completely random set.
-- I disagree.
- You can cluster users based on tweet data, links relationships &/or even user-to-user relationships
- Understand how retweets work and how fast they propagate.
- Sentiment analysis based on a specific keyword.
- Trend analysis.
There are N number of ways this dataset can be helpful. You have 200MM tweets. Enough for a quick experiment using real data.
* Its true that is "random" data. Just unrandom it!
Re: Free SQL dump with 200 million tweets from 13 million users
#16Earlier quoted context omitted.
Additionally, this data probably isn't as useful as many might think. We found that collecting random tweets probably isn't that useful for most research overall, partially because using any of the streaming APIs omit tweets. Even 'full' firehose seems to omit some tweets, so it can't be considered a complete set, nor verified as a completely random set.
-- I disagree. - You can cluster users based on tweet data, links relationships &/or even user-to-user relationships - Understand how retweets work and how fast they propagate. - Sentiment analysis based on a specific keyword. - Trend analysis. There are N number of ways this dataset can be helpful. You have 200MM tweets. Enough for a quick experiment using real data. * Its true that is "random" data. Just unrandom i…
User-to-user relationships aren't that great with incomplete data of the tweets, but also of the social graph. Pulling a large social graph from Twitter is nearly impossible and getting deltas on anything more than a few hundred people is equally impossible.
Propagation of retweets really needs a near complete dataset of those tweets/retweets. A steaming sample of the dataset really isn't great for this.
Sentiment analysis can be done to determine the overall feeling on a topic, but I'd feel really incomplete doing it on this dataset. Again, pulling the stream for the term or keyboard you're looking to sample is much better. Most sentiment analysis on Twitter is pretty flawed anyway.
Trend analysis works on this dataset ok, but measuring the true magnitude of an event would be hard (like Osama being killed) since you don't know what portion of the tweets you've actually got.
I worked with Sethish on the Web Ecology Project. I wouldn't call your dataset useless, but it really would be more useful generally to have a question, then pull the best possible data that will help you answer that question. Otherwise there's going to be a lot more unknowns that make it a weaker piece of research.
Re: Free SQL dump with 200 million tweets from 13 million users
#17Earlier quoted context omitted.
Additionally, this data probably isn't as useful as many might think. We found that collecting random tweets probably isn't that useful for most research overall, partially because using any of the streaming APIs omit tweets. Even 'full' firehose seems to omit some tweets, so it can't be considered a complete set, nor verified as a completely random set.
-- I disagree. - You can cluster users based on tweet data, links relationships &/or even user-to-user relationships - Understand how retweets work and how fast they propagate. - Sentiment analysis based on a specific keyword. - Trend analysis. There are N number of ways this dataset can be helpful. You have 200MM tweets. Enough for a quick experiment using real data. * Its true that is "random" data. Just unrandom i…
[deleted]
Re: Free SQL dump with 200 million tweets from 13 million users
#18Earlier quoted context omitted.
-- I disagree. - You can cluster users based on tweet data, links relationships &/or even user-to-user relationships - Understand how retweets work and how fast they propagate. - Sentiment analysis based on a specific keyword. - Trend analysis. There are N number of ways this dataset can be helpful. You have 200MM tweets. Enough for a quick experiment using real data. * Its true that is "random" data. Just unrandom i…
User-to-user relationships aren't that great with incomplete data of the tweets, but also of the social graph. Pulling a large social graph from Twitter is nearly impossible and getting deltas on anything more than a few hundred people is equally impossible. Propagation of retweets really needs a near complete dataset of those tweets/retweets. A steaming sample of the dataset really isn't great for this. Sentiment an…
your points are valid.
I want to clarify that this dump is for learning purposes due the lack of "open data".
If people can play with real data from real people and get "real" inputs, that can encourage curious programmers to join the data-mining party. I know there are other dumps out there, np with that. This is just another dump, it may help people come with ideas without the need of coding a a multi-threaded scrapper.
Re: Free SQL dump with 200 million tweets from 13 million users
#19Re: Free SQL dump with 200 million tweets from 13 million users
#20Earlier quoted context omitted.
To be honest I have no idea. It crawled 13MM users, some accounts can be very old with very old tweets... You can look at the CD_data table and look for the tweet html code and parse the timestamp.
Apparently Twitter now has 100+ Million tweets per DAY. So you caught about 2 days worth but randomly in time.
[deleted]