Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Free SQL dump with 200 million tweets from 13 million users

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 41 posts

Re: Free SQL dump with 200 million tweets from 13 million users

#31

Post by achompas »

tibbon wrote: post #13
sethish wrote: post #7

Twitter changed their ToS to explicitly disallow distributing twitter dumps like this: http://chronicle.com/blogs/profhacker/the-end-of-twapperkeep... I was a part of the webecology project (and 140kit.com, both of which gave large twitter datasets to researchers.

Additionally, this data probably isn't as useful as many might think. We found that collecting random tweets probably isn't that useful for most research overall, partially because using any of the streaming APIs omit tweets. Even 'full' firehose seems to omit some tweets, so it can't be considered a complete set, nor verified as a completely random set.

This is a gold mine for a budding programmer. Anyone interested in learning MapReduce frameworks or messing with sentiment analysis/classification should get this data.

With that said...the data is now unavailable?

Re: Free SQL dump with 200 million tweets from 13 million users

#32

Post by sigil »

sethish wrote: post #7

Twitter changed their ToS to explicitly disallow distributing twitter dumps like this: http://chronicle.com/blogs/profhacker/the-end-of-twapperkeep... I was a part of the webecology project (and 140kit.com, both of which gave large twitter datasets to researchers.

I have 500GB of tweets from the random sample stream, in raw json form, going back to August 2010. Aside from some very broad measures it hasn't been all that useful to us. I never considered distributing it until now, but the TOS clause forbidding redistribution wrankles me. Are we really all okay with contributing content under these terms?

Re: Free SQL dump with 200 million tweets from 13 million users

#33

Post by a3_nm »

sethish wrote: post #7

Twitter changed their ToS to explicitly disallow distributing twitter dumps like this: http://chronicle.com/blogs/profhacker/the-end-of-twapperkeep... I was a part of the webecology project (and 140kit.com, both of which gave large twitter datasets to researchers.

This is ridiculous. Are ToS of this kind really enforceable? (What about outside the US?)

(Anyway, I guess that in practice, if a torrent with a dump of tweets appears somewhere, it's pretty hard to find out who did it. Yes, Twitter could do some clever watermarking of the API results or correlate the dump contents with the server logs, but it would probably be a lot of work.)

Re: Free SQL dump with 200 million tweets from 13 million users

#37

Post by PostOnce »

chrisjsmith wrote: post #25

All that is meaningless chatter between people and information about bathroom habits. Perhaps if we pooled that distributed effort into something constructive, the world would be a better place.

http://twitter.com/#!/id_aa_carmack

"Msbuild seems to limit to 100 files on a cl command line, which introduces noticeable sync losses when parallel building on 24 threads."

It's not all meaningless, you just choose to follow meaningless users.

Re: Free SQL dump with 200 million tweets from 13 million users

#39

Post by JeeyoungKim »

JeeyoungKim wrote: post #38

Does anybody want to share MD5 hash of the file? I'm trying to decompress this file, and I'm keep getting an error.

wait, the torrent link has it. I do have the same md5hash, and yet, it's keep crashing whenever i'm trying to uncompress this shit... wtf is going on.
Post reply on HN