Reconstructing Twitter's Firehose

Post by toomuchtodo » Thu, Feb 28, 2019, 3:10 AM UTC

m-p-3 wrote: post #12

cdoxsey wrote: post #6

If you need access to Twitter data you should sign up for a developer account and contact them. It's expensive but not prohibitively so for a company that needs Twitter data to do business. For academic research Twitter offers products which dramatically reduce the amount of data you need to consume via the full search API or historical powertrack.

I'm wondering if the Wayback Machine has access to it.

Post by dev_dull » Thu, Feb 28, 2019, 3:21 AM UTC

jaytaylor wrote: post #41

undefined1 wrote: post #11

> Twitter’s firehose is a complete stream of all tweets made on their platform and is only available to a few businesses at an extraordinary price. Anyone happen to know what that price is?

Hundreds of thousands per month. At least that was the price in 2012.

Post by mhfs » Thu, Feb 28, 2019, 3:40 AM UTC

dev_dull wrote: post #52

mhfs wrote: post #32

Earlier quoted context omitted.

I worked with a company in the past that abused the twitter api to an unimaginable level from a single IP address. That was a few years ago but at the time I’m pretty sure their blacklisting was between inexistent and pathetic.

It’s sad that you think this somehow reflects badly on Twitter. I appreciate a company that will opt to be conservative rather than ban-hammer innocent people on accident just to stop a single idiot.

Post by jrochkind1 » Thu, Feb 28, 2019, 3:46 AM UTC

toomuchtodo wrote: post #51

m-p-3 wrote: post #12

Earlier quoted context omitted.

I'm wondering if the Wayback Machine has access to it.

I would be extremely interested in getting the Twitter corpus ingested into the Internet Archive. To my knowledge, the Archive does not have access to the firehose.

Post by aboutruby » Thu, Feb 28, 2019, 4:17 AM UTC

stuck_in_matrix wrote: post #40

aboutruby wrote: post #33

Earlier quoted context omitted.

I see there is interest in this observation so I used a little ruby (.to_s(2)[-22..-17].to_i(2)) to get the datacenter id. Then ran it on a few Twitter accounts: https://pastebin.com/w8Dnj5kM It does work, and it's going to be hard to patch edit: I realized you don't only get one location but the whole location history of a Twitter user. Also locating Twitter's data centers as it doesn't seem to be public information

This is really interesting. When I did the original analysis on datacenter / server ids, I didn't think about correlation with user accounts. Nice observation!