Reconstructing Twitter's Firehose • Hacker News Board

Re: Reconstructing Twitter's Firehose

#11

Post by undefined1 » Wed, Feb 27, 2019, 11:30 PM UTC

> Twitter’s firehose is a complete stream of all tweets made on their platform and is only available to a few businesses at an extraordinary price.

Anyone happen to know what that price is?

Re: Reconstructing Twitter's Firehose

#12

Post by m-p-3 » Wed, Feb 27, 2019, 11:37 PM UTC

cdoxsey wrote: post #6

If you need access to Twitter data you should sign up for a developer account and contact them. It's expensive but not prohibitively so for a company that needs Twitter data to do business. For academic research Twitter offers products which dramatically reduce the amount of data you need to consume via the full search API or historical powertrack.

I'm wondering if the Wayback Machine has access to it.

Re: Reconstructing Twitter's Firehose

#13

Post by throwawaymath » Wed, Feb 27, 2019, 11:38 PM UTC

nspkr wrote: post #4

>Twitter’s statuses lookup API endpoint allows for a total of 1,200 API calls every 15 minutes. Each call allows the user to pass 100 ids for a total of 120,000 id requests every 15 minutes using both APP auth [...] Use the secret consumer keys from Twitter to bypass these limits: https://gist.github.com/shobotch/5160017

I can't imagine these will last long now that they've been posted in a comment on a front page HN story.

Re: Reconstructing Twitter's Firehose

#14

Post by gberger » Wed, Feb 27, 2019, 11:47 PM UTC

throwawaymath wrote: post #13

nspkr wrote: post #4

>Twitter’s statuses lookup API endpoint allows for a total of 1,200 API calls every 15 minutes. Each call allows the user to pass 100 ids for a total of 120,000 id requests every 15 minutes using both APP auth [...] Use the secret consumer keys from Twitter to bypass these limits: https://gist.github.com/shobotch/5160017

I can't imagine these will last long now that they've been posted in a comment on a front page HN story.

These are hardcoded in the apps. If they disable a key, they would need to release a new version of the app (ok), and all users would need to update (infeasible).

Re: Reconstructing Twitter's Firehose

#15

Post by ihuman » Wed, Feb 27, 2019, 11:50 PM UTC

throwawaymath wrote: post #13

nspkr wrote: post #4

>Twitter’s statuses lookup API endpoint allows for a total of 1,200 API calls every 15 minutes. Each call allows the user to pass 100 ids for a total of 120,000 id requests every 15 minutes using both APP auth [...] Use the secret consumer keys from Twitter to bypass these limits: https://gist.github.com/shobotch/5160017

I can't imagine these will last long now that they've been posted in a comment on a front page HN story.

The Twitter for Android key and secret is at least 8 years old

https://twitter.com/kevinriggle/status/23932444186

Re: Reconstructing Twitter's Firehose

#16

Post by cwkoss » Wed, Feb 27, 2019, 11:58 PM UTC

Do the server and datacenter ID's effectively leak location data?

Re: Reconstructing Twitter's Firehose

#17

Post by _wmd » Thu, Feb 28, 2019, 12:04 AM UTC

tbodt wrote: post #10

Twitter will just randomize the machine IDs and datacenter IDs

Doing this for historical data would break their API. They'd also need a more complex scheme than snowflake currently provides

Re: Reconstructing Twitter's Firehose

#18

Post by andrewguenther » Thu, Feb 28, 2019, 12:05 AM UTC

This is a really neat approach! A friend of mine actually did his thesis working with Netflix to analyze Twitter and determine if Netflix was down[1]. He took a much more brute force approach, taking advantage of the fact that his search space was limited to re-assemble a "firehose" for a given search term leveraging hitting the search API from multiple machines and then reconciling the stream and removing duplicates. This approach definitely wouldn't scale to the entirety of the firehose, but I think that, in general, interest in Twitter data revolves around more specific queries.

[1] https://users.soe.ucsc.edu/~eaugusti/pages/papers/docs/SPOON...

Re: Reconstructing Twitter's Firehose

#19

Post by nspkr » Thu, Feb 28, 2019, 12:09 AM UTC

throwawaymath wrote: post #13

nspkr wrote: post #4

>Twitter’s statuses lookup API endpoint allows for a total of 1,200 API calls every 15 minutes. Each call allows the user to pass 100 ids for a total of 120,000 id requests every 15 minutes using both APP auth [...] Use the secret consumer keys from Twitter to bypass these limits: https://gist.github.com/shobotch/5160017

I can't imagine these will last long now that they've been posted in a comment on a front page HN story.

I've been using them for years now.

Re: Reconstructing Twitter's Firehose

#20

Post by pomber » Thu, Feb 28, 2019, 12:11 AM UTC

edent wrote: post #2

That's excellent stuff. I had no idea that's how snowflake IDs were constructed. Now makes me wonder how hard it would be to make a tweet which linked to itself...?

It's not that hard: https://twitter.com/mauritscorneIis

Code is here: https://github.com/pomber/escher-bot

It's a pity that twitter UI doesn't embed the recursive tweet.