Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reconstructing Twitter's Firehose

docs.google.com

Post reply on HN
Print view View original

1–10 of 112 posts

Re: Reconstructing Twitter's Firehose

#4

Post by nspkr »

>Twitter’s statuses lookup API endpoint allows for a total of 1,200 API calls every 15 minutes. Each call allows the user to pass 100 ids for a total of 120,000 id requests every 15 minutes using both APP auth [...]

Use the secret consumer keys from Twitter to bypass these limits: https://gist.github.com/shobotch/5160017

Re: Reconstructing Twitter's Firehose

#5

Post by tpetry »

rhcom2 wrote: post #3

Interesting technical write up but a little naive about its uses as it is probably against the Twitter Developer agreement to use the API as it circumvents the rate limits.

You can‘t circumvent the rate limit, that is the idea of a rate limit. The article is stating this all the time.

The genious idea is simply shrinking down the search space by probabilistic assumptions. Never thought about using the snowflake ids to get historical data, i am impressed.

Re: Reconstructing Twitter's Firehose

#6

Post by cdoxsey »

If you need access to Twitter data you should sign up for a developer account and contact them. It's expensive but not prohibitively so for a company that needs Twitter data to do business.

For academic research Twitter offers products which dramatically reduce the amount of data you need to consume via the full search API or historical powertrack.

Re: Reconstructing Twitter's Firehose

#8

Post by spullara »

edent wrote: post #2

That's excellent stuff. I had no idea that's how snowflake IDs were constructed. Now makes me wonder how hard it would be to make a tweet which linked to itself...?

It has been done but I can't find a link to a modern example, only those before Snowflake IDs.
Post reply on HN