This looks to considerably lower the pain of mapping retweet networks, thanks!
Reconstructing Twitter's Firehose
21–30 of 112 posts
Re: Reconstructing Twitter's Firehose
#22They used word "efficiently" and some Python code in the same paragraph. Having worked for quite a while with Python now, I'd say their notion of efficiency is different than mine.
Re: Reconstructing Twitter's Firehose
#23This is proper "hacker news" right here
Re: Reconstructing Twitter's Firehose
#24[deleted]
Re: Reconstructing Twitter's Firehose
#25This is really interesting, an unforeseen insight into ids that can be enumerated even partially. I suppose encrypting or decrypting them is too costly so any other ideas for getting the properties of Snowflake at scale without this sort of attack being possible?
Re: Reconstructing Twitter's Firehose
#26>Twitter’s statuses lookup API endpoint allows for a total of 1,200 API calls every 15 minutes. Each call allows the user to pass 100 ids for a total of 120,000 id requests every 15 minutes using both APP auth [...] Use the secret consumer keys from Twitter to bypass these limits: https://gist.github.com/shobotch/5160017
Seems like a good way to get your IP blacklisted
Re: Reconstructing Twitter's Firehose
#27>Twitter’s statuses lookup API endpoint allows for a total of 1,200 API calls every 15 minutes. Each call allows the user to pass 100 ids for a total of 120,000 id requests every 15 minutes using both APP auth [...] Use the secret consumer keys from Twitter to bypass these limits: https://gist.github.com/shobotch/5160017
Seems like a good way to get your IP blacklisted
If you can't bypass an IP block then you need to git gud
Re: Reconstructing Twitter's Firehose
#28(Using figures cited in the article) Assuming that most of the time only 20 machines are responsible for ID generation, and ~50% of tweets use the first available sequence number for a given millisecond (so half the time machines only process one new tweet per millisecond), can we estimate Twitter's new tweet QPS average to be 20,000?
Edit: actually this is an upper bound for the 50% because it assumes that every machine processes a tweet every millisecond, which may not be the case
Re: Reconstructing Twitter's Firehose
#29Earlier quoted context omitted.
I can't imagine these will last long now that they've been posted in a comment on a front page HN story.
These are hardcoded in the apps. If they disable a key, they would need to release a new version of the app (ok), and all users would need to update (infeasible).
[deleted]
Re: Reconstructing Twitter's Firehose
#30If sequence ID means what I think it means... then if I reply to a twitter thread that already has 4095 replies... there could be the possibility of a tweet ID collision, possibly causing another tweet made simultaneously to mine to be unretrievable...
After years of waiting fruitlessly for Twitter to implement my feature request, it may finally be possible to delete someone else's tweet.