41–50 of 112 posts
> Twitter’s firehose is a complete stream of all tweets made on their platform and is only available to a few businesses at an extraordinary price. Anyone happen to know what that price is?
The fact that 3 machines handle 20% of tweets suggests that tweets are not in fact assigned to machines in a uniformly random manner. I would guess that there is a geographic bias as to which machines handle which tweets.
(Using figures cited in the article) Assuming that most of the time only 20 machines are responsible for ID generation, and ~50% of tweets use the first available sequence number for a given millisecond (so half the time machines only process one new tweet per millisecond), can we estimate Twitter's new tweet QPS average to be 20,000? Edit: actually this is an upper bound for the 50% because it assumes that every mac…
Someone suggested trying it when the New Year starts in Japan. Apparently there are tens of thousands of tweets per second then.
The statistical claims in the article make the assumption that tweets are being sampled uniformly at random, which is most likely false. The fact that 3 machines handle 20% of tweets suggests that tweets are not in fact assigned to machines in a uniformly random manner. I would guess that there is a geographic bias as to which machines handle which tweets.
I'm rewriting the code to include a prescan of the time range to determine which server ids are in play at the time and which server ids are most active.
Figuring out how to deconstruct Snowflake was challenging and there is still a lot of analysis left to do.
They used word "efficiently" and some Python code in the same paragraph. Having worked for quite a while with Python now, I'd say their notion of efficiency is different than mine.
They used word "efficiently" and some Python code in the same paragraph. Having worked for quite a while with Python now, I'd say their notion of efficiency is different than mine.
It probably is! When your bottleneck is a remote API limit, CPU usage might not be worth optimizing.
Also, when I provide code examples, I try and use a language that most programmers will have some exposure to and generally Python is high on that list.
I am the author of this document. If anyone has any questions, I'd be happy to answer them!
eg. 5d
Do not use, access or analyze the Twitter API to monitor or measure the availability, performance, functionality, usage statistics or results of Twitter Services or for any other benchmarking or competitive purposes, including without limitation, monitoring or measuring:
the responsiveness of Twitter Services; or
aggregate Twitter user metrics such as total number of active users, accounts, total number of Periscope Broadcast views, user engagements or account engagements.
Earlier quoted context omitted.
I see there is interest in this observation so I used a little ruby (.to_s(2)[-22..-17].to_i(2)) to get the datacenter id. Then ran it on a few Twitter accounts: https://pastebin.com/w8Dnj5kM It does work, and it's going to be hard to patch edit: I realized you don't only get one location but the whole location history of a Twitter user. Also locating Twitter's data centers as it doesn't seem to be public information
This is really interesting. When I did the original analysis on datacenter / server ids, I didn't think about correlation with user accounts. Nice observation!