71–80 of 112 posts
Earlier quoted context omitted.
I'm wondering if the Wayback Machine has access to it.
I would be extremely interested in getting the Twitter corpus ingested into the Internet Archive. To my knowledge, the Archive does not have access to the firehose.
(Using figures cited in the article) Assuming that most of the time only 20 machines are responsible for ID generation, and ~50% of tweets use the first available sequence number for a given millisecond (so half the time machines only process one new tweet per millisecond), can we estimate Twitter's new tweet QPS average to be 20,000? Edit: actually this is an upper bound for the 50% because it assumes that every mac…
If Twitter is already storing the tweet timestamp, is there a reason to generate sequential IDs? It seems like they could move to a UUID scheme and protect their "firehose" from reverse-engineering.
Earlier quoted context omitted.
I worked with a company in the past that abused the twitter api to an unimaginable level from a single IP address. That was a few years ago but at the time I’m pretty sure their blacklisting was between inexistent and pathetic.
It’s sad that you think this somehow reflects badly on Twitter. I appreciate a company that will opt to be conservative rather than ban-hammer innocent people on accident just to stop a single idiot.
Off-topic but reminds me of ~2004 where we had a particularly pervasive cheater in our dedicated game server and the end decision of the admin was to just ban the entire IP range of said cheaters' ISP. Not particularly conservative - very effective.
That's quite amazing! One thing is that the data center ids are in the tweet ids, so it could be used to get a rough location of Twitter users.
I see there is interest in this observation so I used a little ruby (.to_s(2)[-22..-17].to_i(2)) to get the datacenter id. Then ran it on a few Twitter accounts: https://pastebin.com/w8Dnj5kM It does work, and it's going to be hard to patch edit: I realized you don't only get one location but the whole location history of a Twitter user. Also locating Twitter's data centers as it doesn't seem to be public information
This is if you only collect username, timestamp and tweet, excluding any additional metadata such as data center, likes and retweets, not to mention images and videos.
Full calculation here: https://docs.google.com/spreadsheets/d/1BIAguT9Qvy0GK-dalpQf...
Earlier quoted context omitted.
These are hardcoded in the apps. If they disable a key, they would need to release a new version of the app (ok), and all users would need to update (infeasible).
Wouldn't they just enforce the limit with those keys as well?
>Twitter’s statuses lookup API endpoint allows for a total of 1,200 API calls every 15 minutes. Each call allows the user to pass 100 ids for a total of 120,000 id requests every 15 minutes using both APP auth [...] Use the secret consumer keys from Twitter to bypass these limits: https://gist.github.com/shobotch/5160017
You can do the same for the Reddit API, change the agent header and then you have unlimited API calls.