81–90 of 112 posts
Wonder how it fits with developer terms. eg. 5d Do not use, access or analyze the Twitter API to monitor or measure the availability, performance, functionality, usage statistics or results of Twitter Services or for any other benchmarking or competitive purposes, including without limitation, monitoring or measuring: the responsiveness of Twitter Services; or aggregate Twitter user metrics such as total number of ac…
To store such a firehose stream of data, you will need approximately 0.3gb of storage per one second of data. This is if you only collect username, timestamp and tweet, excluding any additional metadata such as data center, likes and retweets, not to mention images and videos. Full calculation here: https://docs.google.com/spreadsheets/d/1BIAguT9Qvy0GK-dalpQf...
To store such a firehose stream of data, you will need approximately 0.3gb of storage per one second of data. This is if you only collect username, timestamp and tweet, excluding any additional metadata such as data center, likes and retweets, not to mention images and videos. Full calculation here: https://docs.google.com/spreadsheets/d/1BIAguT9Qvy0GK-dalpQf...
Since a couple of years ago the tweet limit was bumped to 280 characters, so I assume the amount is double of what you calculated.
If Twitter is already storing the tweet timestamp, is there a reason to generate sequential IDs? It seems like they could move to a UUID scheme and protect their "firehose" from reverse-engineering.
They're snowflake IDs, not exactly sequential. Timestamp bits, shifted all the way left Data center id shifted left Server id shifted left Increment this millisecond Since the left bits are the timestamp, they are date/time sortable and a much more useful index/primary key/whatever than a UUID.
Earlier quoted context omitted.
Hence the distinction between “free as in speech” and “free as in beer”.
Just referring to the speech part... Is the speech free as in speech if it isn't free as in beer? If there is a gateway to prevent it from being freely distributed? With GNU the free as in speech is also free as in beer -- if all you want is the speech (aka code). If you want services around the code it will cost when companies are trying to make a business on it. Correct me if I'm wrong (and I probably am), but does…
Earlier quoted context omitted.
Not only that, but you need to prove you can handle the traffic.
Not in my experience. Why would gnip / twitter care if you can handle it, as long as you pay and abide by the terms of the contract? The burden to fetch the data within the specified window is on the client, it's not a push system. For context: Back in the day, I worked at Klout. We had to pay a crippling monthly sum just to get access to the @mentions stream.
They don't specifically say you have to verify anything, but they do dance around being able to handle the volume in a couple of different ways in their streaming guides for us average mortals [1]. It mostly seems to come down to the fact that they are, at least to some degree, buffering on their end to make sure you don't miss anything if there's suddenly a spike that's too large for you to handle or if your connection degrades for a while, etc.
It also wouldn't surprise me if it were originally simply a limitation of their internal systems and they couldn't buffer or allow any kind of replay because they were literally just writing everything off to a stream as it came in and didn't have any mechanism for going back and reading from the actual database after the fact.
1: https://developer.twitter.com/en/docs/tweets/filter-realtime...
How can Twitter be a free speech platform if you cant receive the speech for free?
>Twitter’s statuses lookup API endpoint allows for a total of 1,200 API calls every 15 minutes. Each call allows the user to pass 100 ids for a total of 120,000 id requests every 15 minutes using both APP auth [...] Use the secret consumer keys from Twitter to bypass these limits: https://gist.github.com/shobotch/5160017
As a developer, I have no understanding of how is data monetized. How much is user data worth? What kind of user data is worth more than others? What is the data used for (why would anyone pay "extraordinary price" for tweets?). Who pays for user data - is there some public market that I am not aware of?