Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reconstructing Twitter's Firehose

docs.google.com

Post reply on HN
Print view View original

91–100 of 112 posts

Re: Reconstructing Twitter's Firehose

#91

Post by dewey »

dandare wrote: post #90

>Twitter does sell premium data services including their much coveted “firehose” stream. Twitter’s firehose is a complete stream of all tweets made on their platform and is only available to a few businesses at an extraordinary price. As a developer, I have no understanding of how is data monetized. How much is user data worth? What kind of user data is worth more than others? What is the data used for (why would any…

I’d guess adtech, market research and investment companies would be interested.

Re: Reconstructing Twitter's Firehose

#92

Post by keithwinstein »

ohadron wrote: post #78

To store such a firehose stream of data, you will need approximately 0.3gb of storage per one second of data. This is if you only collect username, timestamp and tweet, excluding any additional metadata such as data center, likes and retweets, not to mention images and videos. Full calculation here: https://docs.google.com/spreadsheets/d/1BIAguT9Qvy0GK-dalpQf...

I believe your calculations are too high by a factor of 1000... 53 bytes/(typical tweet) * 6000 typical tweets/second is 318 KB (or 0.3 MB) per second, not 318 MB.

Re: Reconstructing Twitter's Firehose

#93

Post by notamy »

HelloFellowDevs wrote: post #74

Discord also utilizes Twitters Snowflake algorithm for the ridiculous amount of messages that are sent in chats.[1] [1] https://discordapp.com/developers/docs/reference/

Not only messages - EVERYTHING in Discord is identified by a snowflake - messages, users, channels, guilds (servers), bots/applications, ...

Re: Reconstructing Twitter's Firehose

#94

Post by jjeaff »

metildaa wrote: post #79
mrits wrote: post #62

Earlier quoted context omitted.

Wouldn't they just enforce the limit with those keys as well?

Sounds like a solid reason to not build around a platform that can arbitrarily kill your project or business. ActivityPub seems to be gaining traction, w/o the beholden to a single entity issue.

A platform that can arbitrarily take down your business like a mobile app on iPhone or Android?

Re: Reconstructing Twitter's Firehose

#95

Post by throwfaraway113 »

dandare wrote: post #90

>Twitter does sell premium data services including their much coveted “firehose” stream. Twitter’s firehose is a complete stream of all tweets made on their platform and is only available to a few businesses at an extraordinary price. As a developer, I have no understanding of how is data monetized. How much is user data worth? What kind of user data is worth more than others? What is the data used for (why would any…

Firehose access is prohibitively expensive. Last I heard the cost was based on something like " 30% your company's annual revenue " (this is some sort of second hand rumour I heard when I asked).

Services like gnip have everything from the firehose (ever) and it can be retrieved by paying a monthly fee (hundreds, thousand of $ for 500k twwets)

Re: Reconstructing Twitter's Firehose

#96

Post by [deleted] »

mttpgn wrote: post #30

If sequence ID means what I think it means... then if I reply to a twitter thread that already has 4095 replies... there could be the possibility of a tweet ID collision, possibly causing another tweet made simultaneously to mine to be unretrievable... After years of waiting fruitlessly for Twitter to implement my feature request, it may finally be possible to delete someone else's tweet.

[deleted]

Re: Reconstructing Twitter's Firehose

#97

Post by detaro »

rokob wrote: post #65
stuck_in_matrix wrote: post #45

Earlier quoted context omitted.

When I did the analysis, I was puzzled why certain machines handle a higher percentage of tweets compared to others -- so you are most likely correct that there may be some geographic consideration to the distribution. I'm rewriting the code to include a prescan of the time range to determine which server ids are in play at the time and which server ids are most active. Figuring out how to deconstruct Snowflake was c…

> Figuring out how to deconstruct Snowflake was challenging and there is still a lot of analysis left to do. Why don't you just read the code?

How do you read the code of an implementation detail of Twitter's servers? There's no guarantee that the example code they released years ago still matches what they use.

Re: Reconstructing Twitter's Firehose

#98

Post by adrianN »

thsowers wrote: post #26
nspkr wrote: post #4

>Twitter’s statuses lookup API endpoint allows for a total of 1,200 API calls every 15 minutes. Each call allows the user to pass 100 ids for a total of 120,000 id requests every 15 minutes using both APP auth [...] Use the secret consumer keys from Twitter to bypass these limits: https://gist.github.com/shobotch/5160017

Seems like a good way to get your IP blacklisted

Seems like a good way to get an angry e-mail from your ISP, or Twitter's lawyer.

Re: Reconstructing Twitter's Firehose

#99

Post by mttpgn »

marcinzm wrote: post #31
mttpgn wrote: post #30

If sequence ID means what I think it means... then if I reply to a twitter thread that already has 4095 replies... there could be the possibility of a tweet ID collision, possibly causing another tweet made simultaneously to mine to be unretrievable... After years of waiting fruitlessly for Twitter to implement my feature request, it may finally be possible to delete someone else's tweet.

The id of tweets includes the time stamp. The sequence is just an incremental number per server per millisecond. As far as I can see it’s not based on retweets.

Thanks for the quick explainer on what the sequence is. Reassuring to know it's not the ordered number of tweets in a thread.

Re: Reconstructing Twitter's Firehose

#100

Post by andreareina »

HelloFellowDevs wrote: post #74

Discord also utilizes Twitters Snowflake algorithm for the ridiculous amount of messages that are sent in chats.[1] [1] https://discordapp.com/developers/docs/reference/

Link doesn't work for me, I just get a partially-loaded page with an eternally-spinning spinner (FF65 on OSX).

Gotta go to https://discordapp.com/developers/docs and then click on the Reference link in the sidebar.

Post reply on HN