Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reconstructing Twitter's Firehose

docs.google.com

Post reply on HN
Print view View original

61–70 of 112 posts

Re: Reconstructing Twitter's Firehose

#61

Post by jaytaylor »

dev_dull wrote: post #53
jaytaylor wrote: post #41

Earlier quoted context omitted.

Hundreds of thousands per month. At least that was the price in 2012.

Not only that, but you need to prove you can handle the traffic.

Not in my experience. Why would gnip / twitter care if you can handle it, as long as you pay and abide by the terms of the contract? The burden to fetch the data within the specified window is on the client, it's not a push system.

For context: Back in the day, I worked at Klout. We had to pay a crippling monthly sum just to get access to the @mentions stream.

Re: Reconstructing Twitter's Firehose

#62

Post by mrits »

gberger wrote: post #14
throwawaymath wrote: post #13

Earlier quoted context omitted.

I can't imagine these will last long now that they've been posted in a comment on a front page HN story.

These are hardcoded in the apps. If they disable a key, they would need to release a new version of the app (ok), and all users would need to update (infeasible).

Wouldn't they just enforce the limit with those keys as well?

Re: Reconstructing Twitter's Firehose

#63

Post by jaytaylor »

aboutruby wrote: post #33
aboutruby wrote: post #7

That's quite amazing! One thing is that the data center ids are in the tweet ids, so it could be used to get a rough location of Twitter users.

I see there is interest in this observation so I used a little ruby (.to_s(2)[-22..-17].to_i(2)) to get the datacenter id. Then ran it on a few Twitter accounts: https://pastebin.com/w8Dnj5kM It does work, and it's going to be hard to patch edit: I realized you don't only get one location but the whole location history of a Twitter user. Also locating Twitter's data centers as it doesn't seem to be public information

They could "fix" it by periodically rotating the DC and server IDs.

I wish it was not so easily fixable, because this will break the key space reduction trick ;), but unfortunately such a solution is feasible and would come with the side-effect of drastically increasing the required scanning space.

Re: Reconstructing Twitter's Firehose

#64

Post by [deleted] »

cdoxsey wrote: post #6

If you need access to Twitter data you should sign up for a developer account and contact them. It's expensive but not prohibitively so for a company that needs Twitter data to do business. For academic research Twitter offers products which dramatically reduce the amount of data you need to consume via the full search API or historical powertrack.

[deleted]

Re: Reconstructing Twitter's Firehose

#65

Post by rokob »

stuck_in_matrix wrote: post #45
socketnaut wrote: post #42

The statistical claims in the article make the assumption that tweets are being sampled uniformly at random, which is most likely false. The fact that 3 machines handle 20% of tweets suggests that tweets are not in fact assigned to machines in a uniformly random manner. I would guess that there is a geographic bias as to which machines handle which tweets.

When I did the analysis, I was puzzled why certain machines handle a higher percentage of tweets compared to others -- so you are most likely correct that there may be some geographic consideration to the distribution. I'm rewriting the code to include a prescan of the time range to determine which server ids are in play at the time and which server ids are most active. Figuring out how to deconstruct Snowflake was c…

> Figuring out how to deconstruct Snowflake was challenging and there is still a lot of analysis left to do.

Why don't you just read the code?

Re: Reconstructing Twitter's Firehose

#68

Post by pas »

andy_ppp wrote: post #25

This is really interesting, an unforeseen insight into ids that can be enumerated even partially. I suppose encrypting or decrypting them is too costly so any other ideas for getting the properties of Snowflake at scale without this sort of attack being possible?

If you wanted to "encrypt" them, you could increase the key space and add a random salt to each one.

Re: Reconstructing Twitter's Firehose

#69

Post by pas »

daveguy wrote: post #38
oliveshell wrote: post #37

Earlier quoted context omitted.

Hence the distinction between “free as in speech” and “free as in beer”.

Just referring to the speech part... Is the speech free as in speech if it isn't free as in beer? If there is a gateway to prevent it from being freely distributed? With GNU the free as in speech is also free as in beer -- if all you want is the speech (aka code). If you want services around the code it will cost when companies are trying to make a business on it. Correct me if I'm wrong (and I probably am), but does…

GNU/FSF supports selling code for money. Twitter does the same with tweets.

You can download your own tweets for free, or even tweets of someone else freely, just doing it for all someone elses costs money.

So, with GNU usually you get both, but that's not necessary. And usually seen as completely meaningless, as if there would be a very useful GPL open source software that was not free to download and use, then a simple consortium of people could get together and buy one copy and then redistribute it for free. (Though they would need to do this for all future versions.) -- which is similar to what the author proposes with regards to forming a group of ~1 700 users/folks to brute force the firehose.

Post reply on HN