Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reconstructing Twitter's Firehose

docs.google.com

Post reply on HN
Print view View original

31–40 of 112 posts

Re: Reconstructing Twitter's Firehose

#31

Post by marcinzm »

mttpgn wrote: post #30

If sequence ID means what I think it means... then if I reply to a twitter thread that already has 4095 replies... there could be the possibility of a tweet ID collision, possibly causing another tweet made simultaneously to mine to be unretrievable... After years of waiting fruitlessly for Twitter to implement my feature request, it may finally be possible to delete someone else's tweet.

The id of tweets includes the time stamp. The sequence is just an incremental number per server per millisecond. As far as I can see it’s not based on retweets.

Re: Reconstructing Twitter's Firehose

#32

Post by mhfs »

thsowers wrote: post #26
nspkr wrote: post #4

>Twitter’s statuses lookup API endpoint allows for a total of 1,200 API calls every 15 minutes. Each call allows the user to pass 100 ids for a total of 120,000 id requests every 15 minutes using both APP auth [...] Use the secret consumer keys from Twitter to bypass these limits: https://gist.github.com/shobotch/5160017

Seems like a good way to get your IP blacklisted

I worked with a company in the past that abused the twitter api to an unimaginable level from a single IP address.

That was a few years ago but at the time I’m pretty sure their blacklisting was between inexistent and pathetic.

Re: Reconstructing Twitter's Firehose

#33

Post by aboutruby »

aboutruby wrote: post #7

That's quite amazing! One thing is that the data center ids are in the tweet ids, so it could be used to get a rough location of Twitter users.

I see there is interest in this observation so I used a little ruby (.to_s(2)[-22..-17].to_i(2)) to get the datacenter id.

Then ran it on a few Twitter accounts: https://pastebin.com/w8Dnj5kM

It does work, and it's going to be hard to patch

edit: I realized you don't only get one location but the whole location history of a Twitter user. Also locating Twitter's data centers as it doesn't seem to be public information

Re: Reconstructing Twitter's Firehose

#34

Post by tomglynch »

mhfs wrote: post #32
thsowers wrote: post #26

Earlier quoted context omitted.

Seems like a good way to get your IP blacklisted

I worked with a company in the past that abused the twitter api to an unimaginable level from a single IP address. That was a few years ago but at the time I’m pretty sure their blacklisting was between inexistent and pathetic.

Using these keys?

Re: Reconstructing Twitter's Firehose

#35

Post by mhfs »

tomglynch wrote: post #34
mhfs wrote: post #32

Earlier quoted context omitted.

I worked with a company in the past that abused the twitter api to an unimaginable level from a single IP address. That was a few years ago but at the time I’m pretty sure their blacklisting was between inexistent and pathetic.

Using these keys?

No. Different strategy.

Re: Reconstructing Twitter's Firehose

#38

Post by daveguy »

oliveshell wrote: post #37
thoughtstheseus wrote: post #36

How can Twitter be a free speech platform if you cant receive the speech for free?

Hence the distinction between “free as in speech” and “free as in beer”.

Just referring to the speech part... Is the speech free as in speech if it isn't free as in beer? If there is a gateway to prevent it from being freely distributed? With GNU the free as in speech is also free as in beer -- if all you want is the speech (aka code). If you want services around the code it will cost when companies are trying to make a business on it. Correct me if I'm wrong (and I probably am), but doesn't every additional dollar reduce free as in speech because of reduced access?

Re: Reconstructing Twitter's Firehose

#40

Post by stuck_in_matrix »

aboutruby wrote: post #33
aboutruby wrote: post #7

That's quite amazing! One thing is that the data center ids are in the tweet ids, so it could be used to get a rough location of Twitter users.

I see there is interest in this observation so I used a little ruby (.to_s(2)[-22..-17].to_i(2)) to get the datacenter id. Then ran it on a few Twitter accounts: https://pastebin.com/w8Dnj5kM It does work, and it's going to be hard to patch edit: I realized you don't only get one location but the whole location history of a Twitter user. Also locating Twitter's data centers as it doesn't seem to be public information

This is really interesting. When I did the original analysis on datacenter / server ids, I didn't think about correlation with user accounts. Nice observation!
Post reply on HN