Show HN: Twitter API Reverse Engineered

This was inevitably going to get more popular after the new API pricing was announced. Unfortunately this also means they're going to start being more aggressive on bot detection.

We may soon find ourselves needing to run headless browsers to scrape data, like with TikTok: https://nullpt.rs/reverse-engineering-tiktok-vm-1

Re: Show HN: Twitter API Reverse Engineered

#5

Post by banditelol » Thu, Apr 13, 2023, 1:32 AM UTC

I noticed it uses fixed Auth header, is it from your session or consistent across session/user?

Re: Show HN: Twitter API Reverse Engineered

#6

Post by wolfgang42 » Thu, Apr 13, 2023, 1:43 AM UTC

banditelol wrote: post #5

I noticed it uses fixed Auth header, is it from your session or consistent across session/user?

For some reason Twitter’s frontend uses a hard-coded bearer token, at least for anonymous users. You’ll see exactly the same string if you load a Twitter page and look at the XHR requests in your own browser. (It seems to change occasionally, but old ones keep working in my experience.)

Re: Show HN: Twitter API Reverse Engineered

#7

Post by hobenshield » Thu, Apr 13, 2023, 1:58 AM UTC

TobyTheDog123 wrote: post #4

This was inevitably going to get more popular after the new API pricing was announced. Unfortunately this also means they're going to start being more aggressive on bot detection. We may soon find ourselves needing to run headless browsers to scrape data, like with TikTok: https://nullpt.rs/reverse-engineering-tiktok-vm-1

[flagged]

Re: Show HN: Twitter API Reverse Engineered

#8

Post by hobenshield » Thu, Apr 13, 2023, 2:02 AM UTC

simonw wrote: post #3

Huh, TIL Twitter uses GraphQL under the hood: https://github.com/trevorhobenshield/twitter-api-client/blob...

https://blog.twitter.com/engineering/en_us/topics/infrastruc...

Re: Show HN: Twitter API Reverse Engineered

#9

Post by djbusby » Thu, Apr 13, 2023, 2:39 AM UTC

TobyTheDog123 wrote: post #4

This was inevitably going to get more popular after the new API pricing was announced. Unfortunately this also means they're going to start being more aggressive on bot detection. We may soon find ourselves needing to run headless browsers to scrape data, like with TikTok: https://nullpt.rs/reverse-engineering-tiktok-vm-1

These puppeteer, playwright wrappers make bots so easy. And you can distribute so easy too. Back in my day (IE6) it wasn't quite so simple. Trying to make your site undcrape-able while still readable is (still) impossible.

Re: Show HN: Twitter API Reverse Engineered

#10

Post by jaggederest » Thu, Apr 13, 2023, 3:33 AM UTC

wolfgang42 wrote: post #6

banditelol wrote: post #5

I noticed it uses fixed Auth header, is it from your session or consistent across session/user?

For some reason Twitter’s frontend uses a hard-coded bearer token, at least for anonymous users. You’ll see exactly the same string if you load a Twitter page and look at the XHR requests in your own browser. (It seems to change occasionally, but old ones keep working in my experience.)

I do this a lot, good ol' 0xDEADBEEF makes it easier to track whether the header is actually missing (eg misconfigured) or just undefined but coming through correctly.