Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Curl-Impersonate

github.com

Post reply on HN
Print view View original

11–20 of 114 posts

Re: Curl-Impersonate

#11

Post by TekMol »

What is the use case? If you have to read data from one specific website which uses handshake info to avoid being read by software?

When I have to do HTTP requests these days, I default to a headless browser right away, because that seems to be the best bet. Even then, some website are not readable because they use captchas and whatnot.

Re: Curl-Impersonate

#13

Post by mschuster91 »

TekMol wrote: post #11

What is the use case? If you have to read data from one specific website which uses handshake info to avoid being read by software? When I have to do HTTP requests these days, I default to a headless browser right away, because that seems to be the best bet. Even then, some website are not readable because they use captchas and whatnot.

> What is the use case? If you have to read data from one specific website which uses handshake info to avoid being read by software?

Evade captchas. curl user agent / heuristics are blocked by many sites these days - I'd guess many popular CDNs have pre-defined "block bots" stuff that blocks everything automated that is not a well-known search engine indexer.

Re: Curl-Impersonate

#14

Post by throwaway99210 »

jollyllama wrote: post #12

>The Client Hello message that most HTTP clients and libraries produce differs drastically from that of a real browser. Why is this?

Based on what I've seen, most command-line clients and basic HTTP libraries typically ship with leaner, more static configurations (e.g., no GREASE extensions in the Client Hello, limited protocols in the ALPN extension header, smaller number of Signature Algorithms). Mirroring real browser TLS fingerprints is also more difficult due to the randomization of the Client Hello parameters (e.g., current versions of Chrome)

Re: Curl-Impersonate

#15

Post by Retr0id »

jollyllama wrote: post #12

>The Client Hello message that most HTTP clients and libraries produce differs drastically from that of a real browser. Why is this?

The protocols are flexible and most browsers bring their own HTTP+TLS clients

Re: Curl-Impersonate

#16

Post by cle »

The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi

I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyone is only using software approved by them, and to ensure that they can identify you. From multiple angles too (security, bots, DDoS, etc.), and it's not just limited to browsers either.

End goal seems to be: prove your identity to the megacorps so they can track everything you do and also ensure you are only doing things they approve of. I think the security arguments are just convenient rationalizations in service of this goal.

Re: Curl-Impersonate

#17

Post by zlagen »

jollyllama wrote: post #12

>The Client Hello message that most HTTP clients and libraries produce differs drastically from that of a real browser. Why is this?

They use different SSL libraries/configuration. Chrome uses BoringSSL and other libraries may use OpenSSL or some other library. Besides that the SSL library may be configured with different cipher suites and extensions. The solution these impersonators provide is to use the same SSL library and configuration as a real browser.

Re: Curl-Impersonate

#18

Post by oefrha »

What are some example sites where this is both necessary and sufficient? In my experience sites with serious anti-bot protection basically always have JavaScript-based browser detection, and some are capable of defeating puppeteer-extra-plugin-stealth even in headful mode. I doubt sites without serious anti-bot detection will do TLS fingerprinting. I guess it is useful for the narrower use case of getting a short-lived token/cookie with a headless browser on a heavily defended site, then performing requests using said tokens with this lightweight client for a while?

Re: Curl-Impersonate

#19

Post by throwaway99210 »

cle wrote: post #16

The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyo…

> I can't help but feel like these are the dying breaths of the open Internet though

I agree with the over zealous tracking by the megacorps but this is also due to bad actors, I work for a financial company and the amount of API abuse, ATO, DDoS, nefarious bot traffic, etc. we see on a daily basis is absolutely insane

Re: Curl-Impersonate

#20

Post by adastral »

TekMol wrote: post #11

What is the use case? If you have to read data from one specific website which uses handshake info to avoid being read by software? When I have to do HTTP requests these days, I default to a headless browser right away, because that seems to be the best bet. Even then, some website are not readable because they use captchas and whatnot.

> I default to a headless browser

Headless browsers consume orders of magnitude more resources, and execute far more requests (e.g. fetching images) than a common webscraping job would require. Having run webscraping at scale myself, the cost of operating headless browsers made us only use them as a last resort.

Post reply on HN