Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

github.com

Post reply on HN
Print view View original

31–40 of 62 posts

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#31

Post by buro9 »

fake-name wrote: post #30
userbinator wrote: post #14

I run a MITM proxy for adblocking/general filtering and within the past little while I've noticed CloudFlare and other "bot protection" tends to get me blocked out of increasingly more sites I come across in search results, so this will be very useful for fixing that. However, I should caution that in this era of companies being particularly user-hostile and authoritarian, especially Big Tech, I would be more careful…

Cloudflare is likely one of the worst things that has happened to the internet in recent history. Like, I get the need for some protective mechanisms for interactive content/posting/etc, but there should be zero cases where a simple HTTP 200 GET requires javascript/client side crap. If they serve me a slightly stale version of the remote resource (5 minutes/whatnot) that's fine. They've effectively just turned into a…

You can't turn it off as a Cloudflare customer either.

The best you've got is "essentially off" but that wording is such because even with everything disabled there are still edge cases where their security will enforce a JS challenge or CAPTCHA.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#32

Post by bburky »

lwthiker wrote: post #29
pabs3 wrote: post #18

Do you plan on getting this merged back into curl with an option to enable it? I can see that being useful for some people.

I hope to do so in the future, for now the implementation is extremely hacky so I doubt it can get accepted into curl.

There was a conversation on their mailing list contemplating dropping NSS support. https://curl.se/mail/lib-2022-01/0120.html If you have a use case for NSS in curl, you may want to speak up. Perhaps "I want curl to look exactly like a browser" is a significant use case?

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#33

Post by flawi »

7v3x3n3sem9vv wrote: post #19
jandrese wrote: post #17

Given the relative market shares it might make more sense to impersonate Chrome.

and make it seem like Firefox has less market share? sounds like a good way to kill Firefox even faster, my 2 cents.

Counter argument is service providers just choosing to block anything that looks like Firefox since the market share is so small and it's being used to circumvent their precious protections.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#34

Post by 1vuio0pswjnm7 »

"Some web services therefore use the TLS handshake to fingerprint which HTTP client is accessing them. Notably, some bot protection platforms use this to identify curl and block it."

As a user of non-browser clients (not curl though) I have not run into this in the wild.^1

Anyone have an example of a site that blocks non-browser clients based on TLS fingerprint.

1. As far as I know. The only site I know of today that is blocking non-browser clients appears to be www.startpage.com. Perhaps this is the heuristic they are using. More likely it is something simpler I have not figured out yet.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#35

Post by jve »

kalleboo wrote: post #15
gruez wrote: post #4

>impersonate Firefox 95 you should really be impersonating an ESR version (eg. 91). Versions from the release channel is updated every month or so, and everyone has autoupdate enabled. Therefore unless you keep it up to date, your fingerprint is going to stick out like a sore thumb in a few months. On the other hand, ESR sticks to one version and shouldn't change significantly during its one year lifetime. It's still…

They should really be impersonating Chrome. If this takes off, Firefox has such a small user share that I could see sites just banning Firefox altogether, like they do with Tor

I suspect Tor is being banned not because of a small user share.

Perhaps you may get broken sites with Firefox, because no-one cared. But banning? Seems like a stretch.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#36

Post by rtpg »

fake-name wrote: post #30
userbinator wrote: post #14

I run a MITM proxy for adblocking/general filtering and within the past little while I've noticed CloudFlare and other "bot protection" tends to get me blocked out of increasingly more sites I come across in search results, so this will be very useful for fixing that. However, I should caution that in this era of companies being particularly user-hostile and authoritarian, especially Big Tech, I would be more careful…

Cloudflare is likely one of the worst things that has happened to the internet in recent history. Like, I get the need for some protective mechanisms for interactive content/posting/etc, but there should be zero cases where a simple HTTP 200 GET requires javascript/client side crap. If they serve me a slightly stale version of the remote resource (5 minutes/whatnot) that's fine. They've effectively just turned into a…

Not to be too dismissive of this, but for companies trying to just run a service and getting constantly bombarded by stuff like DDoS issues, Cloudflare and its ilk lets them service a large portion of "legitimate" users, compared to none.

I don't really know how you resolve that absent just like... putting everything behind logins, though.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#37

Post by charcircuit »

fake-name wrote: post #30
userbinator wrote: post #14

I run a MITM proxy for adblocking/general filtering and within the past little while I've noticed CloudFlare and other "bot protection" tends to get me blocked out of increasingly more sites I come across in search results, so this will be very useful for fixing that. However, I should caution that in this era of companies being particularly user-hostile and authoritarian, especially Big Tech, I would be more careful…

Cloudflare is likely one of the worst things that has happened to the internet in recent history. Like, I get the need for some protective mechanisms for interactive content/posting/etc, but there should be zero cases where a simple HTTP 200 GET requires javascript/client side crap. If they serve me a slightly stale version of the remote resource (5 minutes/whatnot) that's fine. They've effectively just turned into a…

> If they serve me a slightly stale version of the remote resource (5 minutes/whatnot) that's fine.

Not all sites are configured to do this. Some pages are expensive to render and have no cache layer.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#38

Post by [deleted] »

1vuio0pswjnm7 wrote: post #34

"Some web services therefore use the TLS handshake to fingerprint which HTTP client is accessing them. Notably, some bot protection platforms use this to identify curl and block it." As a user of non-browser clients (not curl though) I have not run into this in the wild.^1 Anyone have an example of a site that blocks non-browser clients based on TLS fingerprint. 1. As far as I know. The only site I know of today that…

[deleted]

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#39

Post by kalleboo »

jve wrote: post #35
kalleboo wrote: post #15

Earlier quoted context omitted.

They should really be impersonating Chrome. If this takes off, Firefox has such a small user share that I could see sites just banning Firefox altogether, like they do with Tor

I suspect Tor is being banned not because of a small user share. Perhaps you may get broken sites with Firefox, because no-one cared. But banning? Seems like a stretch.

Tor is banned (or rather, tar-pitted in endless CAPTCHAs) because the amount of legit users is massively dwarfed by the abuse

If everyone running scrapers and attack probes start showing up as Firefox, then they'll end up in the same situation.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#40

Post by oefrha »

jve wrote: post #35
kalleboo wrote: post #15

Earlier quoted context omitted.

They should really be impersonating Chrome. If this takes off, Firefox has such a small user share that I could see sites just banning Firefox altogether, like they do with Tor

I suspect Tor is being banned not because of a small user share. Perhaps you may get broken sites with Firefox, because no-one cared. But banning? Seems like a stretch.

If there are a lot of abuse masquerading as Firefox, outstripping legit users, they can totally throw up a CAPTCHA for Firefox but not for Chrome. An outright ban isn’t the only annoying outcome.
Post reply on HN