Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

github.com

Post reply on HN
Print view View original

11–20 of 62 posts

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#11

Post by tenebrisalietum »

javajosh wrote: post #10

Handy. Is the TCP handshake, or other details about socket behavior, ever get used for assessing the remote process, and in turn libraries written to mimic known patterns?

Yes. https://nmap.org/book/osdetect-methods.html

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#12

Post by tootahe45 »

Would be cool if there was something like this for Python. Last time i tried to scrape something interesting i found that one of Cloudflare's enterprise options was easily blocking all of the main http libraries due to the identifiable TLS handshake.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#13

Post by tyingq »

tootahe45 wrote: post #12

Would be cool if there was something like this for Python. Last time i tried to scrape something interesting i found that one of Cloudflare's enterprise options was easily blocking all of the main http libraries due to the identifiable TLS handshake.

I think most of the scraping libraries have stagnated since it's hard to scrape without a headless browser these days...too many sites with client-side rendered content.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#14

Post by userbinator »

I run a MITM proxy for adblocking/general filtering and within the past little while I've noticed CloudFlare and other "bot protection" tends to get me blocked out of increasingly more sites I come across in search results, so this will be very useful for fixing that.

However, I should caution that in this era of companies being particularly user-hostile and authoritarian, especially Big Tech, I would be more careful with sharing stuff like this. Being forced to run JS is bad enough; profiling users based on other traits, and essentially determining if they are using "approved" software, is a dystopia we should fight strongly against. Stallman's Right To Read comes to mind as a very relevant warning story.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#15

Post by kalleboo »

gruez wrote: post #4

>impersonate Firefox 95 you should really be impersonating an ESR version (eg. 91). Versions from the release channel is updated every month or so, and everyone has autoupdate enabled. Therefore unless you keep it up to date, your fingerprint is going to stick out like a sore thumb in a few months. On the other hand, ESR sticks to one version and shouldn't change significantly during its one year lifetime. It's still…

They should really be impersonating Chrome. If this takes off, Firefox has such a small user share that I could see sites just banning Firefox altogether, like they do with Tor

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#16

Post by jart »

oefrha wrote: post #9
wswope wrote: post #2

Very cool! Thanks for sharing - it’s always nice to learn about fingerprinting tricks and workarounds, from both a privacy and a “don’t unintentionally look like a bot” perspective. What inspired the project?

Motivation is in the blog post: https://lwthiker.com/reversing/2022/02/17/curl-impersonate-f...

Good blog post. Stuff like this makes me wonder if by 2030 (1) the internet will mostly consist of machine generated content; (2) machines written by normal people in Python won't be authorized to access the machine-generated content anymore due to Protectify; (3) most client traffic will originate from Protectify's network, so people like bloggers won't have any visibility into whether their readers are humans or machines; (4) video compression algorithms will become indistinguishable from deepfakes; and (5) airborne pathogens will make alternatives to the above impractical.
Post reply on HN