[flagged]
What's that link?
It's a spambot
Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake
41–50 of 62 posts
Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake
#42I run a MITM proxy for adblocking/general filtering and within the past little while I've noticed CloudFlare and other "bot protection" tends to get me blocked out of increasingly more sites I come across in search results, so this will be very useful for fixing that. However, I should caution that in this era of companies being particularly user-hostile and authoritarian, especially Big Tech, I would be more careful…
Maybe I'm just a techno-optimist, but I suspect big tech companies don't give a hoot about you running "unapproved" software, but rather care about their services being abused and "unapproved" software is just a useful signal that fails on a tiny percentage of total legit users.
Just trying to keep services operational is a fine goal to pursue as an operator, but forcing users to small inbound funnels for the service is detrimental too. There needs to be better research to be done to allow simpler ways of operation to continue working.
A browser is becoming a universal agent by itself, but many people (maybe increasingly) use terminal to access to the resources, and stonewalling these paths are never OK in my book.
Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake
#43Earlier quoted context omitted.
I hope to do so in the future, for now the implementation is extremely hacky so I doubt it can get accepted into curl.
There was a conversation on their mailing list contemplating dropping NSS support. https://curl.se/mail/lib-2022-01/0120.html If you have a use case for NSS in curl, you may want to speak up. Perhaps "I want curl to look exactly like a browser" is a significant use case?
Agreed, it is very important to bring this up on the mailing list. It might also be plausible to make curl look like Chrome if curl had BoringSSL support.
Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake
#44Cool, can't wait for anti-bot protection to start rejecting me because I use firefox.
Only a matter of time I'm afraid :( Firefox usage share is already low enough for many sites to make pages for Chrome and maybe Safari only.
Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake
#45I run a MITM proxy for adblocking/general filtering and within the past little while I've noticed CloudFlare and other "bot protection" tends to get me blocked out of increasingly more sites I come across in search results, so this will be very useful for fixing that. However, I should caution that in this era of companies being particularly user-hostile and authoritarian, especially Big Tech, I would be more careful…
> profiling users based on other traits, and essentially determining if they are using "approved" software, is a dystopia we should fight strongly against. Stallman's Right To Read comes to mind as a very relevant warning story.
Right to Read indeed... fanfiction.net has over the last months become really annoying. Especially at night, when you have the FFN UI set to dark, and then out of nothing a bright white Cloudflare page appears. Or why the Cloudflare "anti bot" protection leads to an endless loop when the browser is the Android web view inside a third-party Reddit client.
Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake
#46This is cool, but is it really needed that often?
There are some industries (virtually all of Wall Street, for example, and certain parts of government) where the company needs to surveil 100% of what their employees do on the web from inside the office. These companies have been running MITM proxies for decades.
Wouldn't any website that rejects a non-browsery TLS client be blocking out these people as well?
Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake
#47Would be cool if there was something like this for Python. Last time i tried to scrape something interesting i found that one of Cloudflare's enterprise options was easily blocking all of the main http libraries due to the identifiable TLS handshake.
Are you sure they blocked you because of the handshake?
Always thought it was the myriad of cookies and expiry time of said cookies that tend to make non-browser clients more obvious to CF.
Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake
#48This is cool, but is it really needed that often? There are some industries (virtually all of Wall Street, for example, and certain parts of government) where the company needs to surveil 100% of what their employees do on the web from inside the office. These companies have been running MITM proxies for decades. Wouldn't any website that rejects a non-browsery TLS client be blocking out these people as well?
They don't block you completely, just present you with a JS challenge that delays your access to the site. A browser, even if behind a MITM proxy, would be able to solve this challenge.
Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake
#49I run a MITM proxy for adblocking/general filtering and within the past little while I've noticed CloudFlare and other "bot protection" tends to get me blocked out of increasingly more sites I come across in search results, so this will be very useful for fixing that. However, I should caution that in this era of companies being particularly user-hostile and authoritarian, especially Big Tech, I would be more careful…
I've noticed even GitHub has a login wall now for comments on open source projects. They truncate them if you aren't logged in, similar to reddit on mobile, instagram, twitter, etc. Hopefully the mobile version doesn't start pushing you to install some crappy apps where you can't use features like tabbed browsing, tab sync with another machine, etc.
The reasoning behind that might be the myriad of scrape-and-publish SEO spam pages with GitHub content.
Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake
#50Earlier quoted context omitted.
Cloudflare is likely one of the worst things that has happened to the internet in recent history. Like, I get the need for some protective mechanisms for interactive content/posting/etc, but there should be zero cases where a simple HTTP 200 GET requires javascript/client side crap. If they serve me a slightly stale version of the remote resource (5 minutes/whatnot) that's fine. They've effectively just turned into a…
Not to be too dismissive of this, but for companies trying to just run a service and getting constantly bombarded by stuff like DDoS issues, Cloudflare and its ilk lets them service a large portion of "legitimate" users, compared to none. I don't really know how you resolve that absent just like... putting everything behind logins, though.
What you're describing is solved by caching, not requiring JS crap.
So give me xxx minute old stale cached version. I'm fine with that.