Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

github.com

Post reply on HN
Print view View original

51–60 of 62 posts

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#51

Post by fake-name »

charcircuit wrote: post #37
fake-name wrote: post #30

Earlier quoted context omitted.

Cloudflare is likely one of the worst things that has happened to the internet in recent history. Like, I get the need for some protective mechanisms for interactive content/posting/etc, but there should be zero cases where a simple HTTP 200 GET requires javascript/client side crap. If they serve me a slightly stale version of the remote resource (5 minutes/whatnot) that's fine. They've effectively just turned into a…

> If they serve me a slightly stale version of the remote resource (5 minutes/whatnot) that's fine. Not all sites are configured to do this. Some pages are expensive to render and have no cache layer.

I get that, my point is it's the problem.

They solve the DDOS issue by requiring JS captchas (which fundamentally breaks the way the internet should work), rather then serving a cache of the page to reduce load on the real host.

Requiring JS doesn't disambiguate between well behaved automated (or headless. I used a custom proxy for a lot of my content browsing) user agents and malicious users, it breaks /all/ of them.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#52

Post by account42 »

charcircuit wrote: post #37
fake-name wrote: post #30

Earlier quoted context omitted.

Cloudflare is likely one of the worst things that has happened to the internet in recent history. Like, I get the need for some protective mechanisms for interactive content/posting/etc, but there should be zero cases where a simple HTTP 200 GET requires javascript/client side crap. If they serve me a slightly stale version of the remote resource (5 minutes/whatnot) that's fine. They've effectively just turned into a…

> If they serve me a slightly stale version of the remote resource (5 minutes/whatnot) that's fine. Not all sites are configured to do this. Some pages are expensive to render and have no cache layer.

Some people shoot themselves in the foot, yes. There is no reason to not have some amount of microcaching even if it is very short and that puts an upper limit on the request rate per resource behind the caching layer.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#53

Post by raxxorrax »

caulk wrote: post #49
cma wrote: post #23

Earlier quoted context omitted.

I've noticed even GitHub has a login wall now for comments on open source projects. They truncate them if you aren't logged in, similar to reddit on mobile, instagram, twitter, etc. Hopefully the mobile version doesn't start pushing you to install some crappy apps where you can't use features like tabbed browsing, tab sync with another machine, etc.

The reasoning behind that might be the myriad of scrape-and-publish SEO spam pages with GitHub content.

Not sure if I am buying that excuse. I think they want to nudge people to make accounts and login. Really shady in case of Github and many other sides that are successful because of user content in my opinion.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#54

Post by lmz »

fake-name wrote: post #50
rtpg wrote: post #36

Earlier quoted context omitted.

Not to be too dismissive of this, but for companies trying to just run a service and getting constantly bombarded by stuff like DDoS issues, Cloudflare and its ilk lets them service a large portion of "legitimate" users, compared to none. I don't really know how you resolve that absent just like... putting everything behind logins, though.

What you're describing is solved by caching, not requiring JS crap. So give me xxx minute old stale cached version. I'm fine with that.

Until you get misbehaving users crawling every page.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#55

Post by userbinator »

raxxorrax wrote: post #53
caulk wrote: post #49

Earlier quoted context omitted.

The reasoning behind that might be the myriad of scrape-and-publish SEO spam pages with GitHub content.

Not sure if I am buying that excuse. I think they want to nudge people to make accounts and login. Really shady in case of Github and many other sides that are successful because of user content in my opinion.

Ironically, that makes the scraped copies more useful because they aren't truncated (at least for older pages) and I can actually get all the content. I wonder if that might be at least why Google seems to be giving them more weight in search result rankings.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#56

Post by sylware »

Currently, I cannot think about anything else other than "noscript/basic (x)html" /IRC to get us out of this, at least for sites where such protocols are "good enough" to provide their services to users over internet. But how? Enlighten the "javascript web" brain-washed devs to make them realize how toxic what they do is? regulations (at least for critical sites)? And how to deal with the other sites: those which devs are scammers and perfectly aware of how toxic they are and keep doing it.

In my own country, for critical sites, I will probably have go to court since 'noscript/basic (x)html" interop was broken in the last few years.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#57

Post by userbinator »

lmz wrote: post #54
fake-name wrote: post #50

Earlier quoted context omitted.

What you're describing is solved by caching, not requiring JS crap. So give me xxx minute old stale cached version. I'm fine with that.

Until you get misbehaving users crawling every page.

Rate limiting works, and isn't discriminatory against "non-standard" clients.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#58

Post by tootahe45 »

ricardo81 wrote: post #47
tootahe45 wrote: post #12

Would be cool if there was something like this for Python. Last time i tried to scrape something interesting i found that one of Cloudflare's enterprise options was easily blocking all of the main http libraries due to the identifiable TLS handshake.

Are you sure they blocked you because of the handshake? Always thought it was the myriad of cookies and expiry time of said cookies that tend to make non-browser clients more obvious to CF.

The site wasn't using it to block me, just to prompt a captcha, without doing so to 'real' browsers.

The HTTP requests were exact copies of browser requests (in terms of how the server would've seen them), so it was something below HTTP. I ended up finding a lot of info about Cloudflare and the TLS stuff on StackOverflow, with others having similar issues. Someone even made an API to do the TLS stuff as a service, but was too expensive for me. https://pixeljets.com/blog/scrape-ninja-bypassing-cloudflare...

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#59

Post by ricardo81 »

tootahe45 wrote: post #58
ricardo81 wrote: post #47

Earlier quoted context omitted.

Are you sure they blocked you because of the handshake? Always thought it was the myriad of cookies and expiry time of said cookies that tend to make non-browser clients more obvious to CF.

The site wasn't using it to block me, just to prompt a captcha, without doing so to 'real' browsers. The HTTP requests were exact copies of browser requests (in terms of how the server would've seen them), so it was something below HTTP. I ended up finding a lot of info about Cloudflare and the TLS stuff on StackOverflow, with others having similar issues. Someone even made an API to do the TLS stuff as a service, bu…

Thanks for the response, never came across the particular behaviour.

fwiw I think when it comes to the 'copy as curl', the HTTP header ordering may be different and it's worth loading up a page twice as some of the cookies are replaced.

I've used puppeteer as the article talks about. Manages the cookies better. Managed to do continuous requests without getting further CF blocks as opposed to a couple of hundred with cURL (due to cookies different from what CF expect over a time)

IIRC CF does have a sliding scale of how protected you want a site to be, so perhaps the TLS stuff belongs further up the scale.

Re: Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake

#60

Post by kevincox »

buro9 wrote: post #31
fake-name wrote: post #30

Earlier quoted context omitted.

Cloudflare is likely one of the worst things that has happened to the internet in recent history. Like, I get the need for some protective mechanisms for interactive content/posting/etc, but there should be zero cases where a simple HTTP 200 GET requires javascript/client side crap. If they serve me a slightly stale version of the remote resource (5 minutes/whatnot) that's fine. They've effectively just turned into a…

You can't turn it off as a Cloudflare customer either. The best you've got is "essentially off" but that wording is such because even with everything disabled there are still edge cases where their security will enforce a JS challenge or CAPTCHA.

At least on their basic plan there is also little to no indication of how often this is triggering. Leading to having know idea what the various settings are.
Post reply on HN