Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Blocked by Cloudflare

jrhawley.ca

Post reply on HN
Print view View original

61–70 of 473 posts

Re: Blocked by Cloudflare

#61

Post by NicoJuicy »

everybodyknows wrote: post #32
NicoJuicy wrote: post #14

Earlier quoted context omitted.

Everyone forgets that websites become almost unusable because of crawlers and bots. Website owners specifically choose for cloudflare to protect against this, it's not forced upon them by cloudflare.

Website owner complains elsewhere in this topic of blocked users (country-specific): https://news.ycombinator.com/item?id=37050774

That's not a lot of information for such a complex subject.

Eg. If you live in a dictatorship and use a VPN. You're traffic is together with a lot of people.

The website owner can disable cloudflare their checks and that will leave their site unprotected. The choice of that is up to the website owner, no?

Re: Blocked by Cloudflare

#62

Post by NicoJuicy »

delfinom wrote: post #24
NicoJuicy wrote: post #14

Earlier quoted context omitted.

Everyone forgets that websites become almost unusable because of crawlers and bots. Website owners specifically choose for cloudflare to protect against this, it's not forced upon them by cloudflare.

Hah, you know, most crawlers were fine. The only one that actively DDOSed websites was fucking Yandex. It doesn't respect robots.txt and it will actively fight against any rate limits by spawning connections on new IPs the moment one is blocked

? I personally experienced many unbehaving crawlers/bots not respecting robots.txt and behaving like another user agent.

Feel free to prove me wrong and disrupt cloudflare by only handling that use-case

Re: Blocked by Cloudflare

#63

Post by adrr »

zer8k wrote: post #12

Any time a large portion of internet traffic is controlled by a single source it brings problems like this with it. All cloudflare has to do is arbitrarily decide who and who can't use the internet and effectively their word becomes law. Like most things it starts with an innocent premise (e.g. "an easy way to stop bad actors") and ends up extended to any number of arbitrary things. Worse, the argument from privacy a…

You don’t pay Cloudflare money nor are you customer. The customers are the sites that pay them money to protect their infrastructure. The customers have many choices in CDN/WAF providers and Cloudflare isn’t even largest, Akamai is. Fastest growing CDN is cloudfront. There is healthy competition so why would anti trust laws apply?

Re: Blocked by Cloudflare

#64

Post by thedaly »

crote wrote: post #42
sschueller wrote: post #35

Earlier quoted context omitted.

I don't agree. Maybe don't run WordPress that makes 100+ dB queries before first page load on some cheap $2 vps. There are all kinds of tools that you can easily deal with bots and the large DDOS your ISP can handle for you if you are willing to pay for it.

> the large DDOS your ISP can handle for you if you are willing to pay for it That's exactly what people are paying Cloudflare for, because contrary to your local ISP they are actually competent at blocking a DDOS attack. People use services like Cloudflare exactly because they don't want to spend a fortune on complex infrastructure just to deal with abuse. Even a mostly-static page running on a reasonably-specced se…

> That's exactly what people are paying Cloudflare for

Cloudflare actually provides this service for free (for simple use cases at least).

I don't know how to come down on this issue. On one hand, I am against the centralization of cloudflare and the risks that come with it.

On the other hand, cloudflare allows almost anyone to set up a simple website and serve it to large numbers of people with very little resources/cost and advanced protection from DDOS attacks.

Re: Blocked by Cloudflare

#65

Post by adammartinetti »

AegirLeet wrote: post #13

I've had the exact same problem for a while. Here are some of the sites I've been unable to access (found by searching for "just a moment" in my browser history): - https://gitlab.com/users/sign_in - https://steamdb.info/login/ - https://www.zabbix.com/forum/ - https://casetext.com/ - https://namemc.com/login - https://spinroot.com/ - https://camelcamelcamel.com/ It's really annoying and Cloudflare is apparently doin…

Would you be willing to share a rayID you see during one of these looping challenges? I'm the PM for Cloudflare's challenge platform, and we'd love to look into this. RayIDs contain no PII so you can share publicly, or feel free to drop me an email at amartinetti at cloudflare.

We'll also release a reporting mechanism soon, so in the future you can let us know when you see these issues and we can react to them quickly.

Re: Blocked by Cloudflare

#66

Post by runnerup »

derefr wrote: post #46
johnklos wrote: post #31

Earlier quoted context omitted.

How does that explain blocks that happen to less common browsers and/or less common platforms?

Heuristics are about optimizing between false positives and false negatives. Many headless-browser stealth techniques involve rotating between the signatures and reflected metrics of real — but niche and/or ancient — User-Agents. (For some reason, the developers of these stealth systems think that variety beats commonality. Maybe it makes sense if they're specifically trying to overcome Apache mod_security's signatur…

> Many headless-browser stealth techniques involve rotating between the signatures and reflected metrics of real — but niche and/or ancient — User-Agents

I work in this space and we just use fingerprints we collect from actual users over the previous month. The hardest work is: 1) reverse-engineering the javascript of the CAPTCHA/fingerprinting solutions so that we can collect and encapsulate the fingerprints correctly in a way that looks native to cloudflare/recaptcha/etc, and 2) Training AI models to solve the captchas well enough.

Sounds like most of the people you catch are using ancient user-agents. But I doubt most of the people you want to catch are.

Re: Blocked by Cloudflare

#67

Post by adrr »

AegirLeet wrote: post #13

I've had the exact same problem for a while. Here are some of the sites I've been unable to access (found by searching for "just a moment" in my browser history): - https://gitlab.com/users/sign_in - https://steamdb.info/login/ - https://www.zabbix.com/forum/ - https://casetext.com/ - https://namemc.com/login - https://spinroot.com/ - https://camelcamelcamel.com/ It's really annoying and Cloudflare is apparently doin…

Users want to chrome hegemony and don't care about the open web or Firefox. Its the number #1 browser on desktop even though it doesn't come with the OS. Windows comes with edge and macs come with safari. Users have to download Chrome.

Re: Blocked by Cloudflare

#68

Post by michaelt »

zer8k wrote: post #23
AegirLeet wrote: post #13

I've had the exact same problem for a while. Here are some of the sites I've been unable to access (found by searching for "just a moment" in my browser history): - https://gitlab.com/users/sign_in - https://steamdb.info/login/ - https://www.zabbix.com/forum/ - https://casetext.com/ - https://namemc.com/login - https://spinroot.com/ - https://camelcamelcamel.com/ It's really annoying and Cloudflare is apparently doin…

I haven't had any problems on Waterfox. However, it is absurd to me I need javascript to simply visit a website anymore.

Sadly, the fact a given site works for you or me is no guarantee it works for someone else.

These bot detection systems tend to use all manner of imprecise statistical heuristics and weird fingerprinting.

Perhaps AegirLeet has a graphics card that a popular web scraper pretends to have. Maybe they're in a suspicious timezone. Maybe they've installed a font usually only found on a different operating system. Maybe I'm never blocked because I have an excellent IP reputation, due to regular visits to approved websites.

Re: Blocked by Cloudflare

#69

Post by superkuh »

This has been my experience for a handful of years but of course it's getting worse. In the past I'd just be getting blocked from access commercial websites or applications and things I didn't really need. But in the last few years many scientific publishers have put all their content behind cloudflare walls. Pretty much my only hope of being able to read a paper these days is that it came out long enough ago to be on sci-hub or they published the pre-print on arxiv/bioarxiv/etc. Once arxiv goes behind a cloudflare I don't know what I'll do.

Re: Blocked by Cloudflare

#70

Post by j45 »

adrr wrote: post #67
AegirLeet wrote: post #13

I've had the exact same problem for a while. Here are some of the sites I've been unable to access (found by searching for "just a moment" in my browser history): - https://gitlab.com/users/sign_in - https://steamdb.info/login/ - https://www.zabbix.com/forum/ - https://casetext.com/ - https://namemc.com/login - https://spinroot.com/ - https://camelcamelcamel.com/ It's really annoying and Cloudflare is apparently doin…

Users want to chrome hegemony and don't care about the open web or Firefox. Its the number #1 browser on desktop even though it doesn't come with the OS. Windows comes with edge and macs come with safari. Users have to download Chrome.

All browsers so far have come and gone in popularity. Even when it seemed unimaginable.
Post reply on HN