Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Blocked by Cloudflare

jrhawley.ca

Post reply on HN
Print view View original

151–160 of 473 posts

Re: Blocked by Cloudflare

#151

Post by acedTrex »

bradley13 wrote: post #78

Anecdote: For my programming classes, one example I use is a simple browser. It doesn't do CSS or Javacript, so display is primitive, but it works. On some sites. Many sites, especially the big ones, see that it's an unknown browser, and refuse to send content. Probably they think it's a bot. But even if it were, what's wrong with bots, as long as they're well-behaved? What kind of closed web have we let the megacorp…

I mean, the site gets to decide if it will service a request. theres no requirement for a service to respond to everyone.

Re: Blocked by Cloudflare

#152

Post by antonvs »

dmix wrote: post #149
executesorder66 wrote: post #132

Earlier quoted context omitted.

> So many privacy nuts use Chrome Really? That's news to me.

Well, Chromium is quite popular with the security conscious on Linux. At least it was when I was using ArchLinux, they had some good custom build script versions.

Security conscious and privacy conscious aren't the same thing, although there's overlap. I can be concerned about the security of my system without caring about whether I'm being targeted for ads.

Re: Blocked by Cloudflare

#153

Post by shadowgovt »

shiomiru wrote: post #129
adammartinetti wrote: post #76

Hi there, I'm the PM for Cloudflare's challenge platform. I'd love to look into what the cause of the problem is, so you don't see these difficulties. > Cloudflare detected the high frequency of requests and denials (but not their faulty loop that caused this pattern of requests, of course), and tagged my browser as suspicious. I can tell you at least that we don't penalize users for this looping behavior, so this wo…

The cause of the problem is that your software is faulty by design. 1. IP addresses are to be used for packet routing. Certainly not for assigning "behavior scores" to users in the background. IP addresses say nothing about your visitors, my IP address could have been a complete stranger's IP address yesterday. 2. Deciding who can access half the web based on their TLS signature achieves nothing in the long run excep…

You're going by the specified, designed use cases of those technologies.

Every spec is a three-edged sword: the spec, the intent of the spec, and the use of the spec in the wild.

In practice, Cloudflare does a pretty good job on far-more-than average of gluing together some heuristics in an unspec'd way to filter traffic. It sucks because you can't plan around it, but that's rather the point because the malicious actors are trying to plan around it also.

(ETA: Hacker News rate-limited this post. In theory, I could have set up a sock-puppet to try and work around that, but then they would catch that too and I'd be out two accounts. So I just waited out the limit. Measure and counter-measure. ;) ).

Re: Blocked by Cloudflare

#154

Post by dcow »

shiomiru wrote: post #129
adammartinetti wrote: post #76

Hi there, I'm the PM for Cloudflare's challenge platform. I'd love to look into what the cause of the problem is, so you don't see these difficulties. > Cloudflare detected the high frequency of requests and denials (but not their faulty loop that caused this pattern of requests, of course), and tagged my browser as suspicious. I can tell you at least that we don't penalize users for this looping behavior, so this wo…

The cause of the problem is that your software is faulty by design. 1. IP addresses are to be used for packet routing. Certainly not for assigning "behavior scores" to users in the background. IP addresses say nothing about your visitors, my IP address could have been a complete stranger's IP address yesterday. 2. Deciding who can access half the web based on their TLS signature achieves nothing in the long run excep…

I sympathize with your frustration, but you also have to admit that Cloudflare is tasked with an impossible problem: from a sea of requests, identify those that are coming from robots that are disguised as humans.

So there is no perfect solution. You can't use strong identity because a user can share their identity with a robot. You have to use a crapy heuristic that only works most of the time (or tell site owners it's an application layer problem and use this SASS solution to solve the problem).

I mean you admitted that you run a crawler. Cloudflare has detected that you run a crawler and has wants you to prove that you're human to access sites on their network. It actually sounds like their product worked.

In any event, there should probably be better regulation around how this blocking is handled so that users aren't being unjustly blocked. If you want to run a crawler, how do you do it ethically so that you aren't targeted and your traffic blocked? If Cloudflare blocks you from accessing one site should that block extend across their whole network? How long should it last? How do you appeal the block if Cloudflare's heuristics falsely block you? If you're in a life and death situation and need immediate access to medical information and Cloudflare unjustly blocks your access and it causes harm, who's at fault? Etc.

Re: Blocked by Cloudflare

#155

Post by antonvs »

zer8k wrote: post #138
cortesoft wrote: post #91

Earlier quoted context omitted.

Cloudflare does not have nearly enough market share to be an anti-trust concern.

Cloudflare controls about 19% of websites. Of the websites that use a CDN, 80% of them use Cloudflare [0] (in a note at the bottom of link). [0] https://community.cloudflare.com/t/statistically-speaking-wh...

Afaik, neither of those numbers legally constitute a monopoly and wouldn't qualify for antitrust action.

Re: Blocked by Cloudflare

#156

Post by ricardo81 »

WheatMillington wrote: post #145
shiomiru wrote: post #129

Earlier quoted context omitted.

The cause of the problem is that your software is faulty by design. 1. IP addresses are to be used for packet routing. Certainly not for assigning "behavior scores" to users in the background. IP addresses say nothing about your visitors, my IP address could have been a complete stranger's IP address yesterday. 2. Deciding who can access half the web based on their TLS signature achieves nothing in the long run excep…

You're being a little dramatic. It's incredibly unlikely that millions of innocent users have been blocked, and unless you have data to the contrary you shouldn't make such a claim. You know what else is harmful to the concept of the open internet? The enormous malicious botnets and other endemic problems that require a solution like CloudFlare.

> It's incredibly unlikely that millions of innocent users have been blocked

Is there a 'town square' where we can talk about being presented captchas and similar things from 3rd party intermediates.

I think it's incredibly likely that millions of hours have been wasted on such challenges.

Re: Blocked by Cloudflare

#157

Post by lambda »

shiomiru wrote: post #129
adammartinetti wrote: post #76

Hi there, I'm the PM for Cloudflare's challenge platform. I'd love to look into what the cause of the problem is, so you don't see these difficulties. > Cloudflare detected the high frequency of requests and denials (but not their faulty loop that caused this pattern of requests, of course), and tagged my browser as suspicious. I can tell you at least that we don't penalize users for this looping behavior, so this wo…

The cause of the problem is that your software is faulty by design. 1. IP addresses are to be used for packet routing. Certainly not for assigning "behavior scores" to users in the background. IP addresses say nothing about your visitors, my IP address could have been a complete stranger's IP address yesterday. 2. Deciding who can access half the web based on their TLS signature achieves nothing in the long run excep…

And even when it doesn't block you completely, it delays website loading, makes you jump through frustrating captchas, etc.

It's probably third in the list of frustrating web behaviors in the past couple of years (behind GDPR popups and registration/paywalls that seem to have gotten much worse recently).

And somehow there are some sites that I get CF delay walls on every time I visit.

This feature is utterly broken for a good web experience; it pushes users away from sites which use it.

Every time that "checking your browser" page comes up for a legitimate user should be considered a failure. Sure, it can maybe happen a few times in a thousand, but the feature is utterly broken if it comes up every time I visit the same site from the same browser not in private mode.

Re: Blocked by Cloudflare

#158

Post by NicoJuicy »

buzer wrote: post #53
NicoJuicy wrote: post #36

Earlier quoted context omitted.

Based on what? https://developers.cloudflare.com/support/troubleshooting/ge...

https://developers.cloudflare.com/bots/reference/verified-bo... https://radar.cloudflare.com/traffic/verified-bots https://blog.cloudflare.com/friendly-bots/ > At Cloudflare, we manually “verify” good bots, so they don’t get blocked.

Nice links!

But cloudflare references robots.txt a lot ( which i mentioned before => to respect robots.txt)

Additionally, they solve the authentication problem here. As a website owner that got bad crawlers ( that copied user agents), i just whitelisted Google's IP's and blocked all the other crawlers.

It seems that cloudflare is actually fixing this problem and making competition for Google possible here.

I recall an article that Bing actually circumvented robots.txt a bit, because site owners were only allowing Google and blocked all the rest => Bing. Which gave Google an unfair advantage ( searched for it, couldn't find it)

Similar article to highlight the issue: https://www.fastcompany.com/90709672/the-little-known-reason...

Our opinions about this seem to differ severely. Cloudflare actually enables good bots to start competing ( while respecting robots.txt).

Re: Blocked by Cloudflare

#159

Post by adamgamble »

One thing that sometimes gets lost is site owners that use cloudflare have sort of global options for how paranoid they want to be, then they can make specific WAF rules that can be as granular and aggressive as they want. So at least in some cases, cloudflare gets blamed for website owners setting really aggressive rules. The effect on the end user usually looks exactly the same.

Case in point, I set a waf rule that blocked all non verified bot traffic from several big datacenters (Google cloud, OVH, digital ocean, etc). That turned out to be a mistake because a lot of corporations were routing their traffic through those ASNs for some reason. Now they’re blocked. They could have gotten pissed out cloudflare, the error page looks the same, but it was really misconfiguring it.

Re: Blocked by Cloudflare

#160

Post by [deleted] »

cortesoft wrote: post #91
zer8k wrote: post #12

Any time a large portion of internet traffic is controlled by a single source it brings problems like this with it. All cloudflare has to do is arbitrarily decide who and who can't use the internet and effectively their word becomes law. Like most things it starts with an innocent premise (e.g. "an easy way to stop bad actors") and ends up extended to any number of arbitrary things. Worse, the argument from privacy a…

Cloudflare does not have nearly enough market share to be an anti-trust concern.

[deleted]
Post reply on HN