Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You don’t want to be on Cloudflare’s naughty list

ctrl.blog

Post reply on HN
Print view View original

31–40 of 354 posts

Re: You don’t want to be on Cloudflare’s naughty list

#31

Post by judge2020 »

cvwright wrote: post #18
jgrahamc wrote: post #3

Well into the second day of Cloudflare’s blockade of my home internet connection, Google Search also began blocking requests. It required me to resolve a CAPTCHA challenge for every other search. This luckily only lasted a day. Cloudflare shares IP reputation data with partners like Google, coordinated through a program called the Bandwidth Alliance. So, my original offense might not even have been against Cloudflare…

You block this guy from the internet for a week —- for no apparent reason —- and then you come in here with a nitpick about how another related system works? Really?

The point is that Cloudflare does not beam IP reputation data to Google. If Google and CF are blocking this IP separately, what's the chance there's some malicious device or hacked IoT device on the network, participating in DDOS attacks or unauthorized vulnerability scanning of random websites?

Re: You don’t want to be on Cloudflare’s naughty list

#32

Post by therealmarv »

If you surf on desktop sites from Philippines on a mobile phone plan (which is often the best Internet connection in that country) you also get Cloudflare's captchas everywhere.

I told it before and tell it now again: Cloudflare is dividing the World between first and second/third World countries with their captchas. I call it discrimination of second/third World countries! If you are from US and Europe you will never notice it but if you travel a little bit more you see these blocking captchas everywhere.

Re: You don’t want to be on Cloudflare’s naughty list

#33

Post by TakeBlaster16 »

jgrahamc wrote: post #3

Well into the second day of Cloudflare’s blockade of my home internet connection, Google Search also began blocking requests. It required me to resolve a CAPTCHA challenge for every other search. This luckily only lasted a day. Cloudflare shares IP reputation data with partners like Google, coordinated through a program called the Bandwidth Alliance. So, my original offense might not even have been against Cloudflare…

Can you acknowledge the main point of the article? What should someone do if they find themselves misclassified by Cloudflare's systems?

Re: You don’t want to be on Cloudflare’s naughty list

#34

Post by nijave »

daenney wrote: post #26
superkuh wrote: post #6

Daniel Aleksandersen of ctrl.blog has absolutely no foot to stand on here. He is a proponent of this kind of algorithmic blocking for weird browsers and even implemented it on his own site and argued for it. https://www.ctrl.blog/entry/detect-non-browser-form-submissi... It's only after it happened to him that now he's suddenly against it. Until he removes the same type of blocks from his own website I have absolutel…

Burn the witch! Lets read through that page for a second though: Drop support for obsolete HTTP versions Doesn't seem like that's going to cause much issue for any legitimate client from the past 10-20 years. He only recommends blocking HTTP 0.9/1.0, which fair enough Append a #hash to the form’s action URL Hah. Clever man. I don't see how this is going to stop any legitimate user from loading your website or submitt…

All that stuff is easily defeated by automated browsers anyway (i.e. selenium)

Re: You don’t want to be on Cloudflare’s naughty list

#35

Post by mh- »

DethNinja wrote: post #8

There is a chance you might’ve been hacked. You would be surprised to see how easy it is to hack domestic routers. 1. Find and disinfect the devices, including the router. If you don’t have enough technical knowledge, then buy a new router. 2. Use 30 character long random password on the router. 3. Disable UPnP. 4. Anything with WI-FI and weak password can be hacked within minutes, so check your other devices as well…

My assumption is also that something on his network is compromised, and getting his IP into reputation issues.

Tarpitting (serving content slowly from the edge, in order to slow down bots) is necessarily one of the most expensive tools in a WAF/CDN's toolbox.

It's much more likely that something on his network is sending sketchy traffic to CF-fronted/Google sites, and the slow loading he's experiencing elsewhere is because his upstream is being saturated by whatever is happening on his network.

Re: You don’t want to be on Cloudflare’s naughty list

#36

Post by ReptileMan »

bergwerf wrote: post #23
superkuh wrote: post #6

Daniel Aleksandersen of ctrl.blog has absolutely no foot to stand on here. He is a proponent of this kind of algorithmic blocking for weird browsers and even implemented it on his own site and argued for it. https://www.ctrl.blog/entry/detect-non-browser-form-submissi... It's only after it happened to him that now he's suddenly against it. Until he removes the same type of blocks from his own website I have absolutel…

From the link you mentioned: > Bots often mimic the User-Agent of a common browser, but the version numbers used in the bots rarely change. Over time they drift farther and farther behind until a point (maybe two-year-old versions) where you can safely block them without inconveniencing legitimate users. This supports the idea that browsers are subject to constant change and everyone should be forced to come along (r…

Doesn't your chrome app updates? Never used chromebook. Just asking.

Re: You don’t want to be on Cloudflare’s naughty list

#37

Post by kevingadd »

I'm used to getting assaulted by Cloudflare's browser check interstitials along with random Cloudflare and Google CAPTCHAs because (presumably) I run Firefox and an ad-blocker instead of vanilla Google Chrome. It's already tremendously inconvenient to wait multiple seconds on many page loads and click 20 bicycles, I can only imagine how infuriating it would be if every page load started taking 60 seconds because your IP ended up on some random algorithmic blacklist....

Re: You don’t want to be on Cloudflare’s naughty list

#38

Post by tomxor »

jgrahamc wrote: post #3

Well into the second day of Cloudflare’s blockade of my home internet connection, Google Search also began blocking requests. It required me to resolve a CAPTCHA challenge for every other search. This luckily only lasted a day. Cloudflare shares IP reputation data with partners like Google, coordinated through a program called the Bandwidth Alliance. So, my original offense might not even have been against Cloudflare…

FYI, this guy is far from alone, your "protection" has given me a lot of grief over the past few years, particularly on highly NATed mobile networks.

I've been gradually removing cloudflare based CDNs from services I develop and control because I don't want my users being arbitrarily discriminated against.

There was a good article posted on HN recently titled "The ideal level of fraud is non-zero" which I think is highly relevant here... In essence any mechanism employed to prevent illegitimate use comes with a negative cost to legitimate users, if that cost is too high it defeats the purpose. i.e what's the point in a website that is completely immune to a botnet and also cannot be accessed by anyone else? unplugging the ethernet cable also effectively protects against botnets. More subtly the cost of outright rejecting some legitimate users is usually not worth the savings of rejecting 100% of illegitimate ones. I think Cloudflare's service has it the wrong way around: it currently accept blocking legitimate users far too easily, that is not an acceptable cost; whereas you should be letting a higher level of bots through to avoid pissing off legitimate users - if it's not obviously a DDoS, it's probably worth the bandwidth cost.

Consider the bigger picture, if you save a slither of a penny by blocking a bot, but also end up blocking or seriously inconveniencing 10 real users... is it worth it.

Re: You don’t want to be on Cloudflare’s naughty list

#39

Post by ldoughty »

superkuh wrote: post #25
ldoughty wrote: post #19

Earlier quoted context omitted.

I don't know the author or his reputation, but his suggestions that you linked are (in my opinion) standard actions for any dev/server admin getting spammed by their forms... And the suggestions really only impact malicious actors accessing your website from a script... Virtually none of those would be an issue for any browser made in the last 15-20 years, or headless browsers, but would break rudimentary scripts lik…

I guess like ctrl.blog you can't grasp the significance of the issue until it happens to you. My firefox fork is definitely blocked by his algorithmic "bot" detector. Just because your browser isn't doesn't mean it only blocks bots. False positives happen. They happen a lot more than you think. And they are a serious problem. Even more serious when it's cloudflare, but arguing for everyone to implement these algorith…

I don't see the reason for the personal attack.

The blog post also calls out that you should not block based on user agent.

If a form post didn't respect the action property having a #, that name/email HTML names might be reversed (whole the type is correct, and the user displayed values are correct), or include hidden HTML form fields that have been standard since ~97? Back when I made my first few websites, I certainly would agree that they are likely bots.

Again, apparently this person has some hateful following, but I don't appreciate you limping me into this hatred for agreeing with his statements on this one particular issue.

Re: You don’t want to be on Cloudflare’s naughty list

#40

Post by LinuxBender »

daenney wrote: post #26
superkuh wrote: post #6

Daniel Aleksandersen of ctrl.blog has absolutely no foot to stand on here. He is a proponent of this kind of algorithmic blocking for weird browsers and even implemented it on his own site and argued for it. https://www.ctrl.blog/entry/detect-non-browser-form-submissi... It's only after it happened to him that now he's suddenly against it. Until he removes the same type of blocks from his own website I have absolutel…

Burn the witch! Lets read through that page for a second though: Drop support for obsolete HTTP versions Doesn't seem like that's going to cause much issue for any legitimate client from the past 10-20 years. He only recommends blocking HTTP 0.9/1.0, which fair enough Append a #hash to the form’s action URL Hah. Clever man. I don't see how this is going to stop any legitimate user from loading your website or submitt…

He's quite tame compared to me I suppose. I block anything that is not HTTP/2.0 which currently knocks out all the bots and all crawlers except Bing. But I just have hobby sites these days. Nobody would notice or care if my sites went offline.

Using NGinx as an example:

    if ($server_protocol != HTTP/2.0) { return 403 'Nope'; }
Another thing I have found useful to drops some bots is to become invisible to them. Many of the poorly written scanning tools do not properly set MSS for reasons I still don't understand. I use this to my advantage.

Using IPTables as an example:

    /sbin/iptables -t raw -I PREROUTING -i eth0 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m tcpmss ! --mss 420:16384 -j DROP
Any TCP packets setting a very low or high MSS or missing MSS will be silently dropped. I drop about 35K packets per host per day on average. This also drops hping3 floods.
Post reply on HN