Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare CEO is lying to you about the bot traffic jump

flyingpenguin.com

Post reply on HN
Print view View original

131–140 of 149 posts

Re: Cloudflare CEO is lying to you about the bot traffic jump

#131

Post by pixelat3d »

jimrandomh wrote: post #29

I deal with scrapers that sometimes border on DDoSes for LessWrong. The amount of bot traffic varies greatly between sites; if you have more URLs you get more bot traffic (regardless of whether those URLs represent a deep content catalog, or useless URL parameter permutations). It's bad for LW because of the content-catalog depth. It's easy to drastically underestimate the amount of bot traffic, because bots make eff…

Over at AppleInsider, the amount of bot traffic is insane, and it's gotten to the point where it's starting to even jack up some of our Google Analytics. No, GA we didn't see a natural 320% rise in visitors from Singapore...

It's almost a full time job to manage. Our WAF rules are 90% 'oh shit AI scraper bots found a new vector.' It doesn't border on DDoS -- it effectively is. Coupled with all the Google changes that started a few years ago -- which is a separate topic I could rank about for ages -- 2026 is just a VERY bad time to be a website owner. I actually wiped most of our robots.txt rules the other day because literally nobody followed them. Anybody who tells you otherwise is flatly lying.

This is the new normal though, we gotta try and figure it out

Re: Cloudflare CEO is lying to you about the bot traffic jump

#132

Post by pixelat3d »

Groxx wrote: post #55
reconnecting wrote: post #5

Cloudflare bot detection has taught me a reflex to close the tab every time I see its logo.

Yeah, I've seriously considered finding or building a CF-protected-detector browser extension to flag domains. Having one company MITMing so much traffic is straightforwardly dangerous, and not just an annoyance. We need competition.

It's kind-of a requirement in 2026 -- you need something to MTM the amount of bot traffic that's going around these days and CF provides a good product and service (even outside of just the challenges portion.)

If you want to be mad at somebody, then it should be at the AI companies for basically forcing the rest of the internet to use them. They have broken every non-enforceable contract that it was built upon. We went nearly 25 years without needing something a MTM solution like that, and I would say, in the last 5 or so, it has become a hard requirement

Re: Cloudflare CEO is lying to you about the bot traffic jump

#133

Post by Bender »

phillipseamore wrote: post #94

All traffic includes stylesheets, JS, images and so on that your typical browser will faithfully retrieve to render a page but most bots or scrapers will ignore some or all of them.

That aligns with what I observed. Real people using HTTP/2.0 will pull my style-sheet and the bots just request the document I link here on HN.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#134

Post by Bender »

dentemple wrote: post #85

A lot of folks in these comments complaining about Cloudflare, but not many suggesting alternative solutions.

For little hobby sites I use some crude methods that have some false positives but work fairly well at keeping most of the riff-raff of the site. [1] It is work in progress but I am making something at your request. It's far from perfect but it's good enough for me. Probably won't be good enough for some. The document will evolve over the next few days.

[1] - https://blawg.nochan.net/b/Internet-Crap/20260606-How-To-Blo... [work in progress]

Re: Cloudflare CEO is lying to you about the bot traffic jump

#135

Post by Groxx »

Groxx wrote: post #124
reconnecting wrote: post #115

Earlier quoted context omitted.

There are domain zone files, and they sometimes contain NS records. Cloudflare hosted domains are easy to identify via their NS records, which also makes Cloudflare customers an ideal target for phishing.

(replying for myself, and for anyone else interested) Actually I think this might not be too hard. Cloudflare has a relatively small, published list of IP addresses: https://www.cloudflare.com/ips/ and Firefox has a DNS API for extensions: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/Web... (and kinda mostly not chrome, dev channel only: https://developer.chrome.com/docs/extensions/reference/api/d... ) Th…

Neat: https://postimg.cc/dLmzHfhJ

So yea, that strategy seems to work. PoC functional, gonna clean it up later.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#136

Post by trumpdong »

b65e8bee43c2ed0 wrote: post #93
jimrandomh wrote: post #31

Earlier quoted context omitted.

Have you checked your IP address's reputation with a service such as ipqualityscore.com? If cloudflare thinks your traffic is bot traffic, it's likely that there is bot traffic you don't know about coming from your IP, either from a compromised device on your network or a sketchy VPN product.

Google and Cloudflare mark whole ISPs as "sketchy" and there's fuck all you can do about it.

There's one thing: we can try to get every ISP marked as sketchy.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#137

Post by trumpdong »

epistasis wrote: post #69
Groxx wrote: post #52

Earlier quoted context omitted.

I think it counts as "allowed" regardless of utility. CF is so massively over-weight on the internet that it's impossible to trust them with anything because if they can be forced to do something by a hostile government (hint: they can be!) then they can get away with it invisibly and affect billions of people. That is something that should not be allowed to exist. It's one of the reasons monopolies (or even majority…

Then I think the real question is why haven't any serious competitors emerged that can handle the essential services that Cloudflare provides? Are there network effects like what happens with Microsoft in the business computing space? With Microsoft, I'm also aware of a great amount of anti-competitive behavior, and though I haven't seen that from Cloudflare personally and haven't heard accusations of it, I also have…

Cloudflare doesn't offer any essential services. Cloudflare offers snake oil to cure your gut of snakes they've told you are in your gut.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#138

Post by trumpdong »

oytis wrote: post #46
EnergyAmy wrote: post #39

Earlier quoted context omitted.

No, it's more healthy to start the conversation on why we allow corporations to do bad things with excuses like "just serving customer's needs"

I don't get it, what is bad about what they are doing? People need a CDN, they choose the one they find the best.

No, most people don't need a CDN. Cloudflare tells people they need a CDN, and hey, look, what a convenient free CDN right there.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#139

Post by trumpdong »

thejazzman wrote: post #51
giancarlostoro wrote: post #10

Earlier quoted context omitted.

I kind of do the same, not every time, but sometimes if I keep getting it on the same site, I seriously question how accurate their stats are without deep diving them more.

switching to tmobile home internet has been eye opening to me on how different the internet can be from person to person. you don't even get your own ipv4 address. makes me realize the challenge behind blocking something like yt-dlp

It's good that it's hard to block the free flow of information.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#140

Post by trumpdong »

pixelat3d wrote: post #132
Groxx wrote: post #55

Earlier quoted context omitted.

Yeah, I've seriously considered finding or building a CF-protected-detector browser extension to flag domains. Having one company MITMing so much traffic is straightforwardly dangerous, and not just an annoyance. We need competition.

It's kind-of a requirement in 2026 -- you need something to MTM the amount of bot traffic that's going around these days and CF provides a good product and service (even outside of just the challenges portion.) If you want to be mad at somebody, then it should be at the AI companies for basically forcing the rest of the internet to use them. They have broken every non-enforceable contract that it was built upon. We w…

No you don't need that. I don't have it.
Post reply on HN