Cloudflare CEO is lying to you about the bot traffic jump • Hacker News Board

Re: Cloudflare CEO is lying to you about the bot traffic jump

#101

Post by realusername » Fri, Jun 05, 2026, 8:16 PM UTC

gruez wrote: post #56

1vuio0pswjnm7 wrote: post #4

There is a unfortunate incentive created when a "business" (MiTM) depends on "bot traffic", i.e., the continued nuisance of bot traffic, to make money If the "bot traffic" declines, then the "bot protection business" goes down with it Cloudflare communication are sometimes careful to refer to traffic _labeled as_ bot traffic versus actual bot traffic Because the "business" relies on the existance of "bot traffic", th…

>It can be used to force users to use certain software, e.g., certain browsers, and to enable Javascript subjecting users to data collection, surveillance and ads >Certainly the problem is not the individual www user who doesnt use an "approved" graphical, Javascript-enabled browser who gets blocked or fingerprinted trying to make a single request The alternatives to javascript fingerprinting are either ineffective (…

> The alternatives to javascript fingerprinting are either ineffective (TLS fingerprinting and/or IP rate limits), or even worse for privacy (eg. attestation).

Javascript fingerprinting itself is ineffective, these kind of checks only stop the most basic bots and I'd argue the same for attestation.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#102

Post by Groxx » Fri, Jun 05, 2026, 8:17 PM UTC

epistasis wrote: post #69

Groxx wrote: post #52

Earlier quoted context omitted.

I think it counts as "allowed" regardless of utility. CF is so massively over-weight on the internet that it's impossible to trust them with anything because if they can be forced to do something by a hostile government (hint: they can be!) then they can get away with it invisibly and affect billions of people. That is something that should not be allowed to exist. It's one of the reasons monopolies (or even majority…

Then I think the real question is why haven't any serious competitors emerged that can handle the essential services that Cloudflare provides? Are there network effects like what happens with Microsoft in the business computing space? With Microsoft, I'm also aware of a great amount of anti-competitive behavior, and though I haven't seen that from Cloudflare personally and haven't heard accusations of it, I also have…

I suspect a big part of it is that CF is running other businesses on the side, and offering basic features at a loss - they've artificially depressed the price of the service so it's hard to compete with them on only that service.

Everyone using the free service likes that, of course, but honestly I wish we'd make it illegal to do. It's heavily used as a way to steal small markets simply by being successful in a different large one.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#103

Post by Groxx » Fri, Jun 05, 2026, 8:18 PM UTC

reconnecting wrote: post #68

Groxx wrote: post #55

Earlier quoted context omitted.

Yeah, I've seriously considered finding or building a CF-protected-detector browser extension to flag domains. Having one company MITMing so much traffic is straightforwardly dangerous, and not just an annoyance. We need competition.

Why so? They're all in NS already: *.ns.cloudflare.com

"why so" what? not sure what you're questioning/implying here.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#104

Post by sometimelurker » Fri, Jun 05, 2026, 8:29 PM UTC

jimrandomh wrote: post #29

I deal with scrapers that sometimes border on DDoSes for LessWrong. The amount of bot traffic varies greatly between sites; if you have more URLs you get more bot traffic (regardless of whether those URLs represent a deep content catalog, or useless URL parameter permutations). It's bad for LW because of the content-catalog depth. It's easy to drastically underestimate the amount of bot traffic, because bots make eff…

thank you for maintaining LessWrong

Re: Cloudflare CEO is lying to you about the bot traffic jump

#105

Post by gruez » Fri, Jun 05, 2026, 8:33 PM UTC

realusername wrote: post #101

gruez wrote: post #56

Earlier quoted context omitted.

>It can be used to force users to use certain software, e.g., certain browsers, and to enable Javascript subjecting users to data collection, surveillance and ads >Certainly the problem is not the individual www user who doesnt use an "approved" graphical, Javascript-enabled browser who gets blocked or fingerprinted trying to make a single request The alternatives to javascript fingerprinting are either ineffective (…

> The alternatives to javascript fingerprinting are either ineffective (TLS fingerprinting and/or IP rate limits), or even worse for privacy (eg. attestation). Javascript fingerprinting itself is ineffective, these kind of checks only stop the most basic bots and I'd argue the same for attestation.

It's ineffective in the sense that in the worst case, bots can buy used iPads or whatever and use a robot arm + camera to do the scraping, but each incremental step increases the cost for scrapers. TLS fingerprinting means you can't use curl/requests and call it a day. Javascript makes it even more complicated by requiring a headful browser to solve challenges. The purpose is to increase the cost, not to eliminate all bots.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#106

Post by Groxx » Fri, Jun 05, 2026, 8:36 PM UTC

reconnecting wrote: post #9

taeric wrote: post #3

I confess a sad assumption that bot traffic is far higher than we have admitted for a long time. Though, maybe we would see different stats specifically to social media sights to astroturf like counts? Certainly feels that we have known for a long time that bots were larger in ad viewing than ad companies wanted to admit.

I don't understand what difference bots make. For me, a website (the public part) is a storefront. People walk down the street and see what's inside — that's the purpose. If something should not be available immediately, that's the private part of the store. I've been monitoring bot traffic on digital platforms for over 10 years. Sure, the crawler share is growing, some even with malicious intentions, and those I det…

For efficiently-hosted sites with little media it's not too bad. E.g. hosting a static site just doesn't cost much, even if you're hammered occasionally.

That's extremely far from all sites though. It's probably safe to say it's a severe minority, particularly when you ignore personal / non-profit-bringing sites. Tons of small and large sites run stuff like poorly-written wordpress or ruby on rails or thousands of microservices doing god knows what. A major increase in request volume on those can easily mean significant increases in hosting charges (e.g. small-% on big, many multiples on small) or significant effort in optimizing (which is expensive too).

Re: Cloudflare CEO is lying to you about the bot traffic jump

#107

Post by 1vuio0pswjnm7 » Fri, Jun 05, 2026, 8:46 PM UTC

1vuio0pswjnm7 wrote: post #4

There is a unfortunate incentive created when a "business" (MiTM) depends on "bot traffic", i.e., the continued nuisance of bot traffic, to make money If the "bot traffic" declines, then the "bot protection business" goes down with it Cloudflare communication are sometimes careful to refer to traffic _labeled as_ bot traffic versus actual bot traffic Because the "business" relies on the existance of "bot traffic", th…

Consider fingerprinting as a commercially-oriented data collection method in addition to a "bot protection" method

Is data collection for "age verification" truly for age verification purposes only

Many bloggers and online commenters are not willing to accept that assumption

Is fingerprinting different

It is data collection for the purpose of establishing "online identity"

To the extent its based on user-agent strings or other HTTP headers it requires allowing a third party to MiTM TLS connections

Some bloggers and online commenters are opposed to computer users who MiTM _their own_ TLS connections

How is Cloudflare different

Re: Cloudflare CEO is lying to you about the bot traffic jump

#108

Post by Groxx » Fri, Jun 05, 2026, 8:46 PM UTC

sophrosyne42 wrote: post #70

Groxx wrote: post #52

Earlier quoted context omitted.

I think it counts as "allowed" regardless of utility. CF is so massively over-weight on the internet that it's impossible to trust them with anything because if they can be forced to do something by a hostile government (hint: they can be!) then they can get away with it invisibly and affect billions of people. That is something that should not be allowed to exist. It's one of the reasons monopolies (or even majority…

Seems more of a moral hazard of government intervention than of ventures whose economies of scale demand large market cap to most economically serve customer needs.

"Most economically" can be bent to nearly any purpose, there's almost always a cost-benefit to owning more of what you do, or externalizing costs onto others. It's not enough of a reason, and it's why laws restricting private behavior to serve the greater public exist.

Which is very nearly the only reason to have governments: forcing restrictions that lead to better results in aggregate.

(malicious control is another reason, but you kinda can't argue public benefit there)

Re: Cloudflare CEO is lying to you about the bot traffic jump

#109

Post by autoexec » Fri, Jun 05, 2026, 9:09 PM UTC

Groxx wrote: post #102

epistasis wrote: post #69

Earlier quoted context omitted.

Then I think the real question is why haven't any serious competitors emerged that can handle the essential services that Cloudflare provides? Are there network effects like what happens with Microsoft in the business computing space? With Microsoft, I'm also aware of a great amount of anti-competitive behavior, and though I haven't seen that from Cloudflare personally and haven't heard accusations of it, I also have…

I suspect a big part of it is that CF is running other businesses on the side, and offering basic features at a loss - they've artificially depressed the price of the service so it's hard to compete with them on only that service. Everyone using the free service likes that, of course, but honestly I wish we'd make it illegal to do. It's heavily used as a way to steal small markets simply by being successful in a diff…

> Everyone using the free service likes that, of course, but honestly I wish we'd make it illegal to do.

I can see the benefit, but if you made free services illegal companies like google would just offer gmail for a penny.

You'd could go farther and say that no one should be allowed to provide a good or service for less than it costs them to provide it. That still has problems though. For example, Netflix sends servers to ISPs to set up on their network so that segments of that ISP's customers are pulling data from one of those boxes distributing the load. ISPs are happy to do it because it means fewer customers call them to complain about slow speeds/buffering issues with netflix. If netflix wanted to start selling CDN services to others, having all those boxes at every major ISP in the nation means the cost to deliver that service will be much lower for netflix than it would for some new start up. You can't really make it illegal to offer a service at a price lower than it would cost the poorest and least efficient/capable would-be competitor though.

We'd also risk losing a lot. Everyone would be forced to pay youtube some amount proportionate to what it costs for youtube to exist. There are also extremely expensive to run services like the internet archive which would be bad to price people out of

Re: Cloudflare CEO is lying to you about the bot traffic jump

#110

Post by Groxx » Fri, Jun 05, 2026, 9:30 PM UTC

autoexec wrote: post #109

Groxx wrote: post #102

Earlier quoted context omitted.

I suspect a big part of it is that CF is running other businesses on the side, and offering basic features at a loss - they've artificially depressed the price of the service so it's hard to compete with them on only that service. Everyone using the free service likes that, of course, but honestly I wish we'd make it illegal to do. It's heavily used as a way to steal small markets simply by being successful in a diff…

> Everyone using the free service likes that, of course, but honestly I wish we'd make it illegal to do. I can see the benefit, but if you made free services illegal companies like google would just offer gmail for a penny. You'd could go farther and say that no one should be allowed to provide a good or service for less than it costs them to provide it. That still has problems though. For example, Netflix sends serv…

There are plenty of ways to mitigate that "losing a lot" (like government subsidies: run X, get paid $Y per Z -> easier access for both users and providers. we already do this for a lot of research and tons of industry, it's not some utopian dream), and since a lot of the free internet stuff we have now is ad-supported and addiction-oriented... I'm not convinced that'd be bad to lose. The adtech business causes horrific damage, and is consistently used as a weapon against people (ICE buying info commercially to get around laws that would normally block their access, for example). Not everything has to be allowed to exist.

For Internet Archive: IA is a non-profit that currently runs fine on donations, not ads, and in a healthier environment they'd get subsidies because they serve a clear public benefit, one which the government also relies on fairly frequently. They're a wildly different category of business than Cloudflare, both obviously and legally, and I think they'd be fine. And we'd be significantly better off if they had competition, because that'd serve as a distributed, duplicated backup.

I'm not claiming it'd be trivial, or perfect. I'm claiming it'd be worth the effort.