Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare CEO is lying to you about the bot traffic jump

flyingpenguin.com

Post reply on HN
Print view View original

71–80 of 149 posts

Re: Cloudflare CEO is lying to you about the bot traffic jump

#71

Post by pbasista »

autoexec wrote: post #24
thm wrote: post #14

Why did we start treating Cloudflare (a public, for-profit company) as the undisputed authority on anything related to the network layer of the internet in the first place?

Because they inserted themselves into almost everything we do online and basically managed to take control over it. Cloudflare should never have been allowed to man in the middle the entire internet, but now that they have they're the only man on earth with a dataset that size.

> they inserted themselves

I am unaware of such a capability of Cloudflare.

I believe it is the site administrators who have inserted Cloudflare in between their sites and their users.

Usually it is done for rational reasons of establishing a protection against bots. But what is less rational, in my opinion, is when everyone uses the same provider for that.

Because it indirectly turns Cloudflare into a monopoly. And monopolies often converge to a state when they start to abuse their position.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#72

Post by autoexec »

majke wrote: post #34
autoexec wrote: post #24

Earlier quoted context omitted.

Because they inserted themselves into almost everything we do online and basically managed to take control over it. Cloudflare should never have been allowed to man in the middle the entire internet, but now that they have they're the only man on earth with a dataset that size.

(I'm ex CF) This is backwards. Nobody "allowed" anyting. CF serves a customers need. You can argue with the solution but you can't argue with the core problem. It's more healthy to start the conversation of _why_ CF services are valuable.

cloudflare (and CDNs more generally) certainly started out as an own goal. Warnings were ignored and most people did the easy thing instead of the right thing.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#73

Post by taeric »

reconnecting wrote: post #37
taeric wrote: post #16

Earlier quoted context omitted.

For ad views, the concern is specifically that people pay for clicks and views. That that can be so heavily influenced by bot traffic greatly undermines their value. Same general idea goes for any of the algorithmic driven platforms. The algorithms are ostensibly intended to surface organically discovered things by watching how people interact with things. That they are so susceptible to distortion through bot farms…

For ads, I believe this must be a problem for ad platform owners. If the digital platform's storefront is their business, they could afford to spend some budget on bot detection. Bots still come from data center networks, sometimes render pages incompletely, request resources in bulk, and show enough patterns to be flagged internally. If we look at a medium website, most random crawlers will come from Amazon, Microso…

Largely agreed, though I think you are likely underestimating how hard this is to detect. In particular, it is true that many bots can be hosted in data centers, but it is somewhat trivial to launder that traffic through other sources. Malware, in particular, is what I have in mind. Maybe I'm wrong and that has largely gone away?

There is also a bit of mixed incentives. Yes, it is the ad platform that is getting abused. But it is also the ad platform that is charging people based on abused practices.

And it isn't like this is completely made up. Just look at how facebook killed a lot of ton of people during the "pivot to video" programs. I don't know all of the details, as I was thankfully not in any of the involved industries, but my understanding is it is fairly well documented.

Edit: I changed an "isn't" to "is." I think I was trying to reword at one point, but left it in a way that is opposite what I meant.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#74

Post by NetOpWibby »

dwedge wrote: post #60
majke wrote: post #34

Earlier quoted context omitted.

(I'm ex CF) This is backwards. Nobody "allowed" anyting. CF serves a customers need. You can argue with the solution but you can't argue with the core problem. It's more healthy to start the conversation of _why_ CF services are valuable.

> CF serves a customers need CF serves something it convinced customers they need. Static blogs hiding behind bot protection (in some cases blocking legit users from GrapheneOS because it's difficult to fingerprint them) because someone convinced them they'll be DDoSed by bots otherwise is a loss to the Internet. A lot of self-hosters running CF tunnels because they don't know better also contributes. > It's more hea…

   A lot of self-hosters running CF tunnels because they don't know better also contributes.
If you know better, you should contribute.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#75

Post by majke »

dwedge wrote: post #60
majke wrote: post #34

Earlier quoted context omitted.

(I'm ex CF) This is backwards. Nobody "allowed" anyting. CF serves a customers need. You can argue with the solution but you can't argue with the core problem. It's more healthy to start the conversation of _why_ CF services are valuable.

> CF serves a customers need CF serves something it convinced customers they need. Static blogs hiding behind bot protection (in some cases blocking legit users from GrapheneOS because it's difficult to fingerprint them) because someone convinced them they'll be DDoSed by bots otherwise is a loss to the Internet. A lot of self-hosters running CF tunnels because they don't know better also contributes. > It's more hea…

> A lot of self-hosters running CF tunnels because they don't know better also contributes.

Are you saying CF documentation is better than Computer Science / Networking education resources? Why don't people know better? I thought the tunnels are mostly used to bypass NAT's.

> Static blogs hiding behind bot protection

I'm not sure what is the proportion of the static vs dynamic sites, but I would argue that for wordpress CF is adding real value.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#76

Post by autoexec »

layer8 wrote: post #58
majke wrote: post #34

Earlier quoted context omitted.

(I'm ex CF) This is backwards. Nobody "allowed" anyting. CF serves a customers need. You can argue with the solution but you can't argue with the core problem. It's more healthy to start the conversation of _why_ CF services are valuable.

Regulation could conceivably disallow a single company to control such a significant portion of internet traffic. The parent can be interpreted as lamenting the absence of such regulation.

It's certainly a huge risk when one company's outage can take down so much of internet. Decentralization and the ability to route around damage was a core feature of the internet. The more we depend on the internet, the less acceptable that loss becomes.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#77

Post by rs_rs_rs_rs_rs »

dwedge wrote: post #60
majke wrote: post #34

Earlier quoted context omitted.

(I'm ex CF) This is backwards. Nobody "allowed" anyting. CF serves a customers need. You can argue with the solution but you can't argue with the core problem. It's more healthy to start the conversation of _why_ CF services are valuable.

> CF serves a customers need CF serves something it convinced customers they need. Static blogs hiding behind bot protection (in some cases blocking legit users from GrapheneOS because it's difficult to fingerprint them) because someone convinced them they'll be DDoSed by bots otherwise is a loss to the Internet. A lot of self-hosters running CF tunnels because they don't know better also contributes. > It's more hea…

[flagged]

Re: Cloudflare CEO is lying to you about the bot traffic jump

#79

Post by NetOpWibby »

The amount of people in these comments demonizing Cloudflare services or conflating their very existence with rubbish/trash/nonsense because you personally know better is wild.

Y'know what would be better? Making a site showcasing what tools are better than Cloudflare's services! And how to use them! And sharing said site so people know about them!

Re: Cloudflare CEO is lying to you about the bot traffic jump

#80

Post by Symbiote »

Bender wrote: post #15

I tested this theory not long ago and did not see anything that aligned with the hype around bots. [1] There are indeed more bots than humans because of course there are or at least the appearance of . Bots crawl everything linked from popular sites whereas humans only click on things that interest them and even then they do not typically siphon the entire site. There are new bot operators every day due to curiosity…

My employer's site was recording 1,500 requests per second from a single AI bot earlier this week. The requests came from 2.4 million different IPs at the time I looked, between 1-2 requests from each IP, most likely all were unique URLs. That single bot was 55% of traffic. This kind of crawling pushes us to (sometimes beyond) the limit of our capacity.

I have also seen thousands of requests per hour from the IP to a small set of pages, e.g. the homepage. I don't know why; it doesn't matter so I ignore it.

I've recently found there are websites offering curated "AI ready" datasets, and several of these sites claim to have indexed our site, on the 3-4 I looked at it was one of a few hundred datasets. It's interesting enough to be something an AI company would want, so my conclusion is the site is being specifically targeted by the AI bot developers.

Post reply on HN