Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare CEO is lying to you about the bot traffic jump

flyingpenguin.com

Post reply on HN
Print view View original

11–20 of 149 posts

Re: Cloudflare CEO is lying to you about the bot traffic jump

#11

Post by simonw »

"Cloudflare CEO is lying" is a bit of an aggressive take when he linked to the exact data so you can see it for yourself - and that's how this article was able to analyze it: https://radar.cloudflare.com/traffic#bot-vs-human

Update: I see the problem. Here's the full tweet: https://x.com/eastdakota/status/2062212701414187452

"Thought it would be end of 2027, then early 2027, but agentic traffic growing so fast that bots have now passed human traffic online for the first time in the Internet's history."

But the quoted segment in the article was just "…bots passed human traffic online for the first time in the Internet’s history."

It looks to me like the data supports "bots passed human traffic" but does NOT support "agentic traffic", since more of that traffic is from AI crawlers building indexes than from agents that are browsing the web on behalf of their owners.

If that's the point the article is trying to make then the headline is a little more supported, though I'd still say it's too hype-y a headline.

I guess a lot of this rests on what you assume "agentic traffic" to mean.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#12

Post by bobjordan »

The article strikes me as quite uncharitable to characterize it as "a lie". I doubt this CEO just sat down and calculated he was going to write lies. While it's fine to call out he's wrong per his own fuller data set, it's quite a different thing than calling the person out as a "liar" in a rage-bait fashion.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#15

Post by Bender »

I tested this theory not long ago and did not see anything that aligned with the hype around bots. [1] There are indeed more bots than humans because of course there are or at least the appearance of. Bots crawl everything linked from popular sites whereas humans only click on things that interest them and even then they do not typically siphon the entire site. There are new bot operators every day due to curiosity and FOMO.

The only thing I saw that could possibly be construed as abusive were some poorly configured RSS bots. Even when my server told the bot that the page would not change for 4 hours the RSS bots would check every 10 minutes meaning they are ignoring the cache-control header. This was entirely harmless, just slightly annoying. The RSS bots are not new. Most of the bots are not even trying to disguise themselves as humans. Most bots are not programmed to parse cache-controls, rel tags or fetch robots.txt meaning they only follow the pirate code. A bot will do what a bot can do.

I was expecting the bots to mirror a couple git repositories I exposed but they did not go deeper than the README.md. None of them. I think this is the same pattern of catastrophization that exists around AI dooming the world and I don't know why it is spreading. I guess it must work or people would not do it.

[1] - https://blawg.nochan.net/b/Internet-Crap/20260522-Maybe-AI-B...

Re: Cloudflare CEO is lying to you about the bot traffic jump

#16

Post by taeric »

reconnecting wrote: post #9
taeric wrote: post #3

I confess a sad assumption that bot traffic is far higher than we have admitted for a long time. Though, maybe we would see different stats specifically to social media sights to astroturf like counts? Certainly feels that we have known for a long time that bots were larger in ad viewing than ad companies wanted to admit.

I don't understand what difference bots make. For me, a website (the public part) is a storefront. People walk down the street and see what's inside — that's the purpose. If something should not be available immediately, that's the private part of the store. I've been monitoring bot traffic on digital platforms for over 10 years. Sure, the crawler share is growing, some even with malicious intentions, and those I det…

For ad views, the concern is specifically that people pay for clicks and views. That that can be so heavily influenced by bot traffic greatly undermines their value.

Same general idea goes for any of the algorithmic driven platforms. The algorithms are ostensibly intended to surface organically discovered things by watching how people interact with things. That they are so susceptible to distortion through bot farms should be a lot more acknowledged than it is. People trust them far more than they should.

There is also a general cost of running things concern. It isn't like it is completely free to execute on bot traffic.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#17

Post by gonzalohm »

simonw wrote: post #11

"Cloudflare CEO is lying" is a bit of an aggressive take when he linked to the exact data so you can see it for yourself - and that's how this article was able to analyze it: https://radar.cloudflare.com/traffic#bot-vs-human Update: I see the problem. Here's the full tweet: https://x.com/eastdakota/status/2062212701414187452 "Thought it would be end of 2027, then early 2027, but agentic traffic growing so fast that b…

The graph only goes back to May 3rd 2026. I guess that was the start of humanity
Post reply on HN