Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare CEO is lying to you about the bot traffic jump

flyingpenguin.com

Post reply on HN
Print view View original

91–100 of 149 posts

Re: Cloudflare CEO is lying to you about the bot traffic jump

#91

Post by Bender »

Symbiote wrote: post #80
Bender wrote: post #15

I tested this theory not long ago and did not see anything that aligned with the hype around bots. [1] There are indeed more bots than humans because of course there are or at least the appearance of . Bots crawl everything linked from popular sites whereas humans only click on things that interest them and even then they do not typically siphon the entire site. There are new bot operators every day due to curiosity…

My employer's site was recording 1,500 requests per second from a single AI bot earlier this week. The requests came from 2.4 million different IPs at the time I looked, between 1-2 requests from each IP, most likely all were unique URLs. That single bot was 55% of traffic. This kind of crawling pushes us to (sometimes beyond) the limit of our capacity. I have also seen thousands of requests per hour from the IP to a…

the site is being specifically targeted by the AI bot developers

As I was reading your comment it sounded like a targeted attack. I think you are right that it was targeted. I assume you have done research on what content could be rate limited by URI target vs. source IP and give people a message saying content temporarily unavailable due to AI bot attack?

Is the concern that your site is being DDoS'd or that they are reselling your copyrighted material? If reselling I would get corporate lawyers involved and seek damages I am not a lawyer. Feds could subpoena some of the providers for identity of the attackers.

If the concern is DDoS have your team done any analysis of the clients to see what is in common? Based on the number of IP address you are talking about I assume it must be from wireless carriers. Have you looked at TCPSYN TTL and other characteristics? If there is anything in common those connections could be routed internally to another listener that has tighter rate limits meaning that perhaps cellular users could find some content not available until the bots go away or they randomly get one of a dozen different captchas or random javascript puzzles to access each document until the storm subsides. The puzzles could probably be regenerated hourly by AI to keep the attackers on their toes. Another option would be to require an account to access the documents and limit the number of documents each account can download per hour and / or day and / or week then add more friction to account creation or limit account creation to address space of countries you do business with after blocking most proxies and VPN providers.

Another option to limit the blast zone of an attack is to block countries that one does not do business in but that depends on your business model.

CDN's like Cloudflare are not doing anything magic. If they can block the bots so can just about anyone else. Without seeing samples of the attacks I could not make many more suggestions.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#92

Post by reconnecting »

kev009 wrote: post #90
reconnecting wrote: post #86

Earlier quoted context omitted.

Did you check IP addresses, are they all from AS32934?

Yes 57.141.0.42 - - [05/Jun/2026:19:50:19 +0000] "GET /mid/a017bc62-0982-42db-8403-241d69da8d0f@alexander-goetzenstein.my-fqdn.de HTTP/2.0" 303 0 "-" "meta-externalagent/1.1 (+ https://developers.facebook.com/docs/sharing/webmasters/craw... )" 57.141.0.48 - - [05/Jun/2026:19:50:22 +0000] "GET /group/comp.os.linux.advocacy/a/a236f5a5-63a4-4982-8bb6-07ffc684201b@googlegroups.com HTTP/2.0" 200 34838 "-" "meta-externalag…

And assume you have

User-agent: meta-externalagent

Disallow: /

Re: Cloudflare CEO is lying to you about the bot traffic jump

#93

Post by b65e8bee43c2ed0 »

jimrandomh wrote: post #31
nothrows wrote: post #20

Cloudflare is junk. Their entire billion dollar service can't distinguish my (DAILY) GET request to mainstream news sites from bot traffic, nothing they say or do is of any value. I've had the same IP for decades.

Have you checked your IP address's reputation with a service such as ipqualityscore.com? If cloudflare thinks your traffic is bot traffic, it's likely that there is bot traffic you don't know about coming from your IP, either from a compromised device on your network or a sketchy VPN product.

Google and Cloudflare mark whole ISPs as "sketchy" and there's fuck all you can do about it.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#95

Post by speak_plainly »

I host my website on Cloudflare and looking at my stats, I can confirm that bot traffic is not only up, it's totally insane.

There is nothing on the site, just my name and an email account and it should have no traffic at all.

Instead in the last 30 days I've had 26K requests. 93% resulting in a 4xx error as most of these bots seem to be looking for vulnerabilities in various platforms like WordPress.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#96

Post by kev009 »

reconnecting wrote: post #92
kev009 wrote: post #90

Earlier quoted context omitted.

Yes 57.141.0.42 - - [05/Jun/2026:19:50:19 +0000] "GET /mid/a017bc62-0982-42db-8403-241d69da8d0f@alexander-goetzenstein.my-fqdn.de HTTP/2.0" 303 0 "-" "meta-externalagent/1.1 (+ https://developers.facebook.com/docs/sharing/webmasters/craw... )" 57.141.0.48 - - [05/Jun/2026:19:50:22 +0000] "GET /group/comp.os.linux.advocacy/a/a236f5a5-63a4-4982-8bb6-07ffc684201b@googlegroups.com HTTP/2.0" 200 34838 "-" "meta-externalag…

And assume you have User-agent: meta-externalagent Disallow: /

They don't obey *, they don't get their own entry. I'd rather just poison their data, it's a well known behavior from them.

https://www.reddit.com/r/webdev/comments/1sdzd1q/metas_ai_cr...

Re: Cloudflare CEO is lying to you about the bot traffic jump

#97

Post by Symbiote »

Bender wrote: post #91
Symbiote wrote: post #80

Earlier quoted context omitted.

My employer's site was recording 1,500 requests per second from a single AI bot earlier this week. The requests came from 2.4 million different IPs at the time I looked, between 1-2 requests from each IP, most likely all were unique URLs. That single bot was 55% of traffic. This kind of crawling pushes us to (sometimes beyond) the limit of our capacity. I have also seen thousands of requests per hour from the IP to a…

the site is being specifically targeted by the AI bot developers As I was reading your comment it sounded like a targeted attack. I think you are right that it was targeted. I assume you have done research on what content could be rate limited by URI target vs. source IP and give people a message saying content temporarily unavailable due to AI bot attack? Is the concern that your site is being DDoS'd or that they ar…

It's not copyright data (academia / library stuff), so the concern is the DDoS. If researches write to us, we send them a database dump or export, but only one AI company has ever written.

So far there's always been some pattern to allow a block/challenge, e.g. user agent, JA3 / JA4, ASN. (I haven't looked at TCP SYN TTL before.) Usually the IPs are 80% or so in one country (e.g. Brazil, US, Vietnam or India) with the rest all over the world, mostly consumer ISPs although I haven't distinguished between fixed line and mobile.

We tried Cloudflare for a couple of months, on a paid plan, which I think blocked many of the non-distributed crawlers, but didn't help much with these distributed ones.

Meanwhile we have been reducing the cost of rendering the pages.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#98

Post by burnte »

simonw wrote: post #64
burnte wrote: post #44

Earlier quoted context omitted.

It's aggressive but true. Spam traffic exceeded real email back in 2003. He's not even in the right DECADE to declare a first for non-human traffic. It's pure marketing BS that is only remotely true if you accept the conditions he goes to lengths to hide.

What lengths did he go to in order to hide them?

Misleading headline statement, no clarifying statements later that is restricted to HTTP traffic, graphs chosen to support the misleading message. Putting out the truth behind an asterisk isn't being honest.

Re: Cloudflare CEO is lying to you about the bot traffic jump

#99

Post by Symbiote »

reconnecting wrote: post #92
kev009 wrote: post #90

Earlier quoted context omitted.

Yes 57.141.0.42 - - [05/Jun/2026:19:50:19 +0000] "GET /mid/a017bc62-0982-42db-8403-241d69da8d0f@alexander-goetzenstein.my-fqdn.de HTTP/2.0" 303 0 "-" "meta-externalagent/1.1 (+ https://developers.facebook.com/docs/sharing/webmasters/craw... )" 57.141.0.48 - - [05/Jun/2026:19:50:22 +0000] "GET /group/comp.os.linux.advocacy/a/a236f5a5-63a4-4982-8bb6-07ffc684201b@googlegroups.com HTTP/2.0" 200 34838 "-" "meta-externalag…

And assume you have User-agent: meta-externalagent Disallow: /

I have observed the same from Meta's crawler.

  User-agent: *
  Disallow: /
on e.g. our preproduction site, Meta is the only big-tech crawler that accesses it, at least with an honest user agent. (Meta also accesses disallowed paths on the production site.)

Re: Cloudflare CEO is lying to you about the bot traffic jump

#100

Post by Bender »

Symbiote wrote: post #97
Bender wrote: post #91

Earlier quoted context omitted.

the site is being specifically targeted by the AI bot developers As I was reading your comment it sounded like a targeted attack. I think you are right that it was targeted. I assume you have done research on what content could be rate limited by URI target vs. source IP and give people a message saying content temporarily unavailable due to AI bot attack? Is the concern that your site is being DDoS'd or that they ar…

It's not copyright data (academia / library stuff), so the concern is the DDoS. If researches write to us, we send them a database dump or export, but only one AI company has ever written. So far there's always been some pattern to allow a block/challenge, e.g. user agent, JA3 / JA4, ASN. (I haven't looked at TCP SYN TTL before.) Usually the IPs are 80% or so in one country (e.g. Brazil, US, Vietnam or India) with th…

Usually the IPs are 80% or so in one country (e.g. Brazil, US, Vietnam or India)

It sounds like there is the option to at least reduce the load by up to 80%. That's at least a start.

This repo [1] is not perfect but it's a start. I would disable IPv6 access to the site after removing the IPv6 DNS records and waiting a day so that attackers are forced onto IPv4 and clone this repo [1] or use one of the GeoIP databases to limit access from specific countries.

That repo also contains known proxies. That may account for another percentage of that remaining 20%.

As the content is academic in nature I don't know how your team feels about blocking Tor, but there is also a list of many (not all) of the last 30 days of Tor exit nodes in that repo.

The blocks would not have to be permanent, just enabled during the storms if your team so desired.

    # verify you have console access to the server before doing things like this:
    for BadIP in $(grep -Ev ^# ipip_country_br.netset); do ip route add blackhole "${BadIP}" 2>/dev/null;done
Example of country IP addresses for Brazil [2]

As this is academia I don't know if there is a concept of service level agreement or a promise of availability, but during attacks the requests to specific URL's could be redirected to a static pre-compressed landing page served out of memory that says "Access to these documents limited during AI bot attacks, here is where to request a full download instead: "

I forgot to mention, many of the AI bots are limited to HTTP/1.1. During an attack your server could redirect those request to a static page as well. Curious if you can tell by your access logs if the majority of the attack is HTTP/1.1 or HTTP/2.0. Most botters are too lazy to implement HTTP/2.0. Real browser clients use HTTP/2.0. Example of using HTTP/1.1 and HTTP/2.0

    # Fist use HTTP/2.0 with headers I expect.
    curl -i --http2 \
     -A "Mozilla/5.0 (X11; Linux x86_64; rv:151.0) Gecko/20100101 Firefox/151.0" \
     -H "sec-fetch-mode: navigate" \
     -H "accept-language: en-US,en;q=0.9" \
     --url 'https://blawg.nochan.net/.env'

    # Then HTTP/1.1 for the lulz, pay attention to the Location redirect
    curl -i --http1.1 \
     -A "Mozilla/5.0 (X11; Linux x86_64; rv:151.0) Gecko/20100101 Firefox/151.0" \
    -H "sec-fetch-mode: navigate" \
    -H "accept-language: en-US,en;q=0.9" \
    --url 'https://blawg.nochan.net/.env'
Some bots also do not hide that they are bots in that they say they are bot in their user-agent client header. That would be too easy so I doubt that is the case.

[1] - https://github.com/firehol/blocklist-ipsets/

[2] - https://github.com/firehol/blocklist-ipsets/blob/master/ipip...

Post reply on HN