Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devs say AI crawlers dominate traffic, forcing blocks on entire countries

arstechnica.com

Post reply on HN
Print view View original

11–20 of 291 posts

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#11

Post by nosianu »

edoloughlin wrote: post #6

I'm being trite, but if you can detect an AI bot, why not just serve them random data? At least they'll be sharing some of the pain they inflict.

You mean like this?

[2025-03-19] https://blog.cloudflare.com/ai-labyrinth/

> Trapping misbehaving bots in an AI Labyrinth

> Today, we’re excited to announce AI Labyrinth, a new mitigation approach that uses AI-generated content to slow down, confuse, and waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#12

Post by burkaman »

grotorea wrote: post #3

Is this stuff only affecting the not for profit web? What are the for profit sites doing? I haven't seen Anubis around the web elsewhere. Are we just going to get more and tighter login walls and send everything into the deep web?

For profit sites are making deals directly with the AI companies so they can get some more of that profit.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#13

Post by true_blue »

xena wrote: post #4

Wow it is so surreal to see a project of mine on Ars Technica! It's such an honor!

On the few sites I've seen using it so far, it's been a more pleasant (and cuter) experience for me than the captchas I'd probably get otherwise. good work!

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#14

Post by ggm »

noirscape wrote: post #10
ggm wrote: post #2

Entire country blocks are lazy, and pragmatic. The US armed forces at one point blocked AU/NZ on 202/8 and 203/8 on a misunderstanding about packets from China, also from these blocks. Not so useful for military staff seconded into the region seeking to use public internet to get back to base. People need to find better methods. And, crawlers need to pay a stupidity tax or be regulated (dirty word in the tech sector)

They can absolutely work if you aren't expecting any traffic from those countries whatsoever. I don't expect any international calls... ever, so I block international calling numbers on my phone (since they are always spam calls) and it cuts down on the overwhelming majority of them. Don't see why that couldn't apply to websites either.

Sure. Absolutely works. Right up until it doesn't. I think the MIL was the wrong people to assume "we will never need packets from these network blocks"

The other thing is that phone numbers follow a numbering scheme where +1 is north america and +64 is NZ. Its easy to know the longterm geographic consequence of your block, modulo faked out CLID. IP packets don't follow this logic and Amazon can deploy AWS nodes with IPs acquired in Asia, in any DC they like. The smaller hosting companies don't say the IP range they route for banks have no pornographers on them.

It's really not sensible to use IP blocks except for the very specific cases like yours: "I never terminate international calls" is the NAT of firewalls: "I don't want incoming packets from strangers" sure the cheapest path is to block entire swathes of IPv4 and IPv6. But if you are in general service delivery, that rarely works. If you ran a business doing trade in China, you'd remove that block immediately.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#15

Post by kragen »

noirscape wrote: post #10
ggm wrote: post #2

Entire country blocks are lazy, and pragmatic. The US armed forces at one point blocked AU/NZ on 202/8 and 203/8 on a misunderstanding about packets from China, also from these blocks. Not so useful for military staff seconded into the region seeking to use public internet to get back to base. People need to find better methods. And, crawlers need to pay a stupidity tax or be regulated (dirty word in the tech sector)

They can absolutely work if you aren't expecting any traffic from those countries whatsoever. I don't expect any international calls... ever, so I block international calling numbers on my phone (since they are always spam calls) and it cuts down on the overwhelming majority of them. Don't see why that couldn't apply to websites either.

It depends on whether the information on the website is supposed to be publicly available or not. "This information is publicly available except to people from Israel" sends a really terrible message.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#16

Post by charcircuit »

noirscape wrote: post #9
edoloughlin wrote: post #6

I'm being trite, but if you can detect an AI bot, why not just serve them random data? At least they'll be sharing some of the pain they inflict.

Bandwidth isn't free, not at the volume these crawlers scrape at; serving them random data (for example by leading them down an endless tarpit of links that no human would end up visiting) would still incur bandwidth fees. Also it's not identifiable AI bot traffic that's detected (they mask themselves as regular browsers and hop between domestic IP addresses when blocked), it's just really obviously AI scraper traffi…

>Bandwidth isn't free

Via peering agreements it is.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#17

Post by MathMonkeyMan »

tedunangst wrote: post #8

> It remains unclear why these companies don't adopt more collaborative approaches and, at a minimum, rate-limit their data harvesting runs so they don't overwhelm source websites. If the target goes down after you scrape it, that's a feature.

Why? What is the goal of a scraper, and how does disabling the source of the data benefit them?

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#18

Post by TuxMark5 »

MathMonkeyMan wrote: post #17
tedunangst wrote: post #8

> It remains unclear why these companies don't adopt more collaborative approaches and, at a minimum, rate-limit their data harvesting runs so they don't overwhelm source websites. If the target goes down after you scrape it, that's a feature.

Why? What is the goal of a scraper, and how does disabling the source of the data benefit them?

I guess one could make a point that competition will no longer have the access to the scraped data.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#19

Post by ipaddr »

I've had a number of content sites I've shut down a few sites in the last few days because of the toll these aggressive AI bots. Alexa seems like the worst.

These were created 20 years ago and updated over the years. I use to get traffic but that's been slowed to 1,000 or less legitimate visitors over the last year. But now I have to deal with server down emails caused by these aggressive bots that don't respect the robots file.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#20

Post by randmeerkat »

MathMonkeyMan wrote: post #17
tedunangst wrote: post #8

> It remains unclear why these companies don't adopt more collaborative approaches and, at a minimum, rate-limit their data harvesting runs so they don't overwhelm source websites. If the target goes down after you scrape it, that's a feature.

Why? What is the goal of a scraper, and how does disabling the source of the data benefit them?

> Why? What is the goal of a scraper, and how does disabling the source of the data benefit them?

The next scraper doesn’t get the data. People don’t realize we’re not compute limited for ai, we’re data limited. What we’re watching is the “data war”.

Post reply on HN