Devs say AI crawlers dominate traffic, forcing blocks on entire countries
1–10 of 291 posts
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#2Entire country blocks are lazy, and pragmatic. The US armed forces at one point blocked AU/NZ on 202/8 and 203/8 on a misunderstanding about packets from China, also from these blocks. Not so useful for military staff seconded into the region seeking to use public internet to get back to base.
People need to find better methods. And, crawlers need to pay a stupidity tax or be regulated (dirty word in the tech sector)
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#3Is this stuff only affecting the not for profit web? What are the for profit sites doing? I haven't seen Anubis around the web elsewhere. Are we just going to get more and tighter login walls and send everything into the deep web?
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#4Wow it is so surreal to see a project of mine on Ars Technica! It's such an honor!
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#5Is this stuff only affecting the not for profit web? What are the for profit sites doing? I haven't seen Anubis around the web elsewhere. Are we just going to get more and tighter login walls and send everything into the deep web?
I think we killed the old web. We'll see new ways of communicating, publishing, and gathering over the internet. It's sad, but it's also exciting.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#6I'm being trite, but if you can detect an AI bot, why not just serve them random data? At least they'll be sharing some of the pain they inflict.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#7I'm being trite, but if you can detect an AI bot, why not just serve them random data? At least they'll be sharing some of the pain they inflict.
You can detect the patterns in aggregate. You can't detect it easily at an individual request level.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#8> It remains unclear why these companies don't adopt more collaborative approaches and, at a minimum, rate-limit their data harvesting runs so they don't overwhelm source websites.
If the target goes down after you scrape it, that's a feature.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#9I'm being trite, but if you can detect an AI bot, why not just serve them random data? At least they'll be sharing some of the pain they inflict.
Bandwidth isn't free, not at the volume these crawlers scrape at; serving them random data (for example by leading them down an endless tarpit of links that no human would end up visiting) would still incur bandwidth fees.
Also it's not identifiable AI bot traffic that's detected (they mask themselves as regular browsers and hop between domestic IP addresses when blocked), it's just really obviously AI scraper traffic in aggregate: other mass crawlers have no benefit from bringing down their host sites, except for AI.
A search engine has nothing if it brings down the site they're scraping (and has everything to gain from identifying itself as a search engine to try and get favorable request speeds - the only thing they'd need to check is if the site in question isn't serving different data, but that's much cheaper), same with an archive scraper and those two are pretty much the main examples I can think of for most scraping traffic.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#10Entire country blocks are lazy, and pragmatic. The US armed forces at one point blocked AU/NZ on 202/8 and 203/8 on a misunderstanding about packets from China, also from these blocks. Not so useful for military staff seconded into the region seeking to use public internet to get back to base. People need to find better methods. And, crawlers need to pay a stupidity tax or be regulated (dirty word in the tech sector)
They can absolutely work if you aren't expecting any traffic from those countries whatsoever.
I don't expect any international calls... ever, so I block international calling numbers on my phone (since they are always spam calls) and it cuts down on the overwhelming majority of them. Don't see why that couldn't apply to websites either.