The irony is that the people and employees of the AI companies will vehemently defend the morality of capitalism, private property and free markets.
Their robber baron behavior reveals their true values and the reality of capitalism.
Devs say AI crawlers dominate traffic, forcing blocks on entire countries
41–50 of 291 posts
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#42I'm being trite, but if you can detect an AI bot, why not just serve them random data? At least they'll be sharing some of the pain they inflict.
You can detect the patterns in aggregate. You can't detect it easily at an individual request level.
In short if you get several million requests and expect to only get 100 you won't know which are the real requests and which are the AI ones - but it is obvious that the vast majority are AI.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#43I'm being trite, but if you can detect an AI bot, why not just serve them random data? At least they'll be sharing some of the pain they inflict.
Bandwidth isn't free, not at the volume these crawlers scrape at; serving them random data (for example by leading them down an endless tarpit of links that no human would end up visiting) would still incur bandwidth fees. Also it's not identifiable AI bot traffic that's detected (they mask themselves as regular browsers and hop between domestic IP addresses when blocked), it's just really obviously AI scraper traffi…
Reverse Slowloris?
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#44I've had a number of content sites I've shut down a few sites in the last few days because of the toll these aggressive AI bots. Alexa seems like the worst. These were created 20 years ago and updated over the years. I use to get traffic but that's been slowed to 1,000 or less legitimate visitors over the last year. But now I have to deal with server down emails caused by these aggressive bots that don't respect the…
> Alexa seems like the worst. Many of the bots disguise themselves as coming from Amazon or other big company. Amazon has a page where you can check some details to see if it’s really their crawler or someone imitating it.
yup, actually most that I've seen are impersonating amazon
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#45I'm being trite, but if you can detect an AI bot, why not just serve them random data? At least they'll be sharing some of the pain they inflict.
You skipped the last section "Tarpits and labyrinths: The growing resistance" of the article.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#46Earlier quoted context omitted.
Why? What is the goal of a scraper, and how does disabling the source of the data benefit them?
> Why? What is the goal of a scraper, and how does disabling the source of the data benefit them? The next scraper doesn’t get the data. People don’t realize we’re not compute limited for ai, we’re data limited. What we’re watching is the “data war”.
at this point we're _good data_ limited, which has little to do with scraping.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#47Earlier quoted context omitted.
It's funny how every time this topic comes up someone says "I've had this happen and x is the worst" With x being any of the big AI providers. Just a couple minutes ago I read the same in another thread and it was Anthropic. A couple weeks back it was Meta. My conclusion is that they're all equally terrible then.
All of the crawlers present themselves as being from one of the major companies, even if they’re not. Setting user-agent headers is easy.
at the same time, all the AI providers have some kind of web based AI agent, so let snot pretend they're crafting their services in care of other peoples websites.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#48Wow it is so surreal to see a project of mine on Ars Technica! It's such an honor!
You're well into https://refactoringenglish.com/tools/hn-popularity/ so enjoy the fame!
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#49I'm being trite, but if you can detect an AI bot, why not just serve them random data? At least they'll be sharing some of the pain they inflict.
Bandwidth isn't free, not at the volume these crawlers scrape at; serving them random data (for example by leading them down an endless tarpit of links that no human would end up visiting) would still incur bandwidth fees. Also it's not identifiable AI bot traffic that's detected (they mask themselves as regular browsers and hop between domestic IP addresses when blocked), it's just really obviously AI scraper traffi…
arnt a lot of these bots now actively loading javascript? you could just load a simple script that does the job .
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#50Earlier quoted context omitted.
It sends a great message to crack down on these companies, as long as you mention why it’s blocked.
"You're cut off from access to knowledge because you live in the same country as AI researchers"?
"researchers" is like ignoring the whole Capitalists. We don't call them script researchers. We call them script kiddies.
There's the whole other side of these AI researchers, and thats just slop artisans.