Earlier quoted context omitted.
[flagged]
I’m no anime fan, but this sort of judgement on normalcy leaves me with a very sour impression about whoever says this sort of thing.
[flagged]
Devs say AI crawlers dominate traffic, forcing blocks on entire countries
161–170 of 291 posts
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#162I got DoSed by ClaudeBot (Anthropic) just last week. Hitting a website I manage 700,000 times in one month and tripping our bandwidth limit with our hosting provider. What a PITA to have to investigate that, figure it out, block the user agent, and work with hosting provider support to get the limit lifted as a courtesy. Noticed that the ChatGPT bot was 2nd in traffic to this site, just not enough to cause trouble.
I've got claude bot blocked too. It regularly took sites offline and ignored robots.txt. Claude bot is an asshole.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#163Why can't Cloudflare do it? They know in real time which IPs spew millions of scrape requests to various sites, so they can classify them as AI bots and allow site owners to block them.
Cloudflare already offers this feature
https://blog.cloudflare.com/declaring-your-aindependence-blo...
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#164Earlier quoted context omitted.
Not if they hop to a different IP address every few requests. And they generally aren't bothered slow responses. It's not like they have to wait for one request to finish before they make another one (especially if they are making requests from thousands of machines).
You're saying that large companies are hitting individual websites with thousands of unrelated IP addresses?
Yep we've been seeing that on our random small scale site that used to be open (and mostly relevant to a very limited number of people).
It was nice for interested guests to get an impression of what we are doing.
First the AI crawlers came in from foreign countries that could be blocked.
Then they beat down the small server by being very distributed, calling from thousands of ips one or two requests each.
We finally put a stop to it by requiring a login with a message informing people to physically show up to gain access.
Worked fine for over 15 years but AI finally killed it.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#165Earlier quoted context omitted.
From my little understanding, we have a sort of agreement in place with an item called robot.txt that's more or less a hanshake with such scrapers. Of course, the issue is these scrapers are blatantly ignoring robots.txt A license can help as well, but what's a license without enforcement? These companies are simply treating the courts as a cost to do business.
Close, robots.txt was originally for web crawlers, to reduce accidental denial-of-service attacks. It had nothing to do with the scraping (i.e. downloading content and parsing the HTML tags in a programmatic manner).
What do you think a search engine’s crawler bot is doing exactly? I could sure be wrong, but I have a hunch that “downloading content and paraing the HTML tags in a programmatic manner” describes it.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#166Earlier quoted context omitted.
Please remember that Google only needs to be marginally better than the competition. And, of course, their primary biz is ads, not serving great results; that is a distant second priority.
Their biz is ads, but since search is winner takes all they need only be marginally better than the competition... twenty years ago.
> Their biz is ads,
Yea, but, the FTC doesn't want it to be.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#167I have a blog with content about a non tech topic and I had any problem. I am all against AI scrappers, but I didn't notice any change being behind Cloudflare (funny enough, if I ask GPT and Claude about my website, they know it)
They could hit Cloudflare’s caches and run with that. The problems are if your site is dynamic and your CDN has to hit the origin every time.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#168Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#169I think the proper answer is to aim for the bots to get _negative_ utility value from visiting our sites, that is poisoning their well, not just zero value, that is to block them. Did you try to GET a canary page forbidden in robots.txt? Very well, have a bucket load of articles on the benefits of drinking bleach. Is your user-agent too suspicious? No problem, feel free to scrape my insecure code (google "emergent mi…
That’s vaguely what https://blog.cloudflare.com/ai-labyrinth/ is about
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#170I'm being trite, but if you can detect an AI bot, why not just serve them random data? At least they'll be sharing some of the pain they inflict.
Bandwidth isn't free, not at the volume these crawlers scrape at; serving them random data (for example by leading them down an endless tarpit of links that no human would end up visiting) would still incur bandwidth fees. Also it's not identifiable AI bot traffic that's detected (they mask themselves as regular browsers and hop between domestic IP addresses when blocked), it's just really obviously AI scraper traffi…
For Cloudflare, bandwidth is practically free.