Sure, overly spamming websites is shitty behaviour. But blocking AI crawlers hurts you in the end. Guess what will replace SEO in the long run?
So you're willing to pay my hosting bills?
Re-read what I posted and stop projecting.
Devs say AI crawlers dominate traffic, forcing blocks on entire countries
231–240 of 291 posts
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#232Sure, overly spamming websites is shitty behaviour. But blocking AI crawlers hurts you in the end. Guess what will replace SEO in the long run?
>But blocking AI crawlers hurts you in the end. Guess what will replace SEO in the long run? Maybe. But even if that turns out to be true, what good is it for the source website? The "AI" will surely not share any money (or anything else that may help the source website) with the source anyways. Why would they, they already got the content and trained on it.
What good is it? If "AI" doesn't know about you down the line, you won't be discovered. Be it in LLM weights or via crawling (perplexity, jina reader etc.), you won't get any organic traffic. It's not about sharing profits.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#233An aspect I find interesting is that these crawlers are all doing highly redundant work. As in, thousands of crawlers are running around the world, and each crawler may visit the same site and pages multiple times a week.
This seems like an opportunity for a company like Firecrawl, ScrapingBee, etc to offer built-in caching with TTLs so that redundant requests can hit the cache and not contribute to load on the actual site.
Even if each company that operates a crawler cached pages across multiple runs, I'd expect a large improvement in the situation.
For more dynamic pages, this obviously doesn't help. But a lot of the web's content is more static and is being crawled thousands of times.
I built something for my own company that crawls using Playwright and caches in S3/Postgres with a TTL for this purpose.
Does this make sense to anyone else? I'm not sure if I'm missing something that makes this harder than it seems on the surface. (Actual question!)
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#234I got DoSed by ClaudeBot (Anthropic) just last week. Hitting a website I manage 700,000 times in one month and tripping our bandwidth limit with our hosting provider. What a PITA to have to investigate that, figure it out, block the user agent, and work with hosting provider support to get the limit lifted as a courtesy. Noticed that the ChatGPT bot was 2nd in traffic to this site, just not enough to cause trouble.
robots.txt did not work?
I’ve been doing web sites for thirty years, robots.txt is at best a request to polite user agents to respect the server’s desires. None of the malicious crawlers respect it. None of the AI crawlers respect it.
I’ve resorted to returning xml and zip bombs in canary pages. At best it slows them down until I block their network.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#235Earlier quoted context omitted.
This relies a lot on being able to detect bots. Everything you said could be easily bypassed with a small to moderate amount of effort on the side of crawler's creators. Distinguishing genuine traffic has always been hard and it will not get easier in the age of AI.
But the very comment you answered explains how to do it: a page forbidden in robots.txt. Does this method need explanation why it's ideal for sorting humans and google, from malicious crawlers?
robots.txt is a somewhat useful tool for keeping search engines in line, because it's rather easy to prove that a search engine ignores robots.txt: when a noindex page shows up in SERPs. This evidence trail does not exist for AI crawlers.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#236An aspect I find interesting is that these crawlers are all doing highly redundant work. As in, thousands of crawlers are running around the world, and each crawler may visit the same site and pages multiple times a week. This seems like an opportunity for a company like Firecrawl, ScrapingBee, etc to offer built-in caching with TTLs so that redundant requests can hit the cache and not contribute to load on the actua…
What does the user agent oook like for if you wanted to crawl xeiaso.net?
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#237An aspect I find interesting is that these crawlers are all doing highly redundant work. As in, thousands of crawlers are running around the world, and each crawler may visit the same site and pages multiple times a week. This seems like an opportunity for a company like Firecrawl, ScrapingBee, etc to offer built-in caching with TTLs so that redundant requests can hit the cache and not contribute to load on the actua…
I have considered this before, but then if the content can be cached why wouldn't the website just do this themselves?
They have the incentive, it is relatively easy and I don't think there's a huge benefit to centralisation (especially since it will basically be centralised to one of the big providers of caching anyways)
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#238One of the main peddlers for residential proxies is Israel, see BrightData.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#239Earlier quoted context omitted.
We're affected by this. The only thing that would realistically work is the first suggestion. The most unscrupulous AI crawlers distribute their inhuman request rate over dozens of IPs, so every IP just makes 1-2 requests in total. And they use real-world browser user agents, so blocking those could lock out real users. However, sometimes they claim to be using really old Chrome versions, so I feel less bad about loc…
> dozens of IPs, so every IP just makes 1-2 requests in total Dozens of IPs making 1-2 requests per IP hardly seems like something to spend time worrying about.
That's probably per day, per bot. Now how does it look when there are thousands of bots? In most cases I think you're right, but I can also see how it can add up.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#240An aspect I find interesting is that these crawlers are all doing highly redundant work. As in, thousands of crawlers are running around the world, and each crawler may visit the same site and pages multiple times a week. This seems like an opportunity for a company like Firecrawl, ScrapingBee, etc to offer built-in caching with TTLs so that redundant requests can hit the cache and not contribute to load on the actua…
I have considered this before, but then if the content can be cached why wouldn't the website just do this themselves? They have the incentive, it is relatively easy and I don't think there's a huge benefit to centralisation (especially since it will basically be centralised to one of the big providers of caching anyways)
I'm definitely with you that sites should be leveraging CDNs and similar. But I get that many don't want to do any work to support bots that they don't want to exist in the first place.
To me it seems like the companies actually doing the crawling have an incentive to leverage centralized caching. It makes their own crawling faster (since hitting the cache is much faster than using Playwright etc to load the page) and it reduces the impact on all these sites. Which would then also decrease the impact of this whole bot situation overall.