OpenAI straight up DoSed a site I manage for my in-laws a few months ago.
What is it about? I'm curious what kinds of things people ask that floods sites.
AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
21–30 of 148 posts
Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#22OpenAI straight up DoSed a site I manage for my in-laws a few months ago.
What is it about? I'm curious what kinds of things people ask that floods sites.
I suppose that they just keep referring to the website in their chats, and probably they have selected the search function, so before every reply, the crawler hits the website
Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#23Earlier quoted context omitted.
Much of it is not training. The LLMs fetch webpages for answering current questions, summarize or translate a page at the user's request etc. Any bot that answers daily political questions like Grok has many web accesses per prompt.
While it’s true that chatbots fetch information from websites in response to requests, the load from those requests is tiny compared to the volume of requests indexing content to build training corpuses. The reason is that user requests are similar to other web traffic because they reflect user interest. So those requests will mostly hit content that is already popular, and therefore well-cached. Corpus-building craw…
But surely there aren't thousands of new corpuses built every minute.
Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#24They mention anubis, cloudflare, robots.txt – does anyone have experiences with how much any of them help?
robots.txt is obviously only effective against well-behaved bots. OpenAI etc are usually well behaved, but there's at least one large network of rogue scraping bots that ignores robots.txt, fakes the user-agent (usually to some old Chrome version) and cycles through millions of different residential proxy IPs. On my own sites, this network is by far the worst offender and the "well-behaved" bots like OpenAI are barely noticeable.
To stop malicious bots like this, Cloudflare is a great solution if you don't mind using it (you can enable a basic browser check for all users and all pages, or write custom rules to only serve a check to certain users or on certain pages). If you're not a fan of Cloudflare, Anubis works well enough for now if you don't mind the branding.
Here's the cloudflare rule I currently use (vast majority of bot traffic originates from these countries):
ip.src.continent in {"AF" "SA"} or
ip.src.country in {"CN" "HK" "SG"} or
ip.src.country in {"AE" "AO" "AR" "AZ" "BD" "BR" "CL" "CO" "DZ" "EC" "EG" "ET" "ID" "IL" "IN" "IQ" "JM" "JO" "KE" "KZ" "LB" "MA" "MX" "NP" "OM" "PE" "PK" "PS" "PY" "SA" "TN" "TR" "TT" "UA" "UY" "UZ" "VE" "VN" "ZA"} or
ip.src.asnum in {28573 45899 55836}Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#25They mention anubis, cloudflare, robots.txt – does anyone have experiences with how much any of them help?
CloudFlare's Super Bot Fight Mode completely killed the surge in bot traffic for my large forum.
Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#26I run a symbol server, as in, PDB debug symbol server. Amazon's crawler and a few others love requesting the ever loving shit out of it for no obvious reason. Especially since the files are binaries.
I just set a rate-limit in cloudflare because no legitimate symbol server user will ever be excessive.
Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#27In the same time it’s so practical to ask a question and it opens 25 pages to search and summarize the answer. Before that’s more or less what I was trying to do by hand. Maybe not 25 websites because of crap SEO the top 10 contains BS content so I curated the list but the idea is the same no ?
My personal experience is that OpenAI's crawler was hitting a very, very low traffic website I manage 10s of 1000s of times a minute non-stop. I had to block it from Cloudflare.
Where is caching breaking so badly that this is happening? Are OpenAI failing to use etags or honour cache validity?
Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#28Isn't there a class action lawsuit coming from all this? I see a bunch of people here indicating these scrapers are costing real money to people who host even small niche sites.
Is the reason these large companies don't care because they are large enough to hide behind a bunch of lawyers?
Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#29Earlier quoted context omitted.
My personal experience is that OpenAI's crawler was hitting a very, very low traffic website I manage 10s of 1000s of times a minute non-stop. I had to block it from Cloudflare.
Where is caching breaking so badly that this is happening? Are OpenAI failing to use etags or honour cache validity?
Their crawler is vibe-coded.
Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#30Earlier quoted context omitted.
Much of it is not training. The LLMs fetch webpages for answering current questions, summarize or translate a page at the user's request etc. Any bot that answers daily political questions like Grok has many web accesses per prompt.
While it’s true that chatbots fetch information from websites in response to requests, the load from those requests is tiny compared to the volume of requests indexing content to build training corpuses. The reason is that user requests are similar to other web traffic because they reflect user interest. So those requests will mostly hit content that is already popular, and therefore well-cached. Corpus-building craw…
Why would the Register point out Meta and OpenAI as the worst offenders? I'm sure they do not continuously build new corpuses every day. It is probably the search function, as mentioned in the top comments.