AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders • Hacker News Board

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#11

Post by nromiun » Thu, Aug 21, 2025, 1:07 PM UTC

internet_points wrote: post #10

They mention anubis, cloudflare, robots.txt – does anyone have experiences with how much any of them help?

CDNs like Cloudflare are the best. Anubis is a rate limitor for small websites where you can't or won't use CDNs like Cloudflare. I have used Cloudflare on several medium sized websites and it works really well.

Anubis's creator says the same thing:

> In most cases, you should not need this and can probably get by using Cloudflare to protect a given origin. However, for circumstances where you can't or won't use Cloudflare, Anubis is there for you.

Source: https://github.com/TecharoHQ/anubis

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#12

Post by snowwrestler » Thu, Aug 21, 2025, 1:18 PM UTC

bgwalter wrote: post #3

breakyerself wrote: post #2

There's so much bullshit on the internet how do they make sure they're not training on nonsense?

Much of it is not training. The LLMs fetch webpages for answering current questions, summarize or translate a page at the user's request etc. Any bot that answers daily political questions like Grok has many web accesses per prompt.

While it’s true that chatbots fetch information from websites in response to requests, the load from those requests is tiny compared to the volume of requests indexing content to build training corpuses.

The reason is that user requests are similar to other web traffic because they reflect user interest. So those requests will mostly hit content that is already popular, and therefore well-cached.

Corpus-building crawlers do not reflect current user interest and try to hit every URL available. As a result these hit URLs that are mostly uncached. That is a much heavier load.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#13

Post by andai » Thu, Aug 21, 2025, 1:25 PM UTC

pm215 wrote: post #7

shinycode wrote: post #4

In the same time it’s so practical to ask a question and it opens 25 pages to search and summarize the answer. Before that’s more or less what I was trying to do by hand. Maybe not 25 websites because of crap SEO the top 10 contains BS content so I curated the list but the idea is the same no ?

Sure, but if the fetcher is generating "39,000 requests per minute" then surely something has gone wrong somewhere ?

They're not very good at web queries, if you expand the thinking box to see what they're searching for, like half of it is nonsense.

e.g. they'll take an entire sentence the user said and put it in quotes for no reason.

Thankfully search engines started ignoring quotes years ago, so it balances out...

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#14

Post by rco8786 » Thu, Aug 21, 2025, 1:26 PM UTC

OpenAI straight up DoSed a site I manage for my in-laws a few months ago.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#15

Post by rco8786 » Thu, Aug 21, 2025, 1:27 PM UTC

shinycode wrote: post #4

In the same time it’s so practical to ask a question and it opens 25 pages to search and summarize the answer. Before that’s more or less what I was trying to do by hand. Maybe not 25 websites because of crap SEO the top 10 contains BS content so I curated the list but the idea is the same no ?

My personal experience is that OpenAI's crawler was hitting a very, very low traffic website I manage 10s of 1000s of times a minute non-stop. I had to block it from Cloudflare.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#16

Post by hereme888 » Thu, Aug 21, 2025, 1:30 PM UTC

I'm absolutely pro AI-crawlers. The internet is so polluted with garbage, compliments of marketing. My AI agent should find and give me concise and precise answers.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#17

Post by danaris » Thu, Aug 21, 2025, 1:40 PM UTC

breakyerself wrote: post #2

There's so much bullshit on the internet how do they make sure they're not training on nonsense?

I mean...they don't. That's part of the problem with "AI answers" and such.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#18

Post by danaris » Thu, Aug 21, 2025, 1:42 PM UTC

rco8786 wrote: post #15

shinycode wrote: post #4

In the same time it’s so practical to ask a question and it opens 25 pages to search and summarize the answer. Before that’s more or less what I was trying to do by hand. Maybe not 25 websites because of crap SEO the top 10 contains BS content so I curated the list but the idea is the same no ?

My personal experience is that OpenAI's crawler was hitting a very, very low traffic website I manage 10s of 1000s of times a minute non-stop. I had to block it from Cloudflare.

Same here.

I run a very small browser game (~120 weekly users currently), and until I put its Wiki (utterly uninteresting to anyone who doesn't already play the game) behind a login-wall, the bots were causing massive amounts of spurious traffic. Due to some of the Wiki's data coming live from the game through external data feeds, the deluge of bots actually managed to crash the game several times, necessitating a restart of the MariaDB process.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#19

Post by exasperaited » Thu, Aug 21, 2025, 1:50 PM UTC

Xe Iaso is my spirit animal.

> "I don't know what this actually gives people, but our industry takes great pride in doing this"

> "unsleeping automatons that never get sick, go on vacation, or need to be paid health insurance that can produce output that superficially resembles the output of human employees"

> "This is a regulatory issue. The thing that needs to happen is that governments need to step in and give these AI companies that are destroying the digital common good existentially threatening fines and make them pay reparations to the communities they are harming."

<3 <3

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#20

Post by danaris » Thu, Aug 21, 2025, 1:53 PM UTC

8organicbits wrote: post #6

bgwalter wrote: post #3

Earlier quoted context omitted.

Much of it is not training. The LLMs fetch webpages for answering current questions, summarize or translate a page at the user's request etc. Any bot that answers daily political questions like Grok has many web accesses per prompt.

Is an AI chatbot fetching a web page to answer a prompt a 'web scraping bot'? If there is a user actively promoting the LLM, isn't it more of a user agent? My mental model, even before LLMs, was that a human being present changes a bot into a user agent. I'm curious if others agree.

But they're (generally speaking) not being asked for the contents of one specific webpage, fetching that, and summarizing it for the user.

They're going out and scraping everything, so that when they're asked a question, they can pull a plausible answer from their dataset and summarize the page they found it on.

Even the ones that actively go out and search/scrape in response to queries aren't just scraping a single site. At best, they're scraping some subset of the entire internet that they have tagged as being somehow related to the query. So even if what they present to the user is a summary of a single webpage, that is rarely going to be the product of a single request to that single webpage. That request is going to be just one of many, most of which are entirely fruitless for that specific query: purely extra load for their servers, with no gain whatsoever.