AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
1–10 of 148 posts
Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#2There's so much bullshit on the internet how do they make sure they're not training on nonsense?
Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#3There's so much bullshit on the internet how do they make sure they're not training on nonsense?
Much of it is not training. The LLMs fetch webpages for answering current questions, summarize or translate a page at the user's request etc.
Any bot that answers daily political questions like Grok has many web accesses per prompt.
Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#4In the same time it’s so practical to ask a question and it opens 25 pages to search and summarize the answer. Before that’s more or less what I was trying to do by hand. Maybe not 25 websites because of crap SEO the top 10 contains BS content so I curated the list but the idea is the same no ?
Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#5There's so much bullshit on the internet how do they make sure they're not training on nonsense?
By paying a pretty penny for non bullshit data (Scale Ai). That along with Nvidia are the shovels in this gold rush.
Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#6There's so much bullshit on the internet how do they make sure they're not training on nonsense?
Much of it is not training. The LLMs fetch webpages for answering current questions, summarize or translate a page at the user's request etc. Any bot that answers daily political questions like Grok has many web accesses per prompt.
Is an AI chatbot fetching a web page to answer a prompt a 'web scraping bot'? If there is a user actively promoting the LLM, isn't it more of a user agent? My mental model, even before LLMs, was that a human being present changes a bot into a user agent. I'm curious if others agree.
Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#7In the same time it’s so practical to ask a question and it opens 25 pages to search and summarize the answer. Before that’s more or less what I was trying to do by hand. Maybe not 25 websites because of crap SEO the top 10 contains BS content so I curated the list but the idea is the same no ?
Sure, but if the fetcher is generating "39,000 requests per minute" then surely something has gone wrong somewhere ?
Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#8In the same time it’s so practical to ask a question and it opens 25 pages to search and summarize the answer. Before that’s more or less what I was trying to do by hand. Maybe not 25 websites because of crap SEO the top 10 contains BS content so I curated the list but the idea is the same no ?
Sure, but if the fetcher is generating "39,000 requests per minute" then surely something has gone wrong somewhere ?
Even if it is generating 39k req/minute I would expect most of the pages already be locally cached by Meta, or served statically by their respective hosts. We have been working hard on catching websites and it has been a solved problem for the last decade or so.
Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#9Earlier quoted context omitted.
Much of it is not training. The LLMs fetch webpages for answering current questions, summarize or translate a page at the user's request etc. Any bot that answers daily political questions like Grok has many web accesses per prompt.
Is an AI chatbot fetching a web page to answer a prompt a 'web scraping bot'? If there is a user actively promoting the LLM, isn't it more of a user agent? My mental model, even before LLMs, was that a human being present changes a bot into a user agent. I'm curious if others agree.
The Register calls them "fetchers". They still reproduce the content of the original website without the website gaining anything but additional high load.
I'm not sure how many websites are searched and discarded per query. Since it's the remote, proprietary LLM that initiates the search I would hesitate to call them agents. Maybe "fetcher" is the best term.
Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
#10They mention anubis, cloudflare, robots.txt – does anyone have experiences with how much any of them help?