AI companies cause most of traffic on forums
pod.geraspora.de
AI companies cause most of traffic on forums
1–10 of 466 posts
Re: AI companies cause most of traffic on forums
#2If they ignore robots.txt there should be some kind of recourse :(
Re: AI companies cause most of traffic on forums
#3If they ignore robots.txt there should be some kind of recourse :(
[flagged]
Re: AI companies cause most of traffic on forums
#4If they ignore robots.txt there should be some kind of recourse :(
Sadly, as the slide from high-trust society to low-trust society continues, doing "the right thing" becomes less and less likely.
Re: AI companies cause most of traffic on forums
#5Ironic that there is a dichotomy between Google and Bing with orders of magnitude less traffic than AI organizations, because only Google really has fresh docs. Bing isn't terrible but their index is usually days old. But something like Claude is years out of date. Why do they need to crawl that much?
Re: AI companies cause most of traffic on forums
#6If they ignore robots.txt there should be some kind of recourse :(
[flagged]
It's certainly one of the few things that actually gets their attention. But aren't there more important things than this for the Luigis among us?
I would suspect there's good money in offering a service to detect AI content on all of these forums and reject it. That will then be used as training data to refine them which gives such a service infinite sustainability.
Re: AI companies cause most of traffic on forums
#7If they ignore robots.txt there should be some kind of recourse :(
zip b*mbs?
Re: AI companies cause most of traffic on forums
#8Ironic that there is a dichotomy between Google and Bing with orders of magnitude less traffic than AI organizations, because only Google really has fresh docs. Bing isn't terrible but their index is usually days old. But something like Claude is years out of date. Why do they need to crawl that much?
They don’t. They are wasting their resources and other people’s resources because at the moment they have essentially unlimited cash to burn burn burn.
Re: AI companies cause most of traffic on forums
#9Are these IPs actually from OpenAI/etc. (https://openai.com/gptbot.json), or is it possibly something else masquerading as these bots? The real GPTBot/Amazonbot/etc. claim to obey robots.txt, and switching to a non-bot UA string seems extra questionable behaviour.
Re: AI companies cause most of traffic on forums
#10OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scraper...
For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/
> The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the content on blog post and pages on your site when one of these bots slithers by.