AI-Shunning robots.txt

#1

Post by glynnormington » Thu, Mar 28, 2024, 11:46 AM UTC

Re: AI-Shunning robots.txt

#2

Post by vouaobrasil » Thu, Mar 28, 2024, 12:32 PM UTC

Nice. Let's all contribute to this...ideally, web-hosts should provide this sort of thing by default so we can starve AI companies from training data and combine it with other strategies to put them out of business for good.

Re: AI-Shunning robots.txt

#3

Post by belter » Thu, Mar 28, 2024, 12:32 PM UTC

Or redirect them to poisoned material?

Re: AI-Shunning robots.txt

#4

Post by nerdjon » Thu, Mar 28, 2024, 12:32 PM UTC

I am curious, do we have any evidence that AI is adhering to robots.txt and isn’t ignoring it since they are not technically crawling in the traditional sense?

Even if they are right now it would be a quick switch for them to just ignore it.

Re: AI-Shunning robots.txt

#5

Post by andybak » Thu, Mar 28, 2024, 12:33 PM UTC

As someone who uses and benefits from the results of AI crawlers, I would only want to block crawls under very specific circumstances.

I would back a general move to block crawlers from non-open models (whatever that means and if such a thing was practical) as it might be a strong lever to encourage good behaviour.

Re: AI-Shunning robots.txt

#6

Post by jddj » Thu, Mar 28, 2024, 12:33 PM UTC

The named source, https://darkvisitors.com, is interesting.

Re: AI-Shunning robots.txt

#7

Post by andybak » Thu, Mar 28, 2024, 12:34 PM UTC

vouaobrasil wrote: post #2

Nice. Let's all contribute to this...ideally, web-hosts should provide this sort of thing by default so we can starve AI companies from training data and combine it with other strategies to put them out of business for good.

How about AI from non-companies? Or genuinely non-profit or open projects?

Also - out of curiosity - do you use any AI yourself?

Re: AI-Shunning robots.txt

#8

Post by andybak » Thu, Mar 28, 2024, 12:35 PM UTC

nerdjon wrote: post #4

I am curious, do we have any evidence that AI is adhering to robots.txt and isn’t ignoring it since they are not technically crawling in the traditional sense? Even if they are right now it would be a quick switch for them to just ignore it.

This is about crawling for training data by the look of things. Not sure if the CHatGPT browsing mode uses a different user-agent but most of the entries in that list look like crawlers.

Re: AI-Shunning robots.txt

#9

Post by cabirum » Thu, Mar 28, 2024, 12:39 PM UTC

The crawlers can simply stop identifying themselves via custom user agent, can't they?

Also why are "AI" crawlers are worse than "normal" crawlers?

Either way, this is an exercise in futility.

Re: AI-Shunning robots.txt

#10

Post by bakugo » Thu, Mar 28, 2024, 12:41 PM UTC

This makes complete sense because, as we all know, AI companies are very concerned with respecting the rights of the people they steal data from, and totally won't just ignore this.