Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#1

Post by Jeannen » Wed, May 10, 2023, 12:20 PM UTC

I started to add an ai.txt to my projects. The file is just a basic text file with some useful info about the website like what it is about, when was it published, the author, etc etc.

It can be great if the website somehow ends up in a training dataset (who knows), and it can be super helpful for AI website crawlers, instead of using thousands of tokens to know what your website is about, they can do it with just a few hundred.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#2

Post by jruohonen » Wed, May 10, 2023, 12:24 PM UTC

A better idea along the same lines: RFC 5785.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#3

Post by a2800276 » Wed, May 10, 2023, 12:31 PM UTC

jruohonen wrote: post #2

A better idea along the same lines: RFC 5785.

Ah, yes, but what about RFC 5226?

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#4

Post by jruohonen » Wed, May 10, 2023, 12:37 PM UTC

a2800276 wrote: post #3

jruohonen wrote: post #2

A better idea along the same lines: RFC 5785.

Ah, yes, but what about RFC 5226?

I am not sure about that but I think IANA is quite open to recognizing new well-known URIs:

https://www.iana.org/assignments/well-known-uris/well-known-...

Basically, assuming that you have a spec, I think it amounts to filing a PR or discussing it on a mailing list.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#5

Post by deafpolygon » Wed, May 10, 2023, 12:55 PM UTC

Shouldn't ai respect robots.txt?

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#6

Post by samwillis » Wed, May 10, 2023, 12:56 PM UTC

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative.

Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare.

In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to align legislation with what the public wants. We need test cases.

The thing I somewhat struggle with is that after 20-30 years of calls for shorter copyright terms, lesser restrictions on content you access publicly, and what you can do with it, we are now in the situation where the arguments are quickly leaning the other way. "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration...

In many ways an ai.txt would be worse than doing nothing as it's a meaningless veneer that would be ignored, but pointed to as the answer.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#7

Post by theandrewbailey » Wed, May 10, 2023, 12:58 PM UTC

Can you give a live example? What is in this ai.txt that isn't in an about page that almost every site has?

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#8

Post by bryanrasmussen » Wed, May 10, 2023, 1:00 PM UTC

aside from the other comments here - robots.txt does work to some extent because it tells the crawler something it might be useful for the crawler to know - if you have blocked it from crawling part of yur site it might be actually beneficial to the crawler to follow that restriction (to be a good citizen) because if it doesn't you might block it by seeing a user agent showing up a part of the site it shouldn't.

AI.txt doesn't have this feedback to the AI to improve it. Also it seems likely users might have reason to lie.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#9

Post by [deleted] » Wed, May 10, 2023, 1:00 PM UTC

[deleted]

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#10

Post by shanebellone » Wed, May 10, 2023, 1:01 PM UTC

samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

"Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare."

I like the idea of "ai.txt" but those who eat resources rarely listen to ToS. Frankly, I serve 503s to all identifiable bots, unless they are on my explicit allow list.