Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 296 posts

Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#1

Post by Jeannen »

I started to add an ai.txt to my projects. The file is just a basic text file with some useful info about the website like what it is about, when was it published, the author, etc etc.

It can be great if the website somehow ends up in a training dataset (who knows), and it can be super helpful for AI website crawlers, instead of using thousands of tokens to know what your website is about, they can do it with just a few hundred.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#4

Post by jruohonen »

a2800276 wrote: post #3
jruohonen wrote: post #2

A better idea along the same lines: RFC 5785.

Ah, yes, but what about RFC 5226?

I am not sure about that but I think IANA is quite open to recognizing new well-known URIs:

https://www.iana.org/assignments/well-known-uris/well-known-...

Basically, assuming that you have a spec, I think it amounts to filing a PR or discussing it on a mailing list.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#6

Post by samwillis »

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative.

Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare.

In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to align legislation with what the public wants. We need test cases.

The thing I somewhat struggle with is that after 20-30 years of calls for shorter copyright terms, lesser restrictions on content you access publicly, and what you can do with it, we are now in the situation where the arguments are quickly leaning the other way. "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration...

In many ways an ai.txt would be worse than doing nothing as it's a meaningless veneer that would be ignored, but pointed to as the answer.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#8

Post by bryanrasmussen »

aside from the other comments here - robots.txt does work to some extent because it tells the crawler something it might be useful for the crawler to know - if you have blocked it from crawling part of yur site it might be actually beneficial to the crawler to follow that restriction (to be a good citizen) because if it doesn't you might block it by seeing a user agent showing up a part of the site it shouldn't.

AI.txt doesn't have this feedback to the AI to improve it. Also it seems likely users might have reason to lie.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#10

Post by shanebellone »

samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

"Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare."

I like the idea of "ai.txt" but those who eat resources rarely listen to ToS. Frankly, I serve 503s to all identifiable bots, unless they are on my explicit allow list.

Post reply on HN