Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 296 posts

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#11

Post by George83728 »

samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

robots.txt works for the major search engines who voluntarily abide by it, so it isn't a failed system. Just because it doesn't work on everybody doesn't mean it's useless.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#12

Post by Karunamon »

samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

On the contrary, it works perfectly well for normal, non-bad actors running services used by most of the public. That includes search engines and stuff like archive.org. A robots.txt set to deny all will result in your site not showing up on any search engine that matters.

It doesn't work for bad actors, but then again, nothing really does.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#13

Post by alwayslikethis »

shanebellone wrote: post #10
samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

"Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare." I like the idea of "ai.txt" but those who eat resources rarely listen to ToS. Frankly, I serve 503s to all identifiable bots, unless they are on my explicit allow list.

Why not serve fake garbage indistinguishable from real content by a computer, like LLM output? Sending errors just incentivizes bot owners to fix the identifiable parts

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#14

Post by brookst »

samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

> Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare.

Failing to solve every problem does not mean a solution is a failure.

From sunscreen to seatbelts, the world is full of great solutions that occasionally fail due to statistics and large numbers.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#15

Post by bombolo »

samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

> "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration...

AI is being used to do copyright laundering, at the same time "we", the people who can't afford to run our own AI, are still subject to absurd rules that AI owners get to ignore, apparently.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#16

Post by prepend »

samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

> "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration...

While I’m sure others than you share this opinion, I don’t think it’s as uniform as the more common “shorten/rationalize copyright terms and fair use” crowd “we.”

I consider myself a knowledge worker and a pretty staunch proponent of floss and am perfectly fine with training AI on everything publicly available. While create stuff, I don’t make a living off selling particular copies of things I make, so my self preservation bias isn’t kicking in as much as someone who does want to sell items of their work.

But I also made some pretty explicit choices in the 90s based on where I thought IP would go so I was never in a position where I had to sell copies to survive. My decision was more pragmatic first and philosophical second.

I think someone entering the workforce now probably wants to align their livelihood with AI training on everything and not go against that. Even if US/Euro law limits training, there’s no way all other countries are going to, so it’s going to happen. And I don’t think it’s worth locking down the world to try to stop AIs from training on text, images, etc.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#18

Post by samwillis »

brookst wrote: post #14
samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

> Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. Failing to solve every problem does not mean a solution is a failure. From sunscreen to seatbelts, the world is full of great solutions that occasionally fail due to statistics and large numbers.

Ok, fair point, I may be being a little hyperbolic. But my point is that it's not a system that we should copy for preventing the use of content in training AI. It would become a useless distraction.

If you "violate" a robots.txt the server administrator can choose to block your bot (if they can fingerprint it) or IP (if its static).

With an ai.txt there is no potential downside to violating it - unless we get new legislation enforcing its legal standing. The nature of ML models is that it's opaque what content exactly it's trained on, there is no obvious retaliation or retribution.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#19

Post by nstj »

“Google Search works hard to understand the content of a page. You can help us by providing explicit clues about the meaning of a page to Google by including structured data on the page.”[0]

[0]: https://developers.google.com/search/docs/appearance/structu...

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#20

Post by JohnFen »

prepend wrote: post #16
samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

> "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... While I’m sure others than you share this opinion, I don’t think it’s as uniform as the more common “shorten/rationalize copyright terms and fair use” crowd “we.” I consider myself a knowledge worker and a pretty staunch proponent of floss and am perfectly fine with training AI on everything publicly availab…

Fair enough. But there should be some mechanism where people who don't want their works to contribute to AI training to be able to prevent that without having to resort to removing their works from the web.
Post reply on HN