Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 296 posts

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#31

Post by rhn_mk1 »

bombolo wrote: post #15
samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

> "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... AI is being used to do copyright laundering, at the same time "we", the people who can't afford to run our own AI, are still subject to absurd rules that AI owners get to ignore, apparently.

The barrier to running an AI model is getting lower every day, so the threshold for ignoring copyright is getting lower with it.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#32

Post by kordlessagain »

robots.txt is for all crawlers, so there's no need for another file? robots.txt supports comments using # and ideally has a link to the site map, which would tell any robot crawler where the important bits live on the site.

Putting a good comment at the top of robots.txt would be just as good as any other solution, given it could serve as a type of prompt template for processing the data on the site it represents.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#33

Post by mschuster91 »

samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

> Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare.

That depends what you expect from it. For the purpose of limiting crawlers, at least the major search engines respect it.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#34

Post by shanebellone »

twelve40 wrote: post #27
alwayslikethis wrote: post #13

Earlier quoted context omitted.

Why not serve fake garbage indistinguishable from real content by a computer, like LLM output? Sending errors just incentivizes bot owners to fix the identifiable parts

it'd be cool to be able to fingerprint that garbage, too. Like, sprinkle some hashes here and there (or something like that) so that you can later uniquely look up your own "content" being stolen by chatbots and which ones.

You can. I can't think of the appropriate term though. Hopefully someone else chimes in here with a link.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#36

Post by jefftk »

samwillis wrote: post #18
brookst wrote: post #14

Earlier quoted context omitted.

> Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. Failing to solve every problem does not mean a solution is a failure. From sunscreen to seatbelts, the world is full of great solutions that occasionally fail due to statistics and large numbers.

Ok, fair point, I may be being a little hyperbolic. But my point is that it's not a system that we should copy for preventing the use of content in training AI. It would become a useless distraction. If you "violate" a robots.txt the server administrator can choose to block your bot (if they can fingerprint it) or IP (if its static). With an ai.txt there is no potential downside to violating it - unless we get new le…

> It's not a system that we should copy for preventing the use of content in training AI

I don't see the OP saying anything about "ai.txt" being for that? They're advocating it as a way that AIs could use fewer tokens to understand what a site is about.

(Which I also don't think is a good idea, since we already have lots of ways of including structured metadata in pages, but the main problem is not that crawlers would ignore it.)

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#37

Post by majewsky »

samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

> All a robots.txt is is a polite request to please follow the rules in it

At least in my country (Germany), respecting robots.txt is a legal requirement for data mining. See German Copyright Code, section 44b: https://www.gesetze-im-internet.de/urhg/__44b.html

(IANAL)

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#38

Post by devd00d »

matsemann wrote: post #35

Reading the title I thought you meant the opposite. Aka, an ai.txt file that disallow ai to train or use your data similar to robots.txt (but for cases when you still want to be crawled, just not extrapolated)

I thought the exact same. Creating a new type of robots.txt but making it do the opposite does not make sense.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#40

Post by ape4 »

alwayslikethis wrote: post #13
shanebellone wrote: post #10

Earlier quoted context omitted.

"Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare." I like the idea of "ai.txt" but those who eat resources rarely listen to ToS. Frankly, I serve 503s to all identifiable bots, unless they are on my explicit allow list.

Why not serve fake garbage indistinguishable from real content by a computer, like LLM output? Sending errors just incentivizes bot owners to fix the identifiable parts

I like this idea. Of course it would have to be only to robots that visit a page disallowed by the robots.txt
Post reply on HN