Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 296 posts

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#91

Post by hoofhearted »

samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

I don’t really agree with your sentiment.

Robots.txt have served the simple purpose of directing bots like Google to the different parts of your website since the beginning of internet time.

They still serve the same purpose, they tell bots where to go, and most importantly, they tell bots how to find your site map.

Robots.txt is not there to prevent malicious crawlers from accessing pages as you have suggested.

The robots.txt file acts simply like a garden gate. The good and honest people will honor the gate, while the more malicious might ignore it and hop the fence or something.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#92

Post by mort96 »

samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

Do you think there's a space between "you will never ever get to do anything at all with popular media until at least a hundred years after you're dead" and "anyone and any company can do anything they with everything I produce as long as it goes through an LLM"? Is it really so hard to think people may be against both of those extremes?

There's a phrase I like which describes what you're doing. It's "vaguely gesturing at imagined hypocrisy".

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#94

Post by lancesells »

safety1st wrote: post #56
samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

> "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... This gross generalization of other people's views on important issues is really offensive. My view is that the Copyright Act of 1976 had it about right when they established the duration of copyright. My view is that members of Congress were handsomely rewarded by a specific corporation to carve out special…

> Don't recruit me into some bullshit argument that rewrites history and entrenches Disney's ill-gotten monopoly.

You don't think it's them being allowed to buy Marvel, Pixar, Lucasfilm? Is creativity ruined because I can't make a Mickey Mouse cartoon or t-shirt? Does the world need Luke Skywalker coming from any individual studio?

People are free to make the Little Mermaid, Beauty and the Beast, Hunchback of Notre Dame, Aladdin, etc. and there's nothing out there that stops them.

I've got no love for giant corporations but I see it a lot less about copyright than massive corporation gobbling up more corporations. There's no shortage of creativity out there if you look for it.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#95

Post by arwineap »

What if we create a new access.txt which all user agents will use to get access to the resources.

access.txt will return an individual access key for the user agent like a session, and the user agent can only crawl using the access key

This would mean that we could standardize session starts with rate limits. Regular user is unlikely to hit the user rate limits, but bots would get rocked by rate limiting.

Great. Now authorized crawlers, bing, google, etc, all use PKI so that they can sign the request to access.txt to get their access key. If the access.txt request is signed with a known crawler the rate limits can be loosened to levels that a crawler will enjoy

This will allow users / browsers to use normal access patterns without any issue, but crawlers will have to request elevated rate limits to perform their tasks. Crawlers and AI alike could be allowed or disallowed by the service owners, which is really what everyone wanted from robots.txt in the first place

One issue I see with this already is that it solidifies the existing search engines as the market leaders

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#96

Post by doodlesdev »

techaqua wrote: post #82
qbasic_forever wrote: post #54

Your HTML already has semantic meta elements like author and description you should be populating with info like that: https://developer.mozilla.org/en-US/docs/Learn/HTML/Introduc...

and also opengraph meta tags https://ogp.me/

And also schema.org: https://schema.org/

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#97

Post by ihateyouall123 »

safety1st wrote: post #56
samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

> "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... This gross generalization of other people's views on important issues is really offensive. My view is that the Copyright Act of 1976 had it about right when they established the duration of copyright. My view is that members of Congress were handsomely rewarded by a specific corporation to carve out special…

[flagged]

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#98

Post by Brendinooo »

kriro wrote: post #63

I'm curious what the legal ramifications of adding "this code is not to be used for any ML algorithms, failure to adhere to this will result in a fine of at least one million dollars" (in smarter writing) to a software license would be. Seems like a dumb idea/not enforcable but maybe someone with software licensing knowledge can chime in.

I was going to write a "this may sound dumb but..." comment along these lines, thanks for taking the hit.

As users we're forced to browse the Web with a million agreements that say "by using this site you agree to our Terms", what stops you from saying "by crawling this site to train your AI you agree to share profits with us" or whatever, particularly if you can prove that your data ends up being used?

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#100

Post by nashashmi »

What you have described is something akin to what meta tags are for. Do we need another method at a domain or subdomain level? Plus, robots.txt, etc. is limited to domain and subdomain managers.

ai.txt is useful, but I am not sure we have nailed down what it can be used for. One use is to tell AI not to train on the content found within because it could be an AI generation.

Post reply on HN