Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 296 posts

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#101

Post by michaelmior »

usrusr wrote: post #48
brookst wrote: post #14

Earlier quoted context omitted.

> Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. Failing to solve every problem does not mean a solution is a failure. From sunscreen to seatbelts, the world is full of great solutions that occasionally fail due to statistics and large numbers.

That's still not an argument to introduce ai.txt, because everything a hypothetical ai.txt could ever do is already done just as good (or not) by the robots.txt we have. If a training data crawler ignores robots.txt it won't bother checking for an ai.txt either. And if you feel like rolling out the "welcome friend!" doormat to a particular training data crawler, you are free to dedicate as detailed a robots.txt block…

This seems to be assuming a very different purpose for ai.txt than the OP proposed. It sounds like they are intending ai.txt to give useful contextual information to crawlers collecting AI training data. Robots.txt does not have any of this information (although I suppose you could include it in comments).

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#102

Post by xhkkffbf »

safety1st wrote: post #56
samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

> "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... This gross generalization of other people's views on important issues is really offensive. My view is that the Copyright Act of 1976 had it about right when they established the duration of copyright. My view is that members of Congress were handsomely rewarded by a specific corporation to carve out special…

Why does Disney have an "ill-gotten" monopoly? The people who worked for the company created something. Why shouldn't they get to control how it's used. Do you feel like you should have control over what you create? Why not others?

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#103

Post by r3trohack3r »

arwineap wrote: post #95

What if we create a new access.txt which all user agents will use to get access to the resources. access.txt will return an individual access key for the user agent like a session, and the user agent can only crawl using the access key This would mean that we could standardize session starts with rate limits. Regular user is unlikely to hit the user rate limits, but bots would get rocked by rate limiting. Great. Now…

I might not understand you, but what prevents me from conducting a Sybil attack (a.k.a. a sock puppet attack) against this system?

Seems like it relies on everyone playing by the rules and only requesting one license per user. Why would a bot developer be incentivized to follow that rule and not just request 1M licenses?

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#104

Post by mtmail »

javierluraschi wrote: post #85

Related, there is also https://datatxt.org

Which claims 'under active development'. Four years ago the author took the robots.txt RFC and changed a couple of paragraphs https://github.com/datatxtorg/datatxt-spec/commit/36028e2280... Meanwhile the robots.txt was updated in 2022 https://www.rfc-editor.org/rfc/rfc9309.html

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#105

Post by Karellen »

rzzzt wrote: post #73
dingle_thunk wrote: post #21

Isn't an AI a robot?

It's a composition vs. inheritance problem. I postulate that robots need at least a single manipulator in the physical realm: Mechanical arm assembling car doors = robot. CNC machine that follows a path = robot. Mechs with chicken legs = robot. Brain in a vat = not a robot... but can be embedded in a robot.

It's a nice idea, but it totally ignores literally decades of existing use of the word "robot" (or its abbreviation "bot") to describe pure software that accesses internet services. e.g. web crawlers (googlebot), chat bots, automated clickers, etc...

Lexicography tends to be descriptive rather than prescriptive. If enough people use a word to mean a thing, that word means that thing. As least in some contexts. See also "gay", "hacker", etc...

Note that it is possible for a word's meaning to be "reclaimed", but it generally doesn't get that way by some small group of people just shouting "You're doing it wrong!"

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#106

Post by mtmail »

westurner wrote: post #75

security.txt https://github.com/securitytxt/security-txt : > security.txt provides a way for websites to define security policies. The security.txt file sets clear guidelines for security researchers on how to report security issues. security.txt is the equivalent of robots.txt, but for security issues. Carbon.txt: https://github.com/thegreenwebfoundation/carbon.txt : > A proposed convention for website owners and di…

Having a security.txt doesn't stop security researchers asking "Do you have a bounty program?". We replied dozens already that such a file exist, it's not well enough known yet. On the other hand there are search engines crawling those and creating reports, which is nice.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#107

Post by dclowd9901 »

safety1st wrote: post #56
samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

> "We" now want stricter copyright law when it comes to AI, but at the same time shorter copyright duration... This gross generalization of other people's views on important issues is really offensive. My view is that the Copyright Act of 1976 had it about right when they established the duration of copyright. My view is that members of Congress were handsomely rewarded by a specific corporation to carve out special…

> Reducing copyright term extension has everything to do with restoring competition and creativity to our economy

Can you explain your line of thinking here? How does the ability to use another company’s intellectual property restore creativity? It just seems like a path to allow bootlegging.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#108

Post by shadowgovt »

samwillis wrote: post #6

Using robots.txt as a model for anything doesn't work. All a robots.txt is is a polite request to please follow the rules in it, there is no "legal" agreement to follow those rules, only a moral imperative. Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. In the age of AI we need to better understand where copyright applies to it, and potentially need reform of copyright to ali…

I disagree it has failed as a system. While it does not substitute for authentication / authorization, reputable crawlers respect it, and there'd be a lot more traffic load on sites if they didn't have a way to tell reputable crawlers "please stop."

Similarly, extending robots.txt to direct AI would have a similar effect: not sufficient, but useful (if for no other reason than to make it easy to distinguish reputable AI projects from ones that feel like they own the Internet to do with as they please).

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#109

Post by jedberg »

> it can be super helpful for AI website crawlers, instead of using thousands of tokens to know what your website is about, they can do it with just a few hundred.

Why would the crawler trust you to be accurate instead of just figuring it out for itself?

Besides, they want to hoover up all the data for their training set anyway.

Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”

#110

Post by [deleted] »

mrighele wrote: post #61

I would prefer a more generic "license.txt" i.e. a standard sanctioned way of telling the User Agent that the resource under a certain location is provided with a specific license. Maybe a picture is public domain, maybe is copyrighted but freely distributable, or maybe it is but you cannot train AI on it. Same for code, text, articles etc. The difficult part would be to make it formal enough so that it can easily co…

[deleted]
Post reply on HN