Isn't an AI a robot?
Tell HN: We should start to add “ai.txt” as we do for “robots.txt”
21–30 of 296 posts
Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”
#22[deleted]
Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”
#23Earlier quoted context omitted.
> Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. Failing to solve every problem does not mean a solution is a failure. From sunscreen to seatbelts, the world is full of great solutions that occasionally fail due to statistics and large numbers.
Ok, fair point, I may be being a little hyperbolic. But my point is that it's not a system that we should copy for preventing the use of content in training AI. It would become a useless distraction. If you "violate" a robots.txt the server administrator can choose to block your bot (if they can fingerprint it) or IP (if its static). With an ai.txt there is no potential downside to violating it - unless we get new le…
> But my point is that it's not a system that we should copy for preventing the use of content in training AI.
I don't think that's what OP is envisioning based on their post!
Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”
#24Earlier quoted context omitted.
"Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare." I like the idea of "ai.txt" but those who eat resources rarely listen to ToS. Frankly, I serve 503s to all identifiable bots, unless they are on my explicit allow list.
Why not serve fake garbage indistinguishable from real content by a computer, like LLM output? Sending errors just incentivizes bot owners to fix the identifiable parts
"Why not serve fake garbage indistinguishable from real content by a computer, like LLM output?"
Serving more than the minimum wastes resources. Worse yet, a better solution would cost my time.
"Sending errors just incentivizes bot owners to fix the identifiable parts"
Sure, someone could make or configure their scraper perfectly. "Perfect" is now the table stakes though.
Edit:
My solution strives to cause an unproportional expense in order to circumvent. I want 10x on my time.
Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”
#25Isn't an AI a robot?
Good point.
Also Killer Robots are Robots: https://www.youtube.com/watch?v=4K6XJuH6P_w
Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”
#26Earlier quoted context omitted.
> Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. Failing to solve every problem does not mean a solution is a failure. From sunscreen to seatbelts, the world is full of great solutions that occasionally fail due to statistics and large numbers.
Ok, fair point, I may be being a little hyperbolic. But my point is that it's not a system that we should copy for preventing the use of content in training AI. It would become a useless distraction. If you "violate" a robots.txt the server administrator can choose to block your bot (if they can fingerprint it) or IP (if its static). With an ai.txt there is no potential downside to violating it - unless we get new le…
OP is trying to give helpful info to the AI, not set boundaries for it.
Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”
#27Earlier quoted context omitted.
"Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare." I like the idea of "ai.txt" but those who eat resources rarely listen to ToS. Frankly, I serve 503s to all identifiable bots, unless they are on my explicit allow list.
Why not serve fake garbage indistinguishable from real content by a computer, like LLM output? Sending errors just incentivizes bot owners to fix the identifiable parts
it'd be cool to be able to fingerprint that garbage, too. Like, sprinkle some hashes here and there (or something like that) so that you can later uniquely look up your own "content" being stolen by chatbots and which ones.
Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”
#28Shouldn't ai respect robots.txt?
They should, but the greed of the AI companies and their lack of respect for other peoples property means there’s pretty much zero chance of that.
Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”
#29Earlier quoted context omitted.
> Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare. Failing to solve every problem does not mean a solution is a failure. From sunscreen to seatbelts, the world is full of great solutions that occasionally fail due to statistics and large numbers.
Ok, fair point, I may be being a little hyperbolic. But my point is that it's not a system that we should copy for preventing the use of content in training AI. It would become a useless distraction. If you "violate" a robots.txt the server administrator can choose to block your bot (if they can fingerprint it) or IP (if its static). With an ai.txt there is no potential downside to violating it - unless we get new le…
> But my point is that it's not a system that we should copy for preventing the use of content in training AI
The purpose OP is suggesting in the submission is the opposite, help AI crawlers to understand what the page/website is about without actually having to infer the purpose from the content itself.
Re: Tell HN: We should start to add “ai.txt” as we do for “robots.txt”
#30I proposed META tags for the same reason. I don't think this is going to happen though.