Go ahead and block AI web crawlers

curious, will the robots.txt be really honored? maybe legal issue if not?

I don't see how a robots.txt itself would be legally binding, but I guess you could make an argument for the terms of service? Would love to know more about this as well.

The AI models would not exist without data the so called luddites created.

robots.txt being an honor system has always been a bit weird and search engine treatment is odder still.

Google will still index your disallowed pages but refuse to show a description since they are couldn't read the page. The premise being that Google could have discovered your URLs from other pages and they don't consider a robots.txt disallow to indicate that you don't want them in search, just that you don't want Google to directly read the page.

To actually disallow indexing of a page a page you need to allow it in robots and then add a noindex to the page itself.

No one has really explained how these AI agents will treat robots.txt. Does it just mean that Bard/Gemini won't directly read the page but might somehow incorporate it into a dataset?

For a test I asked Gemini "What is the URL of the String api documentation for Java 6?" which is disallowed by robots.txt.

The response was: Unfortunately, Oracle does not maintain separate API documentation for older versions of Java like Java 6. The official documentation website (https://docs.oracle.com/javase/7/docs/api/) only provides documentation for the latest versions.

Oracle has archieved some of the older Java API documentation but 6 is still available at https://docs.oracle.com/javase/6/docs/api/index.html. Funny enough the Java 7 API docs which Gemini linked too is also disallowed but maybe that happened after the model was trained.