Does any of the LLM providers actually use llms.txt? If I remember correctly this "standard" was setup by someone but without involvement of any of the major AI players.
I can definitively say llms.txt is not used by any AI players. I run a blogging platform with around 80k blogs and /llms.txt is not requested by anything (other than humans checking to see if there's an llms.txt path). All regular pages are aggressively scraped to the extent it's a problem I have to consistently manage, but not llms.txt.
> I can definitively say llms.txt is not used by any AI players.
https://developers.openai.com/llms.txt
https://docs.anthropic.com/llms.txt
https://geminicli.com/llms.txt
https://github.com/llms.txt
https://docs.aws.amazon.com/llms.txt
https://openrouter.ai/docs/llms.txt