Llms.txt
llmstxt.org
Llms.txt
1–10 of 191 posts
Re: Llms.txt
#2I'm just left wondering who would volunteer to make their sites easier to scrape. The trend has been the opposite with more and more sites trying to keep LLM scrapers out, whether by politely asking them to go away via robots.txt or proactively blocking their requests entirely.
Re: Llms.txt
#3I'm just left wondering who would volunteer to make their sites easier to scrape. The trend has been the opposite with more and more sites trying to keep LLM scrapers out, whether by politely asking them to go away via robots.txt or proactively blocking their requests entirely.
People who have information they want to share? Programming library docs seem like an obvious choice...
Re: Llms.txt
#4I'm just left wondering who would volunteer to make their sites easier to scrape. The trend has been the opposite with more and more sites trying to keep LLM scrapers out, whether by politely asking them to go away via robots.txt or proactively blocking their requests entirely.
People who have information they want to share? Programming library docs seem like an obvious choice...
Ostensibly, everyone posting information on the open web want to share information -- either directly with people or indirectly via search engines _and_ the current crop of llms (which in my mind, serve the same purpose as search engines)
I suppose the thing that people maybe don't agree with is the lack of attribution when llms regurgitate information back at the user. That, and the fact that these services are also overly aggressive when it comes to spidering your site
Re: Llms.txt
#5LLMs.txt should let me specify the $$$ price that companies must send me to train models on my content.
Re: Llms.txt
#6To disallow:
Amazonbot, anthropic-ai, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-Web, cohere-ai, DataForSeoBot, Diffbot, Webzio-Extended, FacebookBot, FriendlyCrawler, Google-Extended, GPTBot, 0AI-SearchBot, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, omgili, omgilibot, PerplexityBot, Quora-Bot, TurnitinBot
For all of these bots,
User-agent: Disallow: /
For more information, check https://darkvisitors.com/agents
If this takes off, I've made my own variant of llms.txt here: https://boehs.org/llms.txt . I hereby release this file to the public domain, if you wish to adapt and reuse it on your own site.
Hall of shame: https://www.404media.co/websites-are-blocking-the-wrong-ai-s...
Re: Llms.txt
#7[flagged]
Re: Llms.txt
#8This should just be some kind of subset of robots.txt
Re: Llms.txt
#9I'm just left wondering who would volunteer to make their sites easier to scrape. The trend has been the opposite with more and more sites trying to keep LLM scrapers out, whether by politely asking them to go away via robots.txt or proactively blocking their requests entirely.
I have a bunch of open source projects that I want LLMs to know all of the details of so they can help people use them.
Re: Llms.txt
#10This is not how these kinds of things should be designed for the web.
Instead of putting resources in the root of the web, this is what /.well-known/ was designed for. See RFC 5785:
https://datatracker.ietf.org/doc/html/rfc5785
Instead of munging URLs to get alternate formats, this is what content negotiation or rel=alternate were designed for.
I’m not sure making it easier to consume content is something that is needed. I think it might be more useful to define script type=llm that would expose function calling to LLMs embedded in browsers.