Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llms.txt

llmstxt.org

Post reply on HN
Print view View original

31–40 of 191 posts

Re: Llms.txt

#31

Post by CuriousCosmic »

russellbeattie wrote: post #22
Joker_vD wrote: post #19

Earlier quoted context omitted.

If only this RFC was well-known among the people who actually put stuff out on the Web.

If only that RFC didn't make it a hidden directory. I can think of a dozen reasons why hiding that folder is a horrible idea, and not a single one for why it would be a good thing to do.

The reason is because it's supposed to be a standard folder that isn't in use accidentally for other purposes.

It's exceedingly unlikely that a website is going to just happen to make content available in a hidden directory path without it being created by automated tooling (which would likely be aware of such standards).

The entire point is to avoid adopting a path that people already publicly use for something else. A hidden directory is the best way to do that.

Re: Llms.txt

#32

Post by tbrownaw »

Is this trying to be what the semantic web was supposed to be? Or is it trying to be "OpenAPI for things that aren't REST/JSON-RPC APIs"? (Are those even any different?)

And we already have plenty of standards for library documentation. Man pages, info pages, Perldoc, Javadoc, ...

Re: Llms.txt

#33

Post by CuriousCosmic »

JimDabell wrote: post #10

This is not how these kinds of things should be designed for the web. Instead of putting resources in the root of the web, this is what /.well-known/ was designed for. See RFC 5785: https://datatracker.ietf.org/doc/html/rfc5785 Instead of munging URLs to get alternate formats, this is what content negotiation or rel=alternate were designed for. I’m not sure making it easier to consume content is something that is nee…

Fwiw the active RFC is 8615 as RFC5785 is obsolete.

https://datatracker.ietf.org/doc/html/rfc8615

Re: Llms.txt

#34

Post by tbrownaw »

TZubiri wrote: post #12

What problem does this solve?

I think the idea is that LLMs aren't actually that good, so adding a semi-machine-readable version of your site can make it easier for them to surface your work to their own users.

Re: Llms.txt

#35

Post by bnchrch »

nutanc wrote: post #29

Actually what is also needed is a notLLMs.txt. robots.txt exists, but is mainly for crawling and also not sure anyone follows it or even if they don't follow what's the punishment.

Exactly. robots.txt is useless and those that think its useful for preventing unwanted crawling are clueless

Re: Llms.txt

#36

Post by bdcravens »

bnchrch wrote: post #35
nutanc wrote: post #29

Actually what is also needed is a notLLMs.txt. robots.txt exists, but is mainly for crawling and also not sure anyone follows it or even if they don't follow what's the punishment.

Exactly. robots.txt is useless and those that think its useful for preventing unwanted crawling are clueless

It's a "Keep off the grass" sign. The polite will obey, the ones who are truly the problem will not.

Re: Llms.txt

#37

Post by crowcroft »

If an LLM needs something like this for context after crawling your site then you might have bigger problems with your site.

Re: Llms.txt

#38

Post by autoexec »

internetter wrote: post #6

To disallow: Amazonbot, anthropic-ai, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-Web, cohere-ai, DataForSeoBot, Diffbot, Webzio-Extended, FacebookBot, FriendlyCrawler, Google-Extended, GPTBot, 0AI-SearchBot, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, omgili, omgilibot, PerplexityBot, Quora-Bot, TurnitinBot For all of these bots, User-agent: Disallow: / For more infor…

As much as these companies should respect our preferences, it's very clear that they won't. It wouldn't matter to these companies if it was outright illegal, "pretty please" certainly isn't going to cut it. You can't stop scraping and the harder people try the worse their sites become for everyone else. Throwing up a robots.txt or llms.txt that calls out their bad behavior isn't a bad idea, but it's not likely to help anything either.

Re: Llms.txt

#39

Post by bawolff »

I'm not that familiar with llms, but surely we are already at the point where web pages can be easily scrapped? Is markdown really an easier format to understand than html? If this is actually useful wouldn't .txt be supperior to markdown for this usecase?

Does this solve a problem llms actually have?

Not trying to be negative, i'm honestly curious.

Post reply on HN