Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llms.txt

llmstxt.org

Post reply on HN
Print view View original

41–50 of 191 posts

Re: Llms.txt

#41

Post by bawolff »

Joker_vD wrote: post #19
JimDabell wrote: post #10

This is not how these kinds of things should be designed for the web. Instead of putting resources in the root of the web, this is what /.well-known/ was designed for. See RFC 5785: https://datatracker.ietf.org/doc/html/rfc5785 Instead of munging URLs to get alternate formats, this is what content negotiation or rel=alternate were designed for. I’m not sure making it easier to consume content is something that is nee…

If only this RFC was well-known among the people who actually put stuff out on the Web.

I think it is. Lots of actually used standards use it. I see it in the wild all the time.

Certainly more well used than content negotiation which grandparent also mentioned.

Re: Llms.txt

#42

Post by autoexec »

bawolff wrote: post #39

I'm not that familiar with llms, but surely we are already at the point where web pages can be easily scrapped? Is markdown really an easier format to understand than html? If this is actually useful wouldn't .txt be supperior to markdown for this usecase? Does this solve a problem llms actually have? Not trying to be negative, i'm honestly curious.

Yeah, I'm not sure what the point of markdown is here either. I would expect that anything that looks remotely like a URL will be collected and scraped no matter what format it's in.

Re: Llms.txt

#43

Post by ftmch »

russellbeattie wrote: post #22
Joker_vD wrote: post #19

Earlier quoted context omitted.

If only this RFC was well-known among the people who actually put stuff out on the Web.

If only that RFC didn't make it a hidden directory. I can think of a dozen reasons why hiding that folder is a horrible idea, and not a single one for why it would be a good thing to do.

Also "well-known" was always such an awkward name to me.

Re: Llms.txt

#44

Post by KTibow »

JimDabell wrote: post #10

This is not how these kinds of things should be designed for the web. Instead of putting resources in the root of the web, this is what /.well-known/ was designed for. See RFC 5785: https://datatracker.ietf.org/doc/html/rfc5785 Instead of munging URLs to get alternate formats, this is what content negotiation or rel=alternate were designed for. I’m not sure making it easier to consume content is something that is nee…

It's based on the series of robots.txt, humans.txt, security.txt, etc

Re: Llms.txt

#45

Post by idf00 »

bawolff wrote: post #39

I'm not that familiar with llms, but surely we are already at the point where web pages can be easily scrapped? Is markdown really an easier format to understand than html? If this is actually useful wouldn't .txt be supperior to markdown for this usecase? Does this solve a problem llms actually have? Not trying to be negative, i'm honestly curious.

Yes. Converting docs to markdown and using them in claude projects, for example, makes a big difference.

Re: Llms.txt

#46

Post by haswell »

ray_v wrote: post #4
phren0logy wrote: post #3

Earlier quoted context omitted.

People who have information they want to share? Programming library docs seem like an obvious choice...

Ostensibly, everyone posting information on the open web want to share information -- either directly with people or indirectly via search engines _and_ the current crop of llms (which in my mind, serve the same purpose as search engines) I suppose the thing that people maybe don't agree with is the lack of attribution when llms regurgitate information back at the user. That, and the fact that these services are also…

That’s really my primary issue. Google indexing my content and directing traffic to my site is one thing.

But unlike search indexing, there is no exchange of value when these LLMs are trained on my content. We all collectively get nothing for our work. It’s theft dressed up as business as usual. I’ll do whatever I reasonably can to avoid feeding the machine and hope some of the ongoing and inevitable legal fights will rein things in a bit.

Re: Llms.txt

#47

Post by autoexec »

azhenley wrote: post #5

LLMs.txt should let me specify the $$$ price that companies must send me to train models on my content.

No, see you're supposed to create and upload this specially formatted file on all your webservers for free, just to make it a little easier for them to take all your content for free, so that they can then use your content in their products for free, so they can charge other humans money to get your content from their product without any humans ever having to visit your actual website again. What's not to like?

If they had to pay for all the content they take/use/redistribute they wouldn't be able to make enough money off of your work for it to be worthwhile.

Re: Llms.txt

#48

Post by autoexec »

ray_v wrote: post #4
phren0logy wrote: post #3

Earlier quoted context omitted.

People who have information they want to share? Programming library docs seem like an obvious choice...

Ostensibly, everyone posting information on the open web want to share information -- either directly with people or indirectly via search engines _and_ the current crop of llms (which in my mind, serve the same purpose as search engines) I suppose the thing that people maybe don't agree with is the lack of attribution when llms regurgitate information back at the user. That, and the fact that these services are also…

I like sharing information. Information wants to be free after all. Companies on the other hand want to charge people money to use their LLMs and associated AI products, so suddenly we've got a bunch of people profiting off of our content, potentially butchering it or hallucinating all over it in the process.

Re: Llms.txt

#49

Post by itsbjoern »

internetter wrote: post #6

To disallow: Amazonbot, anthropic-ai, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-Web, cohere-ai, DataForSeoBot, Diffbot, Webzio-Extended, FacebookBot, FriendlyCrawler, Google-Extended, GPTBot, 0AI-SearchBot, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, omgili, omgilibot, PerplexityBot, Quora-Bot, TurnitinBot For all of these bots, User-agent: Disallow: / For more infor…

Always find it amusing when people write about „blocking“ requests using robots.txt as if they are deploying a firewall

Re: Llms.txt

#50

Post by JimDabell »

KTibow wrote: post #44
JimDabell wrote: post #10

This is not how these kinds of things should be designed for the web. Instead of putting resources in the root of the web, this is what /.well-known/ was designed for. See RFC 5785: https://datatracker.ietf.org/doc/html/rfc5785 Instead of munging URLs to get alternate formats, this is what content negotiation or rel=alternate were designed for. I’m not sure making it easier to consume content is something that is nee…

It's based on the series of robots.txt, humans.txt, security.txt, etc

Yes, and robots.txt was reasonable because it was created in the early days of the web. But the others don’t have any excuse (and they were warned about it as soon as they were announced and ignored people).
Post reply on HN