Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llms.txt

llmstxt.org

Post reply on HN
Print view View original

11–20 of 191 posts

Re: Llms.txt

#11

Post by ljm »

phren0logy wrote: post #3
jsheard wrote: post #2

I'm just left wondering who would volunteer to make their sites easier to scrape. The trend has been the opposite with more and more sites trying to keep LLM scrapers out, whether by politely asking them to go away via robots.txt or proactively blocking their requests entirely.

People who have information they want to share? Programming library docs seem like an obvious choice...

The only reason legit docs are hard to find is because they don’t have Google ads on them and they don’t do SEO.

The solution to the problem isn’t AI. The solution is to break Google’s stranglehold on the web by regulating it.

The solution is to get government up to speed by making it contemporary, so it can understand and respond to current issues. Not leaving it up to people who had their time several decades ago and can’t let go.

Re: Llms.txt

#13

Post by dylan604 »

jsheard wrote: post #2

I'm just left wondering who would volunteer to make their sites easier to scrape. The trend has been the opposite with more and more sites trying to keep LLM scrapers out, whether by politely asking them to go away via robots.txt or proactively blocking their requests entirely.

Sites generated by LLMs

Re: Llms.txt

#15

Post by dylan604 »

TZubiri wrote: post #12

What problem does this solve?

It solves the problem of the cat&mouse game of LLMs updating their scrapers by making site owners provide them the data in a format the LLMs have developed around already.

You're clearly looking at this from the incorrect point of view. Silly human. Think like a bot. --The bot makers

Re: Llms.txt

#17

Post by [deleted] »

ray_v wrote: post #4
phren0logy wrote: post #3

Earlier quoted context omitted.

People who have information they want to share? Programming library docs seem like an obvious choice...

Ostensibly, everyone posting information on the open web want to share information -- either directly with people or indirectly via search engines _and_ the current crop of llms (which in my mind, serve the same purpose as search engines) I suppose the thing that people maybe don't agree with is the lack of attribution when llms regurgitate information back at the user. That, and the fact that these services are also…

[deleted]

Re: Llms.txt

#19

Post by Joker_vD »

JimDabell wrote: post #10

This is not how these kinds of things should be designed for the web. Instead of putting resources in the root of the web, this is what /.well-known/ was designed for. See RFC 5785: https://datatracker.ietf.org/doc/html/rfc5785 Instead of munging URLs to get alternate formats, this is what content negotiation or rel=alternate were designed for. I’m not sure making it easier to consume content is something that is nee…

If only this RFC was well-known among the people who actually put stuff out on the Web.

Re: Llms.txt

#20

Post by seeknotfind »

Joker_vD wrote: post #19
JimDabell wrote: post #10

This is not how these kinds of things should be designed for the web. Instead of putting resources in the root of the web, this is what /.well-known/ was designed for. See RFC 5785: https://datatracker.ietf.org/doc/html/rfc5785 Instead of munging URLs to get alternate formats, this is what content negotiation or rel=alternate were designed for. I’m not sure making it easier to consume content is something that is nee…

If only this RFC was well-known among the people who actually put stuff out on the Web.

How about an LLM agent which automatically finds inconsistencies in RFCs.
Post reply on HN