Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llms.txt

llmstxt.org

Post reply on HN
Print view View original

71–80 of 191 posts

Re: Llms.txt

#71

Post by kilian »

nutanc wrote: post #29

Actually what is also needed is a notLLMs.txt. robots.txt exists, but is mainly for crawling and also not sure anyone follows it or even if they don't follow what's the punishment.

There is a proposal for that too: https://site.spawning.ai/spawning-ai-txt but it's wholly unclear if AI companies actually do something with this or if it's just wishful thinking...

Some AI companies follow robots.txt (OpenAI and Google, for example) but others ignore it. There's also other limitations around using robots.txt to sole this problem: https://searchengineland.com/robots-txt-new-meta-tag-llm-ai-...

Re: Llms.txt

#72

Post by greatNespresso »

Had the exact same thought some time ago now, even proposed it internally at my company. What makes me doubt this will work eventually is that scraping has been going on forever now and yet no standard has been accepted (as you noted robots.txt serves a different purpose, should have been called indexation.txt)

Re: Llms.txt

#73

Post by jph00 »

autoexec wrote: post #42
bawolff wrote: post #39

I'm not that familiar with llms, but surely we are already at the point where web pages can be easily scrapped? Is markdown really an easier format to understand than html? If this is actually useful wouldn't .txt be supperior to markdown for this usecase? Does this solve a problem llms actually have? Not trying to be negative, i'm honestly curious.

Yeah, I'm not sure what the point of markdown is here either. I would expect that anything that looks remotely like a URL will be collected and scraped no matter what format it's in.

Context windows for LLM inference are limited. You can't just throw everything into it -- it won't all fit, and larger amounts of context are slower and more expensive. So it's important to have a carefully curated set of well-formatted documents to work with.

Re: Llms.txt

#74

Post by jph00 »

TZubiri wrote: post #12

What problem does this solve?

From the post describing llms.txt (https://www.answer.ai/posts/2024-09-03-llmstxt.html):

"The problem this solves is that today, constructing the right context for LLMs based on a website is ambiguous — do you:

1. Crawl the sitemap and include every page, trying to automatically format into an LLM-friendly form?

2. Selectively include external links in addition to the sitemap?

3. For specific domains like software documentation should you also try to include all the source code?

Site authors know best, and can provide a list of content that an LLM should use."

(There's quite a bit more info there that answers this question in more detail.)

Re: Llms.txt

#75

Post by nilsherzig »

otherme123 wrote: post #63
jraph wrote: post #61

Earlier quoted context omitted.

I've seen some of these bots take a lot of CPU on my server, especially when browsing my (very small) forgejo instance. I banned them with a 444 error [1] in the reverse proxy settings as a temporary measure that became permanent, and then some more from this list [2], but I will consider yours as well, thanks for sharing. if ($http_user_agent ~ facebook) { return 444; } if ($http_user_agent ~ Amazonbot) { return 444…

In your nginx.conf, http block, add include /etc/nginx/useragent.rules; In /etc/nginx/useragent.rules map $http_user_agent $badagent { default 0; ~facebook 1; [...] ~PerplexityBot 1; } In your site.conf, server block, add if ($badagent) { return 444; }

Anyone knows of a crowd sourced list of these user agents? With the current state of AI startups it will be hard to keep this up to date by myself

Re: Llms.txt

#77

Post by genewitch »

I "scrape" some sites[0], generally one time, using a single thread, and my crap home internet. On a good day i'll set ~2mbit/sec throttle on my side. I do this for archival purposes. So is this generally cool with everyone, or am i supposed to be reading humans.txt or whatever? I hope the spirit of my question makes sense.

[0] my main catchall textual site rip directory is 17GB; but i have some really large sites i heard in advance were probably shuttering, that size or larger.

Re: Llms.txt

#78

Post by romantomjak »

I find it confusing that author proposes llms.txt, but the content is actually markdown? I get that they tried to follow the convention, but then why not make it a simple text file like the robots.txt is?

Re: Llms.txt

#79

Post by mrweasel »

Wouldn't this open up for manipulating LLMs?

You have a site, but the crawlers looks at the llms.txt and uses that, except the content is all wrong and bares no resemblance to the actual content of the page.

If you really care about your content being picked up by the scrapers, why not structure it better? Most of the LLMs are pretty much black boxes, so we don't really know what a better structure would look like, but I would make the guess that involves simplifying your HTML and removing irrelevant tokens.

Re: Llms.txt

#80

Post by pasiaj »

CuriousCosmic wrote: post #33
JimDabell wrote: post #10

This is not how these kinds of things should be designed for the web. Instead of putting resources in the root of the web, this is what /.well-known/ was designed for. See RFC 5785: https://datatracker.ietf.org/doc/html/rfc5785 Instead of munging URLs to get alternate formats, this is what content negotiation or rel=alternate were designed for. I’m not sure making it easier to consume content is something that is nee…

Fwiw the active RFC is 8615 as RFC5785 is obsolete. https://datatracker.ietf.org/doc/html/rfc8615

Having two different RFCs on this is clearly causing confusion!

I think I'll create a new RFC to supersede both to clear up the situation.

Post reply on HN