Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llms.txt

llmstxt.org

Post reply on HN
Print view View original

81–90 of 191 posts

Re: Llms.txt

#81

Post by darrenf »

russellbeattie wrote: post #22
Joker_vD wrote: post #19

Earlier quoted context omitted.

If only this RFC was well-known among the people who actually put stuff out on the Web.

If only that RFC didn't make it a hidden directory. I can think of a dozen reasons why hiding that folder is a horrible idea, and not a single one for why it would be a good thing to do.

> If only that RFC didn't make it a hidden directory.

There's zero guidance on configuring how URIs under `/.well-known/` should be served at all, is there? They just reserve/sandbox the initial path component for the URI schemes which support it. That's it. It's the developers' choice to implement it as a directory - hidden or otherwise - on a filesystem; neither RFC says they SHOULD or MUST be served in such a way.

(The updated RFC says "e.g., on a filesystem" in section 4.1, and mentions directories in section 4.4 in a way that, to my eyes, pretty much recommends against making it hidden)

Re: Llms.txt

#82

Post by jph00 »

mrweasel wrote: post #79

Wouldn't this open up for manipulating LLMs? You have a site, but the crawlers looks at the llms.txt and uses that, except the content is all wrong and bares no resemblance to the actual content of the page. If you really care about your content being picked up by the scrapers, why not structure it better? Most of the LLMs are pretty much black boxes, so we don't really know what a better structure would look like, b…

llms.txt is not for crawlers/scrapers, it's for creating context documents at inference time. You place it on your own site -- presumably if you create an llms.txt you're not looking to manipulate anyone, but to do your best to provide your site's key information in an AI-friendly way.

Re: Llms.txt

#83

Post by 13hunteo »

KTibow wrote: post #44
JimDabell wrote: post #10

This is not how these kinds of things should be designed for the web. Instead of putting resources in the root of the web, this is what /.well-known/ was designed for. See RFC 5785: https://datatracker.ietf.org/doc/html/rfc5785 Instead of munging URLs to get alternate formats, this is what content negotiation or rel=alternate were designed for. I’m not sure making it easier to consume content is something that is nee…

It's based on the series of robots.txt, humans.txt, security.txt, etc

FWIW, security.txt is placed in .well-known/

Re: Llms.txt

#84

Post by jeffhuys »

aftbit wrote: post #23
internetter wrote: post #6

To disallow: Amazonbot, anthropic-ai, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-Web, cohere-ai, DataForSeoBot, Diffbot, Webzio-Extended, FacebookBot, FriendlyCrawler, Google-Extended, GPTBot, 0AI-SearchBot, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, omgili, omgilibot, PerplexityBot, Quora-Bot, TurnitinBot For all of these bots, User-agent: Disallow: / For more infor…

s/consider of/consider if/

Consider if course this

Re: Llms.txt

#85

Post by jph00 »

romantomjak wrote: post #78

I find it confusing that author proposes llms.txt, but the content is actually markdown? I get that they tried to follow the convention, but then why not make it a simple text file like the robots.txt is?

Markdown is plain text. llms.txt is meant to be displayed in plain text format, not rendered to html.

Re: Llms.txt

#86

Post by immibis »

bdcravens wrote: post #36
bnchrch wrote: post #35

Earlier quoted context omitted.

Exactly. robots.txt is useless and those that think its useful for preventing unwanted crawling are clueless

It's a "Keep off the grass" sign. The polite will obey, the ones who are truly the problem will not.

So your site ends up not being findable by Google, and missing from the Wayback Machine once it's dead, but it does nothing for LLMs.

Re: Llms.txt

#87

Post by mrweasel »

jph00 wrote: post #82
mrweasel wrote: post #79

Wouldn't this open up for manipulating LLMs? You have a site, but the crawlers looks at the llms.txt and uses that, except the content is all wrong and bares no resemblance to the actual content of the page. If you really care about your content being picked up by the scrapers, why not structure it better? Most of the LLMs are pretty much black boxes, so we don't really know what a better structure would look like, b…

llms.txt is not for crawlers/scrapers, it's for creating context documents at inference time. You place it on your own site -- presumably if you create an llms.txt you're not looking to manipulate anyone, but to do your best to provide your site's key information in an AI-friendly way.

> if you create an llms.txt you're not looking to manipulate anyone

You don't know me :-)

My suggestion is that someone might want taint the data that goes into an LLM.

Let's say you have a website with guides, examples and tips and tricks for writing bash. What would prevent you from pointing the LLMs to separate content which would contain broken examples and code with a number of security issues, because you long term would want to exploit the code generated by the LLMs.

Re: Llms.txt

#88

Post by dns_snek »

otherme123 wrote: post #65
autoexec wrote: post #38

Earlier quoted context omitted.

As much as these companies should respect our preferences, it's very clear that they won't. It wouldn't matter to these companies if it was outright illegal, "pretty please" certainly isn't going to cut it. You can't stop scraping and the harder people try the worse their sites become for everyone else. Throwing up a robots.txt or llms.txt that calls out their bad behavior isn't a bad idea, but it's not likely to hel…

In one of my robots.txt I have "Crawl-Delay: 20" for all User-Agents. Pretty much every search bot respect that Crawl-Delay, even the shaddy ones. But one of the most known AI bots launched a crawl requesting about 2 pages per second. It was so intense that it got banned by the "limit_req_ " and "limit_rate_ " of the nginx config. Now I have it configured to always get a 444 by user agent and ip range no matter how m…

> a crawl requesting about 2 pages per second. It was so intense [...]

Do 2 pages per second really count as "intense" activity? Even if I was hosting a website on a $5 VPS, I don't think I'd even notice anything short of 100 requests per second, in terms of resource usage.

Re: Llms.txt

#89

Post by BrutalCoding »

dns_snek wrote: post #88
otherme123 wrote: post #65

Earlier quoted context omitted.

In one of my robots.txt I have "Crawl-Delay: 20" for all User-Agents. Pretty much every search bot respect that Crawl-Delay, even the shaddy ones. But one of the most known AI bots launched a crawl requesting about 2 pages per second. It was so intense that it got banned by the "limit_req_ " and "limit_rate_ " of the nginx config. Now I have it configured to always get a 444 by user agent and ip range no matter how m…

> a crawl requesting about 2 pages per second. It was so intense [...] Do 2 pages per second really count as "intense" activity? Even if I was hosting a website on a $5 VPS, I don't think I'd even notice anything short of 100 requests per second, in terms of resource usage.

I assumed that he meant per client. Having a limit of 2 pages a second for a single client seems like a reasonable amount to me.

Re: Llms.txt

#90

Post by Tepix »

internetter wrote: post #6

To disallow: Amazonbot, anthropic-ai, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-Web, cohere-ai, DataForSeoBot, Diffbot, Webzio-Extended, FacebookBot, FriendlyCrawler, Google-Extended, GPTBot, 0AI-SearchBot, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, omgili, omgilibot, PerplexityBot, Quora-Bot, TurnitinBot For all of these bots, User-agent: Disallow: / For more infor…

There's a typo in your file: achive
Post reply on HN