Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llms.txt

llmstxt.org

Post reply on HN
Print view View original

101–110 of 191 posts

Re: Llms.txt

#101

Post by imjonse »

jph00 wrote: post #74
TZubiri wrote: post #12

What problem does this solve?

From the post describing llms.txt ( https://www.answer.ai/posts/2024-09-03-llmstxt.html ): "The problem this solves is that today, constructing the right context for LLMs based on a website is ambiguous — do you: 1. Crawl the sitemap and include every page, trying to automatically format into an LLM-friendly form? 2. Selectively include external links in addition to the sitemap? 3. For specific domains like software…

I agree site authors should be able to tell what content they would like to be used for LLM training (even though that opinion will likely be ignored by LLM training scrapers), but the format of it is really up to those gathering and cleaning the data.

It is extra burden for content authors to start thinking about LLM training requirements especially if those may change at a fast pace.

It is also something LLM scrapers would need to validate/check/reformat anyway to protect from errors/trolling/poisoning of the data since even if most authors would provide curated info, not all will.

Re: Llms.txt

#102

Post by gtirloni »

BrutalCoding wrote: post #89
dns_snek wrote: post #88

Earlier quoted context omitted.

> a crawl requesting about 2 pages per second. It was so intense [...] Do 2 pages per second really count as "intense" activity? Even if I was hosting a website on a $5 VPS, I don't think I'd even notice anything short of 100 requests per second, in terms of resource usage.

I assumed that he meant per client. Having a limit of 2 pages a second for a single client seems like a reasonable amount to me.

If ypu open DevTools and visit any website these days, you'll be surprised.

Re: Llms.txt

#103

Post by idf00 »

yochem wrote: post #94
jph00 wrote: post #85

Earlier quoted context omitted.

Markdown is plain text. llms.txt is meant to be displayed in plain text format, not rendered to html.

Yes, and .py is "plain" text too. The extension however helps with signaling the intend of the file. Also, there is something to say for the argument "there is no such thing as plain text" [0] [0]: https://youtu.be/gd5uJ7Nlvvo

If you had python code and you didn't want it to have syntax highlighting or be run/imported or any of the other normal things that you do with python files, it might make sense to have python code in a .txt. file.

Same idea here IMO. .md would signal the wrong intent, as you don't want to render it to markdown formatting or read as a markdown file normally is. You want it to be read as plain unrendered text.

Sam

Re: Llms.txt

#104

Post by rmholt »

> We furthermore propose that pages on websites that have information that might be useful for LLMs to read provide a clean markdown version of those pages at the same URL as the original page, but with .md appended.

Not happening, that's like asking websites to provide an ad-free, brand identity free version for free. And we can't have that now can we

Re: Llms.txt

#105

Post by idf00 »

imjonse wrote: post #101
jph00 wrote: post #74

Earlier quoted context omitted.

From the post describing llms.txt ( https://www.answer.ai/posts/2024-09-03-llmstxt.html ): "The problem this solves is that today, constructing the right context for LLMs based on a website is ambiguous — do you: 1. Crawl the sitemap and include every page, trying to automatically format into an LLM-friendly form? 2. Selectively include external links in addition to the sitemap? 3. For specific domains like software…

I agree site authors should be able to tell what content they would like to be used for LLM training (even though that opinion will likely be ignored by LLM training scrapers), but the format of it is really up to those gathering and cleaning the data. It is extra burden for content authors to start thinking about LLM training requirements especially if those may change at a fast pace. It is also something LLM scrape…

It's not to help people train models. It's for end-users to use in an LLM context (like Claude projects or cursor) to help them use your tool better.

Re: Llms.txt

#106

Post by Devasta »

Anything that makes things more pleasant for LLMs is to be opposed. Their devs don't care about your opinion, they'll vacuum up whatever they want and use it for any purpose and you degrade yourself if you think the makers of these LLMs can be reasoned with. They are flooding the internet with crap, ruining basically every art site in the process, and destroying any avenues of human connection they can.

Why make life easier for them when they are committed to making life more difficult for you?

Re: Llms.txt

#107

Post by efilife »

otherme123 wrote: post #65
autoexec wrote: post #38

Earlier quoted context omitted.

As much as these companies should respect our preferences, it's very clear that they won't. It wouldn't matter to these companies if it was outright illegal, "pretty please" certainly isn't going to cut it. You can't stop scraping and the harder people try the worse their sites become for everyone else. Throwing up a robots.txt or llms.txt that calls out their bad behavior isn't a bad idea, but it's not likely to hel…

In one of my robots.txt I have "Crawl-Delay: 20" for all User-Agents. Pretty much every search bot respect that Crawl-Delay, even the shaddy ones. But one of the most known AI bots launched a crawl requesting about 2 pages per second. It was so intense that it got banned by the "limit_req_ " and "limit_rate_ " of the nginx config. Now I have it configured to always get a 444 by user agent and ip range no matter how m…

Rookie question, how do you ban an ip range?

Re: Llms.txt

#108

Post by jraph »

Tepix wrote: post #92
otherme123 wrote: post #63

Earlier quoted context omitted.

In your nginx.conf, http block, add include /etc/nginx/useragent.rules; In /etc/nginx/useragent.rules map $http_user_agent $badagent { default 0; ~facebook 1; [...] ~PerplexityBot 1; } In your site.conf, server block, add if ($badagent) { return 444; }

Ideally i would like these crawlers to access /robots.txt but nothing else. Only if they ignore robots.txt the access rules will stop them.

You can probably write a specific location block for robots.txt which will have a higher priority.

See also https://stackoverflow.com/questions/5238377/nginx-location-p...

Re: Llms.txt

#110

Post by ulrikrasmussen »

Wouldn't nice old-school static HTML markup be just as consumable by an LLM? I'd love it if that was served to LLM user agents - I'd spoof my browser to pretend to be an LLM in a jiffy!
Post reply on HN