Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llms.txt

llmstxt.org

Post reply on HN
Print view View original

61–70 of 191 posts

Re: Llms.txt

#61

Post by jraph »

internetter wrote: post #6

To disallow: Amazonbot, anthropic-ai, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-Web, cohere-ai, DataForSeoBot, Diffbot, Webzio-Extended, FacebookBot, FriendlyCrawler, Google-Extended, GPTBot, 0AI-SearchBot, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, omgili, omgilibot, PerplexityBot, Quora-Bot, TurnitinBot For all of these bots, User-agent: Disallow: / For more infor…

I've seen some of these bots take a lot of CPU on my server, especially when browsing my (very small) forgejo instance. I banned them with a 444 error [1] in the reverse proxy settings as a temporary measure that became permanent, and then some more from this list [2], but I will consider yours as well, thanks for sharing.

    if ($http_user_agent ~ facebook) { return 444; }
    if ($http_user_agent ~ Amazonbot) { return 444; }
    if ($http_user_agent ~ Bytespider) { return 444; }
    if ($http_user_agent ~ GPTBot) { return 444; }
    if ($http_user_agent ~ ClaudeBot) { return 444; }
    if ($http_user_agent ~ ImagesiftBot) { return 444; }
    if ($http_user_agent ~ CCBot) { return 444; }
    if ($http_user_agent ~ ChatGPT-User) { return 444; }
    if ($http_user_agent ~ omgili) { return 444; }
    if ($http_user_agent ~ Diffbot) { return 444; }
    if ($http_user_agent ~ Claude-Web) { return 444; }
    if ($http_user_agent ~ PerplexityBot) { return 444; }
(edit: see replies to do it in a cleaner way)

[1] https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#ngin...

[2] https://blog.cloudflare.com/declaring-your-aindependence-blo...

Re: Llms.txt

#62

Post by gdsdfe »

The very idea is a bit silly, why would you help an llm understand a website!? Isn't that proof that the llm is less than capable and you should either use or develop a better model? Like the whole premise makes no sense to me

Re: Llms.txt

#63

Post by otherme123 »

jraph wrote: post #61
internetter wrote: post #6

To disallow: Amazonbot, anthropic-ai, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-Web, cohere-ai, DataForSeoBot, Diffbot, Webzio-Extended, FacebookBot, FriendlyCrawler, Google-Extended, GPTBot, 0AI-SearchBot, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, omgili, omgilibot, PerplexityBot, Quora-Bot, TurnitinBot For all of these bots, User-agent: Disallow: / For more infor…

I've seen some of these bots take a lot of CPU on my server, especially when browsing my (very small) forgejo instance. I banned them with a 444 error [1] in the reverse proxy settings as a temporary measure that became permanent, and then some more from this list [2], but I will consider yours as well, thanks for sharing. if ($http_user_agent ~ facebook) { return 444; } if ($http_user_agent ~ Amazonbot) { return 444…

In your nginx.conf, http block, add

    include /etc/nginx/useragent.rules;
In /etc/nginx/useragent.rules

    map $http_user_agent $badagent {
        default 0;
        ~facebook 1;
        [...]
        ~PerplexityBot 1;
    }
In your site.conf, server block, add

    if ($badagent) {
        return 444;
    }

Re: Llms.txt

#64

Post by imtringued »

tbrownaw wrote: post #32

Is this trying to be what the semantic web was supposed to be? Or is it trying to be "OpenAPI for things that aren't REST/JSON-RPC APIs"? (Are those even any different?) And we already have plenty of standards for library documentation. Man pages, info pages, Perldoc, Javadoc, ...

It looks like a very poorly thought out HATEOAS and the reason why nobody uses HATEOAS is that for some reason the creator insisted that knowing a set of fields associated with a datatype is evil out of band communication and therefore hinders evolvability.

Of course this then leads to a problem. Your API client isn't allowed to invoke hard coded actions or access hard coded fields, it must automatically adjust itself whenever the API changes. In practice means that the types of HATEOAS clients you can write is extremely limited. You can write what basically amounts to an API browser plus a form generator, because anything more complicated needs human level intelligence.

Re: Llms.txt

#65

Post by otherme123 »

autoexec wrote: post #38
internetter wrote: post #6

To disallow: Amazonbot, anthropic-ai, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-Web, cohere-ai, DataForSeoBot, Diffbot, Webzio-Extended, FacebookBot, FriendlyCrawler, Google-Extended, GPTBot, 0AI-SearchBot, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, omgili, omgilibot, PerplexityBot, Quora-Bot, TurnitinBot For all of these bots, User-agent: Disallow: / For more infor…

As much as these companies should respect our preferences, it's very clear that they won't. It wouldn't matter to these companies if it was outright illegal, "pretty please" certainly isn't going to cut it. You can't stop scraping and the harder people try the worse their sites become for everyone else. Throwing up a robots.txt or llms.txt that calls out their bad behavior isn't a bad idea, but it's not likely to hel…

In one of my robots.txt I have "Crawl-Delay: 20" for all User-Agents. Pretty much every search bot respect that Crawl-Delay, even the shaddy ones. But one of the most known AI bots launched a crawl requesting about 2 pages per second. It was so intense that it got banned by the "limit_req_" and "limit_rate_" of the nginx config. Now I have it configured to always get a 444 by user agent and ip range no matter how much they request.

Re: Llms.txt

#66

Post by jeroenhd »

autoexec wrote: post #47
azhenley wrote: post #5

LLMs.txt should let me specify the $$$ price that companies must send me to train models on my content.

No, see you're supposed to create and upload this specially formatted file on all your webservers for free, just to make it a little easier for them to take all your content for free, so that they can then use your content in their products for free, so they can charge other humans money to get your content from their product without any humans ever having to visit your actual website again. What's not to like? If th…

But there is actually a reason to use this standard. See, if your goal is to alter the perception of AI models, like convincing them certain genocides did not exist or that certain people are(n't) criminals, you want AI to index your website as efficiently as possible.

Together with websites that make money off trying to report the truth shielding their content from plagiarism scrapers, this means that setting up a wide range of (AI generated) websites all configured to be ingested easily will allow you to alter public perception much easier.

This spec is very useful in a fairy tale world where everyone wants to help tech giants build better AI models, but also when the goal is to twist the truth rather than improve reliability.

Oh, and I guess projects like Wikipedia are interested in easy information distribution like this. But you can just download a copy of the entire database instead.

Re: Llms.txt

#67

Post by dada78641 »

bawolff wrote: post #39

I'm not that familiar with llms, but surely we are already at the point where web pages can be easily scrapped? Is markdown really an easier format to understand than html? If this is actually useful wouldn't .txt be supperior to markdown for this usecase? Does this solve a problem llms actually have? Not trying to be negative, i'm honestly curious.

I hate to be this person, but... it's scraping, not scrapping. You're scraping the information off the page regardless of what its structure is.

Re: Llms.txt

#68

Post by jph00 »

haswell wrote: post #46
ray_v wrote: post #4

Earlier quoted context omitted.

Ostensibly, everyone posting information on the open web want to share information -- either directly with people or indirectly via search engines _and_ the current crop of llms (which in my mind, serve the same purpose as search engines) I suppose the thing that people maybe don't agree with is the lack of attribution when llms regurgitate information back at the user. That, and the fact that these services are also…

That’s really my primary issue. Google indexing my content and directing traffic to my site is one thing. But unlike search indexing, there is no exchange of value when these LLMs are trained on my content. We all collectively get nothing for our work. It’s theft dressed up as business as usual. I’ll do whatever I reasonably can to avoid feeding the machine and hope some of the ongoing and inevitable legal fights wil…

This proposal isn’t really for training. It’s for end users who want to know what information to include when they’re using models.

Re: Llms.txt

#69

Post by jph00 »

dylan604 wrote: post #15
TZubiri wrote: post #12

What problem does this solve?

It solves the problem of the cat&mouse game of LLMs updating their scrapers by making site owners provide them the data in a format the LLMs have developed around already. You're clearly looking at this from the incorrect point of view. Silly human. Think like a bot. --The bot makers

This proposal isn’t designed to help training. It’s designed to help end-users.

Re: Llms.txt

#70

Post by jraph »

otherme123 wrote: post #63
jraph wrote: post #61

Earlier quoted context omitted.

I've seen some of these bots take a lot of CPU on my server, especially when browsing my (very small) forgejo instance. I banned them with a 444 error [1] in the reverse proxy settings as a temporary measure that became permanent, and then some more from this list [2], but I will consider yours as well, thanks for sharing. if ($http_user_agent ~ facebook) { return 444; } if ($http_user_agent ~ Amazonbot) { return 444…

In your nginx.conf, http block, add include /etc/nginx/useragent.rules; In /etc/nginx/useragent.rules map $http_user_agent $badagent { default 0; ~facebook 1; [...] ~PerplexityBot 1; } In your site.conf, server block, add if ($badagent) { return 444; }

Ah, nice, way better than my string of ifs.
Post reply on HN