[flagged]
Llms.txt
111–120 of 191 posts
Re: Llms.txt
#112Was I the only one that found `docs.fastht.ml/llms.txt` more useful than both fastht.ml and docs.fastht.ml?
Zooming out, it's interesting how many (especially dev-focused) tools & frameworks have landing sites that are so incomprehensible to me. They look like marketing sites but don't even explain what the thing they're offering does. llms.txt almost sounds like a forcing function for someone to write something that is not just more suitable for LLMs, but humans.
This ties in to what others are saying: a good enough LLM should understand a resource that a human can understand, ideally. But also, maybe we should make the main resources more understandable to humans?
Re: Llms.txt
#113LLMs are already nearly as smart as humans. Whatever needs to be known should be able to inferred from the documentation
Re: Llms.txt
#114[flagged]
[deleted]
Re: Llms.txt
#115This is not how these kinds of things should be designed for the web. Instead of putting resources in the root of the web, this is what /.well-known/ was designed for. See RFC 5785: https://datatracker.ietf.org/doc/html/rfc5785 Instead of munging URLs to get alternate formats, this is what content negotiation or rel=alternate were designed for. I’m not sure making it easier to consume content is something that is nee…
[deleted]
Re: Llms.txt
#116Anything that makes things more pleasant for LLMs is to be opposed. Their devs don't care about your opinion, they'll vacuum up whatever they want and use it for any purpose and you degrade yourself if you think the makers of these LLMs can be reasoned with. They are flooding the internet with crap, ruining basically every art site in the process, and destroying any avenues of human connection they can. Why make life…
[deleted]
Re: Llms.txt
#117To disallow: Amazonbot, anthropic-ai, AwarioRssBot, AwarioSmartBot, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-Web, cohere-ai, DataForSeoBot, Diffbot, Webzio-Extended, FacebookBot, FriendlyCrawler, Google-Extended, GPTBot, 0AI-SearchBot, ImagesiftBot, Meta-ExternalAgent, Meta-ExternalFetcher, omgili, omgilibot, PerplexityBot, Quora-Bot, TurnitinBot For all of these bots, User-agent: Disallow: / For more infor…
[deleted]
Re: Llms.txt
#118LLMs.txt should let me specify the $$$ price that companies must send me to train models on my content.
[deleted]
Re: Llms.txt
#119Can we not put another file in the root please? That's what /.well-known/ is for. And while I'm here, authors of unix tools, please use $XDG_CONFIG_HOME. I'm tired of things shitting dot-droppings into my home directory.
Agreed, the home directory is such a mess
Re: Llms.txt
#120Earlier quoted context omitted.
In one of my robots.txt I have "Crawl-Delay: 20" for all User-Agents. Pretty much every search bot respect that Crawl-Delay, even the shaddy ones. But one of the most known AI bots launched a crawl requesting about 2 pages per second. It was so intense that it got banned by the "limit_req_ " and "limit_rate_ " of the nginx config. Now I have it configured to always get a 444 by user agent and ip range no matter how m…
Rookie question, how do you ban an ip range?
You can do it in a few places, but I use my network firewall for this I use PFSense at home, but there are many enterprise grade brands).
It's common to use the host's firewall as well (nftables, firewalld, or iptables).
You can do it at the webserver too, with access.conf in nginx. Apache uses mod_authz.
I usually do it at the network though so it uses the least amount of resources (no connection ever gets to the webserver). Though if you only have access to your webserver it's faster to ban it there than to send a request to the network team (depending on your org, some orgs might have this automated).