How do they get overloaded? Is the website too slow? I have a quite big wiki online and barely see any impact from bots.
In these discussions no one will admit this, but the answer is generally yes. Websites written in python and stuff like that.
We can't have nice things because of AI scrapers
51–60 of 279 posts
Re: We can't have nice things because of AI scrapers
#52Resurfacing a proposal I put out on llms-txt: https://github.com/AnswerDotAI/llms-txt/issues/88 We should add optional `tips` addresses in llms.txt files. We're also working on enabling and solving this at Grove.city. Human Agent Human Tips don't account for all the edge cases, but they're a necessary and happy neutral medium. Moving fast. Would love to share more with the community. Wrote about it here: https://x.co…
I like the idea, (original) content creators being credited is good for the entire ecosystem.
Though if LLMs are willingly ignoring robots.txt, often hiding themselves or using third party scraped data- are they going to pay?
Re: We can't have nice things because of AI scrapers
#53How do they get overloaded? Is the website too slow? I have a quite big wiki online and barely see any impact from bots.
There’s a lot of factors. Depends how well your content lends itself to being cached by a CDN, the tech you (or your predecessors) chose to build it with, and how many unique pages you have. Even with pretty aggressive caching, having a couple million pages indexed adds up real fast. Especially if you weren’t fortunate enough to inherit a project using a framework that makes server side rendering easy.
Re: We can't have nice things because of AI scrapers
#54How do they get overloaded? Is the website too slow? I have a quite big wiki online and barely see any impact from bots.
The API seems to be written in Perl: https://github.com/metabrainz/musicbrainz-server
Re: We can't have nice things because of AI scrapers
#55How do they get overloaded? Is the website too slow? I have a quite big wiki online and barely see any impact from bots.
If you have a lot of pages, AI bots will scrape every single one on a loop - wiki's generally don't have anywhere near the number of pages as an incremented entity primary id. I have a few million pages on a tiny website and it gets hammered by AI bots all day long. I can handle it, but it's a nuisance and they're basically just scraping garbage (statistics pages of historical matches or user pages that have essentially no content).
Many of them don't even self-identify and end up scraping with shrouded user-agents or via bot-farms. I've had to block entire ASNs just to tone it down. It also hurts good-faith actors who genuinely want to build on top of our APIs because I have to block some cloud providers.
I would guess that I'm getting anywhere from 10-25 AI bot requests (maybe more) per real user request - and at scale that ends up being quite a lot. I route bot traffic to separate pods just so it doesn't hinder my real users' experience[0]. Keep in mind that they're hitting deeply cold links so caching doesn't do a whole lot here.
[0] this was more of a fun experiment than anything explicitly necessary, but it's proven useful in ways I didn't anticipate
Re: We can't have nice things because of AI scrapers
#56I haven't really looked but I wonder if there are any IP reputation services tracking AI crawlers the same way they track tor relays and vpns and such. If those databases were accurate it seems like you could prevent those crawlers from ever hitting the site. Maybe they change too often/too quickly.
Re: We can't have nice things because of AI scrapers
#57The SQLite team faced a similar problem last year, and Richard Hipp (the creator of SQLite) made almost the same comment:
"The malefactor behind this attack could just clone the whole SQLite source repository and search all the content on his own machine, at his leisure. But no: Being evil, the culprit feels compelled to ruin it for everyone else. This is why you don't get to keep nice things...."
Re: We can't have nice things because of AI scrapers
#58Earlier quoted context omitted.
> They won't believe a random site when it says "Look, stop hitting our API, you can pick all of this data in one go, over in this gzipped tar file." What mechanism does a site have for doing that? I don't see anything in robots.txt standard about being able to set priority but I could be missing something.
It’s not great, but you could add it to the body of a 429 response.
Genuinely curious: do programs read bodies of 429 responses? In the code bases that I have seen, 429 is not read beyond the code itself
Re: We can't have nice things because of AI scrapers
#59I actually deleted my web site early 2025 and removed the A record from DNS because of AI scraper traffic. It had been up for 22 years. Nothing important or particularly useful on it but it's an indicator of the times.
Re: We can't have nice things because of AI scrapers
#60I actually deleted my web site early 2025 and removed the A record from DNS because of AI scraper traffic. It had been up for 22 years. Nothing important or particularly useful on it but it's an indicator of the times.
But why?