> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.
The article you're replying to describes in explicit detail how the bots and their operators have directly caused and continue to knowingly cause real harm to the author and others in similar positions, both financial costs and administrative/maintenance burdens that would not have otherwise been required.
You then respond "but if you block the bots then I won't be able to use the bots, and that harms me because I might have to read your web site myself..."
Are you serious?
> That might or might not be what you expected, but it's worth taking into account.
I would wager that for almost everyone who is blocking bots after getting functionally DDoSed by them this is absolutely an expected and desired outcome.
> And finally, something worth noting is that there are so many websites whose owners complain about bots, but the real problem is that the website is poorly built and should be improved anyway.
Both can be true. If you operate a git repository with a public-facing web interface for example there are going to be a lot of possible operations that are inherently expensive but also incredibly rarely used by normal users so it doesn't really matter, but the bots now ignore your robots.txt and are programmed to go after every link they can find, so they trigger every single possible expensive operation more times in a night than your actual users ever have in the history of the site while dividing requests across so many different IP addresses that rate limiting becomes impossible at the individual scale. These days they're even feeding the discovered URLs back in to their models to have them invent new possible URLs and trying those in hopes of finding content never publicly linked. They will send you thousands of requests for URLs that they literally made up.
Sometimes the site is in fact badly coded and operations that should be simple have higher costs due to bad design but you don't have to look very far to find situations where legitimately high-cost resources are exposed to the public because they're expected to be used in a non-abusive way. We should always be standing up against abuse of public resources, unless we want to lose them altogether.
> Bot traffic is not necessarily bad.
You are right, but whether it's good or bad more or less comes down to a cost/benefit analysis. As we've already covered infinite times, these bots being used to train LLMs cause significant real costs to the operators of these sites. What benefits do they offer in return? We know the clickthrough rates are terrible, so what other reasons would site operators have to make those real costs worth it? So someone can get a response back from an algorithm that confidently misinterprets or even entirely misreports what the data actually meant?
Even the most die-hard "information wants to be free" types who absolutely want their datasets trained on would probably prefer that the bots accessed the data directly via an API or downloaded a database dump rather than spidering and scraping a web interface intended for humans.