101–110 of 291 posts
I got DoSed by ClaudeBot (Anthropic) just last week. Hitting a website I manage 700,000 times in one month and tripping our bandwidth limit with our hosting provider. What a PITA to have to investigate that, figure it out, block the user agent, and work with hosting provider support to get the limit lifted as a courtesy. Noticed that the ChatGPT bot was 2nd in traffic to this site, just not enough to cause trouble.
Earlier quoted context omitted.
Nobody has unlimited resources. Everything is a cost-benefit analysis. For highly valuable information, they might throw the GDP of a small country at scraping your site. But most information isn't worth that. And there are a lot of bad actors who don't have the resources you're thinking of that are trying to compete with the big guys on a budget. This would cut them out of the equation.
Make all websites intentionally waste energy as a strategy to defeat unscrupulous operators has negative costs and marginal benefits.
The idea that you should pay for content shouldn't be an insane pipedream. It should be the default on the internet.
Maybe then we wouldn't be in the situation where getting new users is an existential threat to the majority of websites.
Earlier quoted context omitted.
The problem is that the server also has to do the work. Fine for an infrequent auth challenge. Not so fine for every single data request.
Right it would need an algorithm with widely different encryption speeds vs decryption speeds. Lattice-based cryptography maybe?
And this is why the Internet has become a maze of captcha’s.
"We have met the enemy and he is us." -- Walt Kelly
I got DoSed by ClaudeBot (Anthropic) just last week. Hitting a website I manage 700,000 times in one month and tripping our bandwidth limit with our hosting provider. What a PITA to have to investigate that, figure it out, block the user agent, and work with hosting provider support to get the limit lifted as a courtesy. Noticed that the ChatGPT bot was 2nd in traffic to this site, just not enough to cause trouble.
robots.txt did not work?
Earlier quoted context omitted.
All of the crawlers present themselves as being from one of the major companies, even if they’re not. Setting user-agent headers is easy.
at the same time, all the AI providers have some kind of web based AI agent, so let snot pretend they're crafting their services in care of other peoples websites.
Excuse my ignorance, but is it time to update the open source licenses in the light of this behavior? If so, what should the evolved license wording be? I appreciate that this could be easily circumvented by a 'bad actor', but it would make this abuse overt...
From my little understanding, we have a sort of agreement in place with an item called robot.txt that's more or less a hanshake with such scrapers. Of course, the issue is these scrapers are blatantly ignoring robots.txt A license can help as well, but what's a license without enforcement? These companies are simply treating the courts as a cost to do business.
All these JS-heavy "anti bot" measures do is further entrench the browser monopoly, making it much harder for the minority of independents, while those who pay big $$$ can still bypass them. Instead I recommend a simple HTML form that asks questions with answers that LLMs cannot yet figure out or get consistently wrong. The more related to the site's content the questions are, the better; I remember some electronics…
How does JS entrench a browser monopoly? If you're not using vendor-specific JS extensions or non-standard APIs any browser should be able to execute your JS. Like most web developers I don't have a lot of patience for the people who refuse to run JS on their clients.
Seeing what used to be simple HTML forms turned into bloated invasive webapps to accomplish the exact same thing seriously angers me; and everyone else who wanted an easily accessible and freedom-preserving Internet.