141–150 of 291 posts
This is going to start happening to brick and mortar businesses through their customer support channels
I wrote the above some time ago. I think its even more true today. Its practically impossible to crawl the way the bigger players do and with the increased focus on legislation in this area its going to lock out smaller teams even faster.
The old web is dead really. There really needs to be a move to more independent websites. Thankfully we are starting to see more of this like the linked searchmysite discussed earlier today https://news.ycombinator.com/item?id=43467541
I'm being trite, but if you can detect an AI bot, why not just serve them random data? At least they'll be sharing some of the pain they inflict.
You mean like this? [2025-03-19] https://blog.cloudflare.com/ai-labyrinth/ > Trapping misbehaving bots in an AI Labyrinth > Today, we’re excited to announce AI Labyrinth, a new mitigation approach that uses AI-generated content to slow down, confuse, and waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives.
I recently started a side-project with a "code everything in prod" approach for fun. I've done this many times over the past 20 years and the bot traffic is usually harmless, but this has been different. I haven't advertised the hostname anywhere, and in less than 24 hours I had a bunch of spam form submissions. I've always expected this after minor publicity, but not "start server, instantly get raided by bots perfo…
Wow it is so surreal to see a project of mine on Ars Technica! It's such an honor!
You can periodically remove tracking data for entries older than a threshold -- e.g. once a minute or so (adjustable) remove tracked entries that haven't made a request in the past minute to keep memory usage down.
That'd effectively rate limit the worst offenders with minimal impact on most well-behaved edge-case users (like me running NoScript for security) while also wasting less energy globally on unnecessary computation through the proof-of-work scheme, wouldn't it? Is there some reason I'm not thinking of that would prevent that from working?
Earlier quoted context omitted.
Check out Anubis - it's not quite what you're suggesting but similar in concept: https://anubis.techaro.lol/
How does it work? I don’t have time to read the code, and the website/docs seem to be under construction. Does it have the client do a bunch of SHA-256 hashes?
SHA2 can run on ASICs and isn't memory-hard, so I'm hoping someone will add something tougher
Earlier quoted context omitted.
Maybe there is a way for the server to ask the client to do the work? Something similar to proof-of-work but on a much smaller scale than Bitcoin.
just add some delay to your response, we don't have to waste any more energy on meaningless calculation.
Earlier quoted context omitted.
Not if they hop to a different IP address every few requests. And they generally aren't bothered slow responses. It's not like they have to wait for one request to finish before they make another one (especially if they are making requests from thousands of machines).
You're saying that large companies are hitting individual websites with thousands of unrelated IP addresses?