After I opened up https://sugaku.net to be usable without login, it was astounding how quickly the crawlers started. I'd like the site to be accessible to all, but I've had to restrict most of the dynamic features to logged in users, restrict robots.txt, use cloudflare to block AI crawlers and bad bots, and I'm still getting ~1M automated requests per day (compared to ~1K organic), so I think I'll need to restrict the site to logged in users soon.
Devs say AI crawlers dominate traffic, forcing blocks on entire countries
121–130 of 291 posts
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#122Earlier quoted context omitted.
I just hit a site with this --- and hit the back button immediately.
Better than a 500 error
Also better than a third party service IMO because of the privacy implications. You could even potentially give users the choice (complete overkill but technically you could do it).
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#123After I opened up https://sugaku.net to be usable without login, it was astounding how quickly the crawlers started. I'd like the site to be accessible to all, but I've had to restrict most of the dynamic features to logged in users, restrict robots.txt, use cloudflare to block AI crawlers and bad bots, and I'm still getting ~1M automated requests per day (compared to ~1K organic), so I think I'll need to restrict th…
One thing that worked well for me was layering obstacles
It really sucks that this is the way things are, but what I did was
10 requests for pages in a minute, you get captchad (with a little apology and the option to bypass it by logging in). asset loads don’t count
After a captcha pass, 100 requests in an hour gets you auth walled
It’s really shitty but my industry is used to content scraping.
This allows legit users to get what they need. Although my users maybe don’t need prolonged access ahem.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#124All these JS-heavy "anti bot" measures do is further entrench the browser monopoly, making it much harder for the minority of independents, while those who pay big $$$ can still bypass them. Instead I recommend a simple HTML form that asks questions with answers that LLMs cannot yet figure out or get consistently wrong. The more related to the site's content the questions are, the better; I remember some electronics…
Or require every fresh "unique" visitor to run some JS that takes X seconds to compute. It's not nice for visitors using a very old smartphone, but it's arguably less-exclusionary than some of the tests and third-party gatekeepers that exist now. In many cases we don't actually care about telling if someone is truly a human alone, as much as ensuring that they aren't a throwaway sockpuppet of a larger automated syste…
that takes X seconds to compute.
Those who have the computing resources to do commercial scraping will easily get past that.
In contrast, there are still many questions which a human can easily answer, but even the best LLMs currently can't.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#125Excuse my ignorance, but is it time to update the open source licenses in the light of this behavior? If so, what should the evolved license wording be? I appreciate that this could be easily circumvented by a 'bad actor', but it would make this abuse overt...
They already ignore copyright. The open source licenses are based on copyright, so changing the licenses wouldn't do squat, they'd still ignore it.
See also: Meta being sued for torrenting. Since this is an Ars Technica article, here's another one: https://arstechnica.com/tech-policy/2025/02/meta-torrented-o...
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#126Earlier quoted context omitted.
It sends a great message to crack down on these companies, as long as you mention why it’s blocked.
"You're cut off from access to knowledge because you live in the same country as AI researchers"?
A DoS attack is a DoS attack even if someone is pretending to be a “Researcher.”
People in Iran, Russia, etc get annoyed with sanctions but that’s kind of the point. If your government isn’t responding appropriately, yes you’ll get shafted it’s what you do after that which solves the problem.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#127nice, a free way to keep our IPFS pins alive
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#128After I opened up https://sugaku.net to be usable without login, it was astounding how quickly the crawlers started. I'd like the site to be accessible to all, but I've had to restrict most of the dynamic features to logged in users, restrict robots.txt, use cloudflare to block AI crawlers and bad bots, and I'm still getting ~1M automated requests per day (compared to ~1K organic), so I think I'll need to restrict th…
One thing that worked well for me was layering obstacles It really sucks that this is the way things are, but what I did was 10 requests for pages in a minute, you get captchad (with a little apology and the option to bypass it by logging in). asset loads don’t count After a captcha pass, 100 requests in an hour gets you auth walled It’s really shitty but my industry is used to content scraping. This allows legit use…
What happens if you use the proper rate limiting status of 429? It includes a next retry time [1]. I'm curious what (probably small) fraction would respect it.
[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#129Earlier quoted context omitted.
This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations. Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200…
Check out Anubis - it's not quite what you're suggesting but similar in concept: https://anubis.techaro.lol/
Maybe once you can use something more professional as an interstitial page
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#130Earlier quoted context omitted.
This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations. Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200…
Check out Anubis - it's not quite what you're suggesting but similar in concept: https://anubis.techaro.lol/
How does it work? I don’t have time to read the code, and the website/docs seem to be under construction.
Does it have the client do a bunch of SHA-256 hashes?