Earlier quoted context omitted.
Check out Anubis - it's not quite what you're suggesting but similar in concept: https://anubis.techaro.lol/
Maybe once you can use something more professional as an interstitial page
[flagged]
Devs say AI crawlers dominate traffic, forcing blocks on entire countries
131–140 of 291 posts
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#132All these JS-heavy "anti bot" measures do is further entrench the browser monopoly, making it much harder for the minority of independents, while those who pay big $$$ can still bypass them. Instead I recommend a simple HTML form that asks questions with answers that LLMs cannot yet figure out or get consistently wrong. The more related to the site's content the questions are, the better; I remember some electronics…
Or require every fresh "unique" visitor to run some JS that takes X seconds to compute. It's not nice for visitors using a very old smartphone, but it's arguably less-exclusionary than some of the tests and third-party gatekeepers that exist now. In many cases we don't actually care about telling if someone is truly a human alone, as much as ensuring that they aren't a throwaway sockpuppet of a larger automated syste…
IIRC that's basically already part of what Cloudflare Turnstile does
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#133Earlier quoted context omitted.
Honestly it's hard to tell how much more value the LLM people are going to get out of another copy of the internet. It feels a lot like they're stuck for improvements but management doesn't want to hear it.
It's a bit strange to talk about stuck when the most recent breakthrough is less than a year old.
I’m not sure what you mean by breakthrough, but if you’re talking about Deepseek, it’s more of an incremental improvement than a breakthrough.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#134After I opened up https://sugaku.net to be usable without login, it was astounding how quickly the crawlers started. I'd like the site to be accessible to all, but I've had to restrict most of the dynamic features to logged in users, restrict robots.txt, use cloudflare to block AI crawlers and bad bots, and I'm still getting ~1M automated requests per day (compared to ~1K organic), so I think I'll need to restrict th…
One thing that worked well for me was layering obstacles It really sucks that this is the way things are, but what I did was 10 requests for pages in a minute, you get captchad (with a little apology and the option to bypass it by logging in). asset loads don’t count After a captcha pass, 100 requests in an hour gets you auth walled It’s really shitty but my industry is used to content scraping. This allows legit use…
What is your website?
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#135> many AI companies engage in web crawling Individuals do too. Tools like https://github.com/unclecode/crawl4ai make it simple to obtain public content but also paywalled content including ebooks, forums and more. I doubt these folks are trying to do a DDoS though.
How do they manage to get 'paywalled' content?
Maybe 'paywalled' is not the best word but using their Identity Based Crawling feature with Managed Browsers[1], you can use an existing account and scrape content that requires authentication. This may not sound like anything new but IMHO, crawl4ai's workflow is easy to follow.
[1] https://docs.crawl4ai.com/advanced/identity-based-crawling
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#136Earlier quoted context omitted.
Check out Anubis - it's not quite what you're suggesting but similar in concept: https://anubis.techaro.lol/
Maybe once you can use something more professional as an interstitial page
From the announcement page: https://xeiaso.net/blog/2025/anubis/
> RPM packages and unbranded (or customly branded) versions are available if you contact me and purchase commercial support. Otherwise your users have to see a happy anime girl every time they solve a challenge. This is a feature.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#137After I opened up https://sugaku.net to be usable without login, it was astounding how quickly the crawlers started. I'd like the site to be accessible to all, but I've had to restrict most of the dynamic features to logged in users, restrict robots.txt, use cloudflare to block AI crawlers and bad bots, and I'm still getting ~1M automated requests per day (compared to ~1K organic), so I think I'll need to restrict th…
One thing that worked well for me was layering obstacles It really sucks that this is the way things are, but what I did was 10 requests for pages in a minute, you get captchad (with a little apology and the option to bypass it by logging in). asset loads don’t count After a captcha pass, 100 requests in an hour gets you auth walled It’s really shitty but my industry is used to content scraping. This allows legit use…
I've wanted to but wasn't sure how to keep track of individuals. What works for you? IP Addresses, cookies, something else?
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#138After I opened up https://sugaku.net to be usable without login, it was astounding how quickly the crawlers started. I'd like the site to be accessible to all, but I've had to restrict most of the dynamic features to logged in users, restrict robots.txt, use cloudflare to block AI crawlers and bad bots, and I'm still getting ~1M automated requests per day (compared to ~1K organic), so I think I'll need to restrict th…
Has someone made honeypot for AI yet?
Take all regular papers and change their words or keywords to something outrageous and watch it feed it to users.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#139Can't you just rate limit beyond what a person would ever notice and do it by slowing the response?
Not if they hop to a different IP address every few requests. And they generally aren't bothered slow responses. It's not like they have to wait for one request to finish before they make another one (especially if they are making requests from thousands of machines).
You're saying that large companies are hitting individual websites with thousands of unrelated IP addresses?
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#140I got DoSed by ClaudeBot (Anthropic) just last week. Hitting a website I manage 700,000 times in one month and tripping our bandwidth limit with our hosting provider. What a PITA to have to investigate that, figure it out, block the user agent, and work with hosting provider support to get the limit lifted as a courtesy. Noticed that the ChatGPT bot was 2nd in traffic to this site, just not enough to cause trouble.
robots.txt did not work?
I have not monitored traffic in this way, but I imagine most AI companies would explicitly follow links listed in robots, even if not mentioned elsewhere on the site.