nice, a free way to keep our IPFS pins alive
This is not how it works.
You have to actively pin data for it to be distributed.
Devs say AI crawlers dominate traffic, forcing blocks on entire countries
171–180 of 291 posts
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#172I got DoSed by ClaudeBot (Anthropic) just last week. Hitting a website I manage 700,000 times in one month and tripping our bandwidth limit with our hosting provider. What a PITA to have to investigate that, figure it out, block the user agent, and work with hosting provider support to get the limit lifted as a courtesy. Noticed that the ChatGPT bot was 2nd in traffic to this site, just not enough to cause trouble.
At which level of DDos one can claim damages from them?
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#173I think the proper answer is to aim for the bots to get _negative_ utility value from visiting our sites, that is poisoning their well, not just zero value, that is to block them. Did you try to GET a canary page forbidden in robots.txt? Very well, have a bucket load of articles on the benefits of drinking bleach. Is your user-agent too suspicious? No problem, feel free to scrape my insecure code (google "emergent mi…
Also, lower upload rate to 5kb/s
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#174Earlier quoted context omitted.
Maybe once you can use something more professional as an interstitial page
From the announcement page: https://xeiaso.net/blog/2025/anubis/ > RPM packages and unbranded (or customly branded) versions are available if you contact me and purchase commercial support. Otherwise your users have to see a happy anime girl every time they solve a challenge. This is a feature.
I'm going to be making distro packages and binaries public. I will have to figure out white label monetization I guess.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#175After I opened up https://sugaku.net to be usable without login, it was astounding how quickly the crawlers started. I'd like the site to be accessible to all, but I've had to restrict most of the dynamic features to logged in users, restrict robots.txt, use cloudflare to block AI crawlers and bad bots, and I'm still getting ~1M automated requests per day (compared to ~1K organic), so I think I'll need to restrict th…
Has someone made honeypot for AI yet? Take all regular papers and change their words or keywords to something outrageous and watch it feed it to users.
This kinda fits, though it's on a personal blog level:
https://www.brainonfire.net/blog/2024/09/19/poisoning-ai-scr...
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#176How do they know which square contains a bicycle?
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#177Yeah my small site got 170k requests from one bot in a few mins. Of course it was rate limited, but didn't seem to know 429 or 444 (drop connection) so kept on coming back for more.
I do have ipset drop too but at the moment takes a person to enable it... just so much effort to stop these **s. Exhausting!
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#178Isn't this by problem solved by using commoncrawl data. I wonder what changed to AI companies to do mass crawling individually.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#179This is going to start happening to brick and mortar businesses through their customer support channels
I’d love to hear more. What are they seeing?
Unfounded. Assuming automated/voice-enabled assistants will spam for opening times, price & description of products/services, scheduling/booking, etc.
In the long run it'll be an arms race but the transition will be rough for businesses as consumers can adopt these tools faster than SMBs or enterprises can integrate them.
Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries
#180I recently started a side-project with a "code everything in prod" approach for fun. I've done this many times over the past 20 years and the bot traffic is usually harmless, but this has been different. I haven't advertised the hostname anywhere, and in less than 24 hours I had a bunch of spam form submissions. I've always expected this after minor publicity, but not "start server, instantly get raided by bots perfo…
This is yet another great example of the innovation that the AI industry is delivering. Why just limit your scraper bots to GET requests when there might be some juicy data to train on hidden behind that form? There's a reason why the a16z funded cracked vibe coder ninjas are taking over software engineering, they're full of wonderful ideas like this.
I work with one of those guys. Morally bankrupt at every level of his existence.