281–290 of 631 posts
Yeah, I've had to mitigate this problem too, on a tiny forum serving a very niche audience. It was brought to its knees by a handful of LLM bots that completely ignored robots.txt. Thankfully, these bots were easy enough to block at the firewall level, but that may not work forever.
Were these bots also using random IPs and random User-Agents and if so, how do you know they are LLM agents?
We blocked all non EU ip ranges. Our customers are all from EU so no intrest in US or Asia. With the new US administration i assume nothing good from them, and Asia is a notorius bad player
Earlier quoted context omitted.
Money is only a bookkeeping tool for complex societies. The aim of the owner class in a worker-less world would be accumulation of important resources to improve their lives and to trade with other owners (money would likely still be used for bookkeeping here). A wealthy resource-owner might strive to maintain a large zone of land, defended by AI weaponry, that contains various industrial/agricultural facilities prod…
Better for them to give us jobs so we owe them and are less likely to revolt!
At this rate, it's more than FOSS infrastructure -- although that's a canary in the coalmine I especially sympathize with -- it's anonymous Internet access altogether. Because you can put your site behind an auth wall, but these new bots can solve the captchas and imitate real users like never before. Particularly if they're hitting you from residential IPs and with fake user agents like the ones in the article -- or…
We're half way there already. It always hits me whenever I am doing some mapping for OpenStreetMap and I'm looking up local businesses without their own internet presence. They use Facebook, Instagram, X, etc. for their digital calling card. I normally don't use Facebook (or Instagram, and gave up on X) and have no account there, and every time I follow one of those links, you get some info, and then you get a dialog…
I don't think these business owners really understand. Most normies just think everyone has a Facebook/Instagram account and can't even imagine a world where that is not the case.
I agree with you that it is extremely frustrating.
To be clear, this is not an attack in the deliberate sense, and has nothing to do with AI except in that AI companies want to crawl the internet. This is more "FOSS sites damaged by extreme incompetence and unaccountability." The crawlers could just as well be search engine startups.
Isn't this just poor, sloppy crawler implementation? You shouldn't need to fetch a repo more than once to add it to a training set.
It’s people that don’t care if they ruin things for everyone else. Crawlers have existed forever in search engine space and mostly behave. This sort of no rate limit, fake user agent, 100s of IPs approach used by AI teams is obviously intentionally not caring who it fucks over. More malicious than sloppy implementation
Earlier quoted context omitted.
My monetization strategy is unironically to offer a de-anime'd version under the name Techaro BotStopper or something.
It's a clever (and hilarious) strategy that will probably sell at least a few licenses. As an anime hater I'd be motivated by this.
Banning these crawlers is not the answer. Poisoning the well is.
It's also just wasting more of the planet's resources as compared to blocking
And more effort, with as only upside that it's not immediately obvious to the bot that it is being blocked so it'll suck in more of your pages
I understand that people are exploring options but I wouldn't label this as a solution to anything as of today's state of the art