To be clear, this is not an attack in the deliberate sense, and has nothing to do with AI except in that AI companies want to crawl the internet. This is more "FOSS sites damaged by extreme incompetence and unaccountability." The crawlers could just as well be search engine startups.
FOSS infrastructure is under attack by AI companies
41–50 of 631 posts
Re: FOSS infrastructure is under attack by AI companies
#42The big takeaway here is that Google's (and advertisement in general) dominance over the web is going away.
This is because the only way to stop the bots is with a captcha, and this also stops search indexers from indexing your site. This will result in search engines not indexing sites, and hence providing no value anymore.
There's probably going to be a small lag as the current knowledge in the current LLMs dry up because no one can scrape the web in an automated fashion anymore.
It'll all burn down.
Re: FOSS infrastructure is under attack by AI companies
#43This is probably a dumb question, but have they tried sending abuse reports to hosting providers? Or even lawsuits? Most hosting providers take it seriously when their client is sending a DoS attack, because if they don't, they can get kicked off the internet by their provider.
Re: FOSS infrastructure is under attack by AI companies
#44...at some point, some people started appreciating mailing lists and the distributed nature of Git again.
Yes and no.
The distributed nature of git is fine until you want to serve it to the world - then, you're back to bad actors. They're looking for commits because it's nicely chunked, I'm taking a guess.
Re: FOSS infrastructure is under attack by AI companies
#45Re: FOSS infrastructure is under attack by AI companies
#46Isn't this just poor, sloppy crawler implementation? You shouldn't need to fetch a repo more than once to add it to a training set.
Re: FOSS infrastructure is under attack by AI companies
#47We're close to finding a clear use-case for Bitcoin with this one.
Interesting. Basically moving the proof-of-work off the user's phone and to a dedicated mine. Websites could just have a lightning wallet or something and auto-charge the user 1e-7 bitcoin to access the page.
Re: FOSS infrastructure is under attack by AI companies
#48[deleted]
Re: FOSS infrastructure is under attack by AI companies
#49Earlier quoted context omitted.
That's what I've been doing! It works shockingly well. https://github.com/TecharoHQ/anubis
Unless I am missing something, the result of that generated work has no monetary value though.
The monetary value is not having a misbehaving AI bot download 73TB or whatever of your data.
Re: FOSS infrastructure is under attack by AI companies
#50Perhaps time to start a central community ban pool for IP ranges?
Doesn't really work if crawlers are coming from the IP ranges of AWS and Azure etc...
Traditionally, holders of IP ranges that attack the internet at large get kicked off the internet by having those ranges blacklisted everywhere. This can also get them in serious trouble with the places they got their IP ranges (I assume AWS has them directly from ARIN, so maybe not) and their upstream bandwidth providers and so on, as well as making them less attractive hosts because they are blocked everywhere.