FOSS infrastructure is under attack by AI companies • Hacker News Board

Re: FOSS infrastructure is under attack by AI companies

#51

Post by blibble » Thu, Mar 20, 2025, 1:29 PM UTC

MartijnBraam wrote: post #29

megadata wrote: post #19

Perhaps time to start a central community ban pool for IP ranges?

Doesn't really work if crawlers are coming from the IP ranges of AWS and Azure etc...

just ban the lot

nothing good comes from there

unfortunately then they instantly switch to home IPs

Re: FOSS infrastructure is under attack by AI companies

#52

Post by immibis » Thu, Mar 20, 2025, 1:29 PM UTC

briandear wrote: post #32

nonrandomstring wrote: post #4

These are DDOS attacks and should be treated in law as such. (Although I do realise that in many countries now we no longer have any effective "rule of law")

You going to prosecute China?

Is that where Sam Altman is hiding?

Re: FOSS infrastructure is under attack by AI companies

#53

Post by Galanwe » Thu, Mar 20, 2025, 1:30 PM UTC

megadata wrote: post #19

Perhaps time to start a central community ban pool for IP ranges?

There's already loads of these. The problem is that most of these IPs are just cloud providers or DC ISPs.

Re: FOSS infrastructure is under attack by AI companies

#54

Post by rglullis » Thu, Mar 20, 2025, 1:30 PM UTC

The days of an open web are long gone. Every server will eventually have to require authentication for access, and to get an account you will have to provide some form of payment or social proof.

Honestly, I don't see it necessarily as a bad thing.

Re: FOSS infrastructure is under attack by AI companies

#55

Post by harhargange » Thu, Mar 20, 2025, 1:30 PM UTC

Can IPFS or torrent and large local databases decentralised by people be a solution to this? I personally have the resources to share and host TBs of data but didn't find a good use to it.

Re: FOSS infrastructure is under attack by AI companies

#56

Post by diggan » Thu, Mar 20, 2025, 1:31 PM UTC

> According to Drew, LLM crawlers don't respect robots.txt requirements and include expensive endpoints like git blame, every page of every git log, and every commit in your repository. They do so using random User-Agents from tens of thousands of IP addresses, each one making no more than one HTTP request, trying to blend in with user traffic.

How do they know that these are LLM crawlers and not anything else?

Re: FOSS infrastructure is under attack by AI companies

#57

Post by nzeid » Thu, Mar 20, 2025, 1:32 PM UTC

We need a project in the spirit of Spamhaus to actively maintain a list of perpetrating IPs. If they're cycling through IPs and IP blocks I don't know how sustainable a CAPTCHA-like solution is.

Re: FOSS infrastructure is under attack by AI companies

#58

Post by soco » Thu, Mar 20, 2025, 1:32 PM UTC

FeepingCreature wrote: post #41

To be clear, this is not an attack in the deliberate sense, and has nothing to do with AI except in that AI companies want to crawl the internet. This is more "FOSS sites damaged by extreme incompetence and unaccountability." The crawlers could just as well be search engine startups.

Correct, they could - but they are not. This is about the unaccountability, and if I was charitable (which I'm not) I'd add also incompetence, of the techbros leading the AI giants. Are we still expecting anything like "ethics"? I hope the few engineers reading HN will still have some, but the higher you go the foreign the concept gets.

Re: FOSS infrastructure is under attack by AI companies

#59

Post by blibble » Thu, Mar 20, 2025, 1:32 PM UTC

xena wrote: post #16

KolmogorovComp wrote: post #9

My question is, can we serve PoW challenges to these AI LLM scrapers that can be profitable?

That's what I've been doing! It works shockingly well. https://github.com/TecharoHQ/anubis

these companies have more compute than everyone else in the world put together

a proof of work function will end up selecting FOR them!

Re: FOSS infrastructure is under attack by AI companies

#60

Post by diggan » Thu, Mar 20, 2025, 1:32 PM UTC

QuadrupleA wrote: post #46

Isn't this just poor, sloppy crawler implementation? You shouldn't need to fetch a repo more than once to add it to a training set.

Sounds like it me. Why build a crawler that fetches one HTML page per commit in a repository instead of doing a bare-clone and then just get the data from there? Surely would contain even more data too, compared to the HTML page.