FOSS infrastructure is under attack by AI companies • Hacker News Board

Re: FOSS infrastructure is under attack by AI companies

#301

Post by freeone3000 » Thu, Mar 20, 2025, 2:52 PM UTC

WhyOhWhyQ wrote: post #277

rectang wrote: post #252

Earlier quoted context omitted.

> Look at the primary economic claim offered by AI companies: to end the need for a substantial portion of all jobs on the planet. And this is why AI training is not "fair use". The AI companies seek to train models in order to compete with the authors of the content used to train the models. A possible eventual downfall of AI is that the risk of losing a copyright infringement lawsuit is not going away. If a court d…

I've pointed this out to a few people in this space. They tend to suggest that the value in AI is so great this means we should get rid of copyright law entirely.

…meaning that whatever model results would have no protection, and would be free for anyone to use?

Re: FOSS infrastructure is under attack by AI companies

#302

Post by szczepano » Thu, Mar 20, 2025, 2:54 PM UTC

tecleandor wrote: post #195

szczepano wrote: post #125

there is simple solution for gentlemens: robots.txt should allow to exclude all AI crawlers and AI crawlers should be forced to add "AI" to their crawl user agent headers and also respect robots.txt saying they can't crawl this website right now we need to do this: User-agent: * Disallow: /

Nice. How do we force them to respect robots.txt?

they respect robots.txt at least major ones like meta, claude, google, openai, based on my infra observations robots.txt is enough in 90%, 10% is just banning ip ranges for couple of days but those are no AI companies

Re: FOSS infrastructure is under attack by AI companies

#303

Post by gyesxnuibh » Thu, Mar 20, 2025, 2:54 PM UTC

Eridrus wrote: post #131

I know a lot of FOSS people are hostile to AI in general, and this is an immediate problem, but I feel like a better solution for everyone would be for there to be some sort of central repo of this information that AI companies can pull from without externalizing their costs like this.

Why buy the cow when you get the milk for free tho?

Re: FOSS infrastructure is under attack by AI companies

#304

Post by chneu » Thu, Mar 20, 2025, 2:55 PM UTC

Cthulhu_ wrote: post #249

Back when search engines caused this, the industry made an agreement and designed the robots.txt spec in order to avoid legal frameworks being made to stop them. Because of that, legal frameworks weren't being made. Now there's a new generation of hungry hungry hippo indexers that didn't agree to that and who feel intense pressure from competition to scoop up as much data as they can, who just ignore it. Legislation…

I'd be shocked if a single member of the US House or Congress know what robots.txt is.

Re: FOSS infrastructure is under attack by AI companies

#305

Post by xeeeeeeeeeeenu » Thu, Mar 20, 2025, 2:55 PM UTC

Aachen wrote: post #289

try_the_bass wrote: post #95

Banning these crawlers is not the answer. Poisoning the well is.

There's one thing worse than a block page being shown to humans because an algorithm decided they're a bot: purposefully false information being shown to humans It's also just wasting more of the planet's resources as compared to blocking And more effort, with as only upside that it's not immediately obvious to the bot that it is being blocked so it'll suck in more of your pages I understand that people are exploring…

There are bot identification methods that may produce lots of false negatives, but zero false positives.

Simple example: no legitimate user has "GPTBot" in their user agent string.

Re: FOSS infrastructure is under attack by AI companies

#306

Post by dougb5 » Thu, Mar 20, 2025, 2:55 PM UTC

cgh wrote: post #242

kijin wrote: post #183

Earlier quoted context omitted.

Exactly. They're renting infrastructure on well-known clouds, not cycling through consumer IPs like yesterday's botnets. Block all web traffic from well-known cloud IPs, and you can keep 99% of the LLM bots away. Alibaba seems to be the most common source of bot traffic on my infrastructure lately, and I also see Huawei Cloud from time to time. Not much AWS, probably because of their high IPv4 pricing. You can allow…

From the article: “…they do so using random User-Agents that overlap with end-users and come from tens of thousands of IP addresses - mostly residential, in unrelated subnets, each one making no more than one HTTP request over any time period we tried to measure - actively and maliciously adapting and blending in with end-user traffic and avoiding attempts to characterize their behavior or block their traffic.” So it…

This matches exactly with what I'm seeing on my own sites too and it's from all over the world, not just China.

(I described my bot woes a few weeks ago at https://news.ycombinator.com/item?id=43208623. The "just block bots!" replies were well-intentioned but naive -- I've still found no signal that works reliably well to distinguish bots from real traffic.)

Re: FOSS infrastructure is under attack by AI companies

#307

Post by rectang » Thu, Mar 20, 2025, 2:55 PM UTC

WhyOhWhyQ wrote: post #277

rectang wrote: post #252

Earlier quoted context omitted.

> Look at the primary economic claim offered by AI companies: to end the need for a substantial portion of all jobs on the planet. And this is why AI training is not "fair use". The AI companies seek to train models in order to compete with the authors of the content used to train the models. A possible eventual downfall of AI is that the risk of losing a copyright infringement lawsuit is not going away. If a court d…

I've pointed this out to a few people in this space. They tend to suggest that the value in AI is so great this means we should get rid of copyright law entirely.

AI is worthless without training data. If all content becomes AI generated because AI outcompetes original content then there will be no data left to train on.

When Google first came out in 1998, it was amazing, spooky how good it was. Then people figured out how to game pagerank and Google's accuracy cratered.

AI is now in a similar bubble period. Throwing out all of copyright law just for the benefit of a few oligarchs would be utter foolishness. Given who is in power right now I'm sure that prospect will find a few friends, but I think the odds of it actually happening before the bubble bursts are pretty small.

Re: FOSS infrastructure is under attack by AI companies

#308

Post by bitmasher9 » Thu, Mar 20, 2025, 2:56 PM UTC

Cthulhu_ wrote: post #249

Back when search engines caused this, the industry made an agreement and designed the robots.txt spec in order to avoid legal frameworks being made to stop them. Because of that, legal frameworks weren't being made. Now there's a new generation of hungry hungry hippo indexers that didn't agree to that and who feel intense pressure from competition to scoop up as much data as they can, who just ignore it. Legislation…

I’m not sure that I like this plan. We shouldn’t let the illegal AIs gain more knowledge and usefulness than the legal ones.

There’s other options besides a blanket ban.

Re: FOSS infrastructure is under attack by AI companies

#309

Post by chneu » Thu, Mar 20, 2025, 2:57 PM UTC

JimDabell wrote: post #151

diggan wrote: post #56

> According to Drew, LLM crawlers don't respect robots.txt requirements and include expensive endpoints like git blame, every page of every git log, and every commit in your repository. They do so using random User-Agents from tens of thousands of IP addresses, each one making no more than one HTTP request, trying to blend in with user traffic. How do they know that these are LLM crawlers and not anything else?

That was my reaction. It seems like the article is saying two mutually exclusive things: - We cannot block them because we can’t differentiate legitimate traffic from illegitimate traffic… - …but we can conclusively identify this traffic as coming from AI crawlers.

It's the behavior of the traffic in hindsight that's obvious. It's difficult to identify in the moment. This is by design.

Getting caught isn't a big deal. Getting caught in the act is. As long as they get their data, it doesn't matter if they're caught afterwards.

Re: FOSS infrastructure is under attack by AI companies

#310

Post by GTP » Thu, Mar 20, 2025, 2:57 PM UTC

harhargange wrote: post #55

Can IPFS or torrent and large local databases decentralised by people be a solution to this? I personally have the resources to share and host TBs of data but didn't find a good use to it.

I doubt, as the article mentions scraping the same resource after just 6 hours. AI companies want to make sure they have fresh data, whileit would be hard to keep such a database updated.