FOSS infrastructure is under attack by AI companies • Hacker News Board

Re: FOSS infrastructure is under attack by AI companies

#401

Post by structural » Thu, Mar 20, 2025, 3:52 PM UTC

aspir wrote: post #323

Aachen wrote: post #275

Earlier quoted context omitted.

It's not like they say, but it's at least three different implementations and I don't think any were cloudflare because I've been running into those pages for years and they've got captchas (functional or not). One of them was Akamai I think indeed

Yeah, I definitely don't want to pivot this thread into a product pitch, as the important thing is helping the open-source projects, but we can work with the maintainers to tune the systems to be as strict/lax as preferred. I'm sure the other services can too, to be fair.

The underlying issue is that many sites aren't going to get feedback from the real people they've blocked, so their operators won't actually know that tuning is required (also, the more strict the system, the higher percentage of requests will be marked as bots, which might lead an operator to want things to be even more strict...)

Re: FOSS infrastructure is under attack by AI companies

#402

Post by diggan » Thu, Mar 20, 2025, 3:52 PM UTC

myaccountonhn wrote: post #397

diggan wrote: post #231

Earlier quoted context omitted.

Are you saying that Gnome shouldn't offer access to their VCS for free, and all Gnome developers should pay a small sum to be able to access it? FOSS is generally built on the idea that anyone can use the code for anything, if you start to add a price for that, not only do you effectively gate your project from "poor people", but it also kind of erodes some of the core principles behind FOSS.

There is nothing that says you can't charge money for FOSS software. FOSS is more about having the ability to inspect and freely change your software to your use-cases.

> There is nothing that says you can't charge money for FOSS software

Well, yes and no. If you had a cost to access the source code, I'm pretty sure I'd stop calling that FOSS. If you only have a price for downloading binaries, sure, still FOSS, since we're talking source code licensing.

> Nothing should be $$ free

I took this statement at face value, and assumed parent argued for basically eliminating FOSS.

Re: FOSS infrastructure is under attack by AI companies

#403

Post by inetknght » Thu, Mar 20, 2025, 3:53 PM UTC

Suppafly wrote: post #394

madeofpalk wrote: post #388

Earlier quoted context omitted.

I don't have a Facebook or Instagram account, but I definitely eat tacos and I was put off when I couldn't see a new taco place's opening hours without an instagram accoutn. I'm not sure why you think why people who don't have a Facebook account wouldn't eat at restaurants

You're in a tiny enough minority that it doesn't matter to them. It's like Amish complaining that they can't use a drive-thru window or something. Except it'd take you 30 seconds, one time, to solve your problem forever.

> Except it'd take you 30 seconds, one time, to solve your problem forever.

And those 30 seconds are a harrowing pit of despair out of which comes the rest of your life filled with advertisements, tracking, second-guessing, and accusations of being a hypocrite.

Re: FOSS infrastructure is under attack by AI companies

#404

Post by orthecreedence » Thu, Mar 20, 2025, 3:57 PM UTC

lelanthran wrote: post #42

The big takeaway here is that Google's (and advertisement in general) dominance over the web is going away. This is because the only way to stop the bots is with a captcha, and this also stops search indexers from indexing your site. This will result in search engines not indexing sites, and hence providing no value anymore. There's probably going to be a small lag as the current knowledge in the current LLMs dry up…

This is fine, as Google's utility as a search engine has turned into a hot pile of garbage, at least for my cases. Where a decade ago I could put in a few keywords and get relevant results, I now have to guide it with several "quoted phrases" and -exclusions to get the result I'm looking for on the second or third result page. It has crumbled under its own weight, and seems to suggest irrelevant trash to me first and foremost because it's the website of some big player or content farm. Either their algorithm is tuned for mass manipulation or they lost the arms race with SEO cretins (or both).

Granted, I'm not looking forward to some LLM condensing all the garbage and handing me a Definitive Answer (TM) based on the information it deems relevant for inclusion.

Re: FOSS infrastructure is under attack by AI companies

#405

Post by ferguess_k » Thu, Mar 20, 2025, 3:59 PM UTC

diggan wrote: post #369

ferguess_k wrote: post #363

Earlier quoted context omitted.

OK I'm dumb.

No, just overly capitalistic, but that's probably more because of your environment than anything, and very possible to change if you want :)

Actually I'm anti-capitalistic. That solution was supposed to undermine corporations who take other people's work for free. Maybe it shouldn't cover the whole FOSS, but I do think it fits OP's use case.

Re: FOSS infrastructure is under attack by AI companies

#406

Post by shadowfacts » Thu, Mar 20, 2025, 3:59 PM UTC

diggan wrote: post #56

> According to Drew, LLM crawlers don't respect robots.txt requirements and include expensive endpoints like git blame, every page of every git log, and every commit in your repository. They do so using random User-Agents from tens of thousands of IP addresses, each one making no more than one HTTP request, trying to blend in with user traffic. How do they know that these are LLM crawlers and not anything else?

In my case, no small fraction of the traffic was from OpenAI and Anthropic. There were also other user agents that literally said "AI".

Re: FOSS infrastructure is under attack by AI companies

#407

Post by wadadadad » Thu, Mar 20, 2025, 4:04 PM UTC

mistrial9 wrote: post #287

Havoc wrote: post #106

Earlier quoted context omitted.

It’s people that don’t care if they ruin things for everyone else. Crawlers have existed forever in search engine space and mostly behave. This sort of no rate limit, fake user agent, 100s of IPs approach used by AI teams is obviously intentionally not caring who it fucks over. More malicious than sloppy implementation

it is an ecosystem of social roles, not just "people" .. casting the decision into individual choices is not the right filter to understand this situation..

I'm not sure I'm following what you mean by 'social roles'. Which roles are you referring to here?

I'll disagree that it's not at least individual malicious choice, though. Someone decided that they needed to fake/change user agents (as one example), and implemented it. Most likely it was more than one person- some manager(s)/teams probably also either suggested or agreed to this choice.

I would like to think at some point in this decision making process, someone would have considered 'is it ethical to change user agents to get around bans? Is it ethical to ignore robots.txt?' and decided not to proceed, but apparently that's not happening here...

Re: FOSS infrastructure is under attack by AI companies

#408

Post by ricudis » Thu, Mar 20, 2025, 4:06 PM UTC

UncleMeat wrote: post #335

dbingham wrote: post #296

Earlier quoted context omitted.

That value is only great if it's shared equitably with the rest of the planet. If it's owned by a few, as it is right now, it's an existential threat to the life, liberty, and pursuit of a happiness of everyone else on the planet. We should be seriously considering what we're going to do in response to that threat if something doesn't change soon.

Yep. The "wouldn't it be great if we had robots do all the labor you are currently doing" argument only works if there is some plan to make sure that my rent gets paid other than me performing labor.

And that's the ultimate fail of capitalist ethics - the notion that we must all work just so we can survive. Look at how many shitty and utterly useless jobs exist just so people can be employed on them to survive.

This has to change somehow.

"Machines will do everything and we'll just reap the profits" is a vision that techno-millenialists are repeating since the beginnings of the Industrial Revolution, but we haven't seen that happening anywhere.

For some strange reason, technological progress seem to be always accompanied with an increase on human labor. We're already past the 8-hours 5-days norm and things are only getting worse.

Re: FOSS infrastructure is under attack by AI companies

#409

Post by kijin » Thu, Mar 20, 2025, 4:08 PM UTC

cgh wrote: post #242

kijin wrote: post #183

Earlier quoted context omitted.

Exactly. They're renting infrastructure on well-known clouds, not cycling through consumer IPs like yesterday's botnets. Block all web traffic from well-known cloud IPs, and you can keep 99% of the LLM bots away. Alibaba seems to be the most common source of bot traffic on my infrastructure lately, and I also see Huawei Cloud from time to time. Not much AWS, probably because of their high IPv4 pricing. You can allow…

From the article: “…they do so using random User-Agents that overlap with end-users and come from tens of thousands of IP addresses - mostly residential, in unrelated subnets, each one making no more than one HTTP request over any time period we tried to measure - actively and maliciously adapting and blending in with end-user traffic and avoiding attempts to characterize their behavior or block their traffic.” So it…

I saw a fair amount of that kind of behavior, too, mostly around the summer of last year. At some point it dropped off sharply. Over the last few months, at least for the servers I keep an eye on, most of the trouble has been from Chinese cloud IPs.

Either the LLM devs got more funding, or maybe the authorities took down the botnet they were using.

Re: FOSS infrastructure is under attack by AI companies

#410

Post by kod » Thu, Mar 20, 2025, 4:08 PM UTC

eadmund wrote: post #367

rectang wrote: post #356

Earlier quoted context omitted.

If you as an individual recognizably regurgitate the essay you read, then you have infringed. If an AI model recongnizably regurgitates the essay it trained on then it has infringed. The AI argument that passing original content through an algorithm insulates the output from claims of infringement because of "fair use" is pigwash.

> If an AI model recongnizably regurgitates the essay it trained on then it has infringed. I completely agree — that’s why I explicitly wrote ‘non-copy paintings’ in my example. > The AI argument that passing original content through an algorithm insulates the output from claims of infringement because of "fair use" is pigwash. Sure, but the argument that training an AI on content is necessarily infringement is equal…

> So long as the resulting model does not contain copies, it is not infringement

That's not true.

The article specifically deals with training by scraping sites. That does necessarily involve producing a copy from the server to the machine(s) doing the scraping & training. If the TOS of the site incorporates robots.txt or otherwise denies a license for such activity, it is arguably infringement. Sourcehut's TOS for example specifically denies the use of automated tools to obtain information for profit.