FOSS infrastructure is under attack by AI companies • Hacker News Board

Re: FOSS infrastructure is under attack by AI companies

#111

Post by WesolyKubeczek » Thu, Mar 20, 2025, 1:47 PM UTC

ebiester wrote: post #44

WesolyKubeczek wrote: post #5

...at some point, some people started appreciating mailing lists and the distributed nature of Git again.

Yes and no. The distributed nature of git is fine until you want to serve it to the world - then, you're back to bad actors. They're looking for commits because it's nicely chunked, I'm taking a guess.

The distributed nature of git lets me be independent of some central instance (you may decide that the master copy resides on Github, but with the advent of mesh VPNs like the ones Zerotier and Tailscale offer, you could also sidestep it and push/pull from your colleagues directly as well). It also lets me dictate who gets to access it.

What the article describes, though, is possibly the worst way a machine can access a git repository, which is using a web UI and scraping that, instead of cloning it and adding all the commits to its training set. I feel like they simply don't give a shit. They got such a huge capital injection that they feel they can afford to not give a shit about their own cost efficiency and that they go using the scorched earth tactics. After all, even their own LLMs can produce a naive scraper that wreaks havoc on the internet infrastructure, and they just let it loose. Got mine, fuck you all the way!

But then they will release some DeepSeek R(xyz), and yay, all the hackernews who were roasting them for such methods, will be applauding them for a new version of an "open source" stochastic parrot. Yay indeed.

Re: FOSS infrastructure is under attack by AI companies

#112

Post by gnz11 » Thu, Mar 20, 2025, 1:47 PM UTC

ferguess_k wrote: post #67

ericholscher wrote: post #3

Yep -- our story here: https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse... (quoted in the OP) -- everyone I know has a similar story who is running large internet infrastructure -- this post does a great job of rounding a bunch of them up in 1 place. I called it when I wrote it, they are just burning their goodwill to the ground. I will note that one of the main startups in the space worked with us direct…

Maybe just feed them dynamically generated garbage information? More fun than no information.

OP’s linked blog post mentioned they got hit with a large spike in bandwidth charges. Sending them garbage information costs money.

Re: FOSS infrastructure is under attack by AI companies

#113

Post by adamc » Thu, Mar 20, 2025, 1:47 PM UTC

phkahler wrote: post #74

So I'll just float an idea again that always gets rejected here. This is yet another problem that could be solved completely by... Eliminating anonymity by default on the internet. To be clear, you could still have anonymous spaces like Reddit where arbitrary user IDs are used and real identities are discarded. People could opt-in to those spaces. But for most people most of the time, things get better when you can v…

In the USA, where an increasingly authoritarian government has come into power... I think many would stop using the internet if that happened.

Re: FOSS infrastructure is under attack by AI companies

#114

Post by imtringued » Thu, Mar 20, 2025, 1:47 PM UTC

maaaaattttt wrote: post #89

UncleMeat wrote: post #30

Earlier quoted context omitted.

Yep. And it is much more far reaching than that. Look at the primary economic claim offered by AI companies: to end the need for a substantial portion of all jobs on the planet. The entire vision is to remake the entire world into one where the owners of these companies own everything and are completely unconstrained. All intellectual property belongs to them. All labor belongs to them. Why would they need good will…

I have this line of thought as well but then I wonder, if we are all out of jobs and out of substantial capital to spend, how do these owners make money ultimately? It's a genuine question and I'm probably missing something obvious. I can see a benevolant/post-scarcity spin to this but the non-benevolant one seems self defeating.

You don't. It's like Minecraft. You can do almost everything in Minecraft alone and everything exists in infinite quantity, so why trade in the first place?

This goes both ways. Let's say there is something you want but you're having trouble obtaining it. You'd need to give something in exchange.

But the seller of what you want doesn't need the things you can easily acquire, because they can get those things just as easily themselves.

The economy collapses back into self sufficiency. That's why most Minecraft economy servers start stagnating and die.

Re: FOSS infrastructure is under attack by AI companies

#115

Post by jrh3 » Thu, Mar 20, 2025, 1:47 PM UTC

This is why we can't have nice things. Restricting access or making it a little more difficult preserves goodness for real people.

Re: FOSS infrastructure is under attack by AI companies

#116

Post by cudgy » Thu, Mar 20, 2025, 1:48 PM UTC

ziddoap wrote: post #68

piokoch wrote: post #63

Copyright the content and sue those who use it for AI training. I believe there is a lot of low hanging fruit for lawyers here. I would be surprised if they weren't preparing to hit Open AI and alikes. Very badly. Google get away with deep linking issues as publishers after all had some interest in being linked from the search engine, here publishers see zero value.

> Copyright the content and sue those who use it for AI training. This costs money, time, and ongoing commitment. FOSS isn't typically known for being overflowing with cash.

Good thing law firms are overflowing with the desire to make large amounts of cash from contingency based cases.

Re: FOSS infrastructure is under attack by AI companies

#117

Post by krapp » Thu, Mar 20, 2025, 1:48 PM UTC

WesolyKubeczek wrote: post #78

anthk wrote: post #11

Earlier quoted context omitted.

And Usenet, and IRC with a registered user prereq to join. Also, set AI tarpits as fake links with recursive calls. Make then mad with non-curated bullshit made from Markov chain generators until their cache begins to rot forever.

Usenet, as far as I remember, used to be a fucking hell to maintain right. With each server having to basically mirror everything, it was a hog on bandwidth and storage, and most server software at its heyday was a hog on filesystems of its day (you had to make sure you have plenty of inodes to spare). The other day, I logged into Usenet using eternalseptember, and found out that it consisted in 95% of zombies sendin…

Culturally, the modern replacement for Usenet is probably Reddit. Architecturally, probably something built on top of a federated protocol like ActivityPub (Mastodon) or Nostr (Lemmy).

But I guess realistically you can't fight entropy forever. Even Hacker News, aggressively moderated as it is, is slowly but irrevocably degrading over time.

Re: FOSS infrastructure is under attack by AI companies

#118

Post by nonrandomstring » Thu, Mar 20, 2025, 1:48 PM UTC

rvnx wrote: post #13

nonrandomstring wrote: post #4

These are DDOS attacks and should be treated in law as such. (Although I do realise that in many countries now we no longer have any effective "rule of law")

In the world, the richest wins, not the nicest (cf. Sam Altman)

lol. Tell that to Nicolae and Elena Ceaucescu, Saddam Hussain, Muammar Gaddafi, and all those other tinpot nobjockeys who thought their money and influence would save them from "nice" people.

Re: FOSS infrastructure is under attack by AI companies

#119

Post by kittikitti » Thu, Mar 20, 2025, 1:48 PM UTC

rglullis wrote: post #54

The days of an open web are long gone. Every server will eventually have to require authentication for access, and to get an account you will have to provide some form of payment or social proof. Honestly, I don't see it necessarily as a bad thing.

"some form of social proof."

Sounds like sanctioned racism.

Re: FOSS infrastructure is under attack by AI companies

#120

Post by mrweasel » Thu, Mar 20, 2025, 1:48 PM UTC

QuadrupleA wrote: post #46

Isn't this just poor, sloppy crawler implementation? You shouldn't need to fetch a repo more than once to add it to a training set.

From reading Drew Devaults angry post from earlier this week, my take is that not only is it poorly implemented crawlers, it's also that it's cheaper to scrape, rather than keep copies on hand. Effectively these companies are outsourcing the storage of "their" training data to everyone on the internet.

Ideally a site would get scraped once, and then the scraper would check if content has changed, e.g. etag, while also learning how frequently content changes. So rather than just hammer some poor personal git repo over and over, it would learn that Monday is a good time to check if something changed and then back off for a week.