Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FOSS infrastructure is under attack by AI companies

thelibre.news

Post reply on HN
Print view View original

101–110 of 631 posts

Re: FOSS infrastructure is under attack by AI companies

#101

Post by jryan49 »

phkahler wrote: post #74

So I'll just float an idea again that always gets rejected here. This is yet another problem that could be solved completely by... Eliminating anonymity by default on the internet. To be clear, you could still have anonymous spaces like Reddit where arbitrary user IDs are used and real identities are discarded. People could opt-in to those spaces. But for most people most of the time, things get better when you can v…

I've been thinking about this even before AI. The internet has become society itself. Complete anonymity on the internet removes any sort of social pressure to act like a civilized person.

Re: FOSS infrastructure is under attack by AI companies

#102

Post by sevensor »

maaaaattttt wrote: post #89
UncleMeat wrote: post #30

Earlier quoted context omitted.

Yep. And it is much more far reaching than that. Look at the primary economic claim offered by AI companies: to end the need for a substantial portion of all jobs on the planet. The entire vision is to remake the entire world into one where the owners of these companies own everything and are completely unconstrained. All intellectual property belongs to them. All labor belongs to them. Why would they need good will…

I have this line of thought as well but then I wonder, if we are all out of jobs and out of substantial capital to spend, how do these owners make money ultimately? It's a genuine question and I'm probably missing something obvious. I can see a benevolant/post-scarcity spin to this but the non-benevolant one seems self defeating.

I don’t think you’re missing anything, I think the plan really is to burn it all down and rule over the ashes. The old saw “if you’re so smart, why aren’t you rich?” works in reverse too. This is a foolish, shortsighted thing to do, and they’re doing it anyway. Not really thinking about where value actually comes from or what their grandchildren’s lives would be like in such a world.

Re: FOSS infrastructure is under attack by AI companies

#103

Post by sir-alien »

lelanthran wrote: post #42

The big takeaway here is that Google's (and advertisement in general) dominance over the web is going away. This is because the only way to stop the bots is with a captcha, and this also stops search indexers from indexing your site. This will result in search engines not indexing sites, and hence providing no value anymore. There's probably going to be a small lag as the current knowledge in the current LLMs dry up…

Can we not just have a whitelist for allowed crawlers and ban the rest by default? Then places like DuckDuckGo and Google can provide a list of IP addresses that their crawlers will come from. Then simply just don't include major LLM providers like OpenAI

Re: FOSS infrastructure is under attack by AI companies

#105

Post by 01HNNWZ0MV43FF »

blibble wrote: post #59
xena wrote: post #16

Earlier quoted context omitted.

That's what I've been doing! It works shockingly well. https://github.com/TecharoHQ/anubis

these companies have more compute than everyone else in the world put together a proof of work function will end up selecting FOR them!

It'll still keep your site from getting hammered

Re: FOSS infrastructure is under attack by AI companies

#106

Post by Havoc »

QuadrupleA wrote: post #46

Isn't this just poor, sloppy crawler implementation? You shouldn't need to fetch a repo more than once to add it to a training set.

It’s people that don’t care if they ruin things for everyone else.

Crawlers have existed forever in search engine space and mostly behave.

This sort of no rate limit, fake user agent, 100s of IPs approach used by AI teams is obviously intentionally not caring who it fucks over. More malicious than sloppy implementation

Re: FOSS infrastructure is under attack by AI companies

#107

Post by xena »

throwaway173738 wrote: post #75

Does anyone maintain a list of ai products developed without scraping?

I don't think it's possible to develop a frontier model without mass scraping. The economics simply don't add up. You need at least 10 trillion tokens to make an 8 billion parameter model. 10 trillion tokens is something like 40 terabytes.

You simply can't get 40 terabytes of text without mass scraping.

Re: FOSS infrastructure is under attack by AI companies

#108

Post by nzeid »

voidUpdate wrote: post #82
nzeid wrote: post #57

We need a project in the spirit of Spamhaus to actively maintain a list of perpetrating IPs. If they're cycling through IPs and IP blocks I don't know how sustainable a CAPTCHA-like solution is.

This sounds backwards to me, if you maintain a list of IPs but they are constantly cycling them, it'll get out of date quickly, but a captcha-like system will (hopefully) always stop bot traffic

Only if your CAPTCHA-like is hurled at every client indiscriminately. Otherwise you'll end up right back where Spamhaus started: maintaining your own list of good and bad actors.

The advantage of a third party service is that you're sharing intel of bad actors.

Re: FOSS infrastructure is under attack by AI companies

#109

Post by InfamousRece »

ferguess_k wrote: post #67
ericholscher wrote: post #3

Yep -- our story here: https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse... (quoted in the OP) -- everyone I know has a similar story who is running large internet infrastructure -- this post does a great job of rounding a bunch of them up in 1 place. I called it when I wrote it, they are just burning their goodwill to the ground. I will note that one of the main startups in the space worked with us direct…

Maybe just feed them dynamically generated garbage information? More fun than no information.

It does not even have to be dynamically generated. Just pre-generate a few thousand static pages of AI slop and serve that. Probably cheaper than dynamic generation.

Re: FOSS infrastructure is under attack by AI companies

#110

Post by verytrivial »

The only thing that really annoys me about this is that the resources get scraped more than once. Sure, update a delta every month or whatever, but can't the AI bros get the sh*t together and share scrapes? It's embarrassing.

The special sauce is in parsing, tokenizing, enriching etc. There is no value in re-scraping, and massive cost, right?

Just do it.

Post reply on HN