Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FOSS infrastructure is under attack by AI companies

thelibre.news

Post reply on HN
Print view View original

521–530 of 631 posts

Re: FOSS infrastructure is under attack by AI companies

#521

Post by xena »

biophysboy wrote: post #513
xena wrote: post #77

Earlier quoted context omitted.

AI expert here. It's probably for collecting training data and the crawlers are probably very unsupervised. I'd guess that they're literally the most simplistic crawler code you can imagine combined with parallelism across machines. The good news is that it's easy to disrupt these crawlers with some easy hacks. Tactical reverse slowloris is probably gonna make a comeback.

If its for training data, why are they straining FOSS so much? Is there thousands of actors repeatedly making training data all the time? I thought it was a sort of one-off thing w/ the big tech players.

Git forges are some of the worst case for this. The scrapers click on every link on every page. If you do this to a git forge, it gets very O(scary) very fast because you have to look at data that is not frequently looked at and will NOT be cached. Most of how git forges are fast is through caching.

The thing about AI scrapers is that they don't just do this once. They do this every day in case every file in a glibc commit from 15 years ago changed. It's absolutely maddening and I don't know why AI companies do this, but if this is not handled then the git forge falls over and nobody can use it.

Anubis is a solution that should not have to exist, but the problem it solves is catastrophically bad, so it needs to exist.

Re: FOSS infrastructure is under attack by AI companies

#522

Post by Aachen »

xeeeeeeeeeeenu wrote: post #305
Aachen wrote: post #289

Earlier quoted context omitted.

There's one thing worse than a block page being shown to humans because an algorithm decided they're a bot: purposefully false information being shown to humans It's also just wasting more of the planet's resources as compared to blocking And more effort, with as only upside that it's not immediately obvious to the bot that it is being blocked so it'll suck in more of your pages I understand that people are exploring…

There are bot identification methods that may produce lots of false negatives, but zero false positives. Simple example: no legitimate user has "GPTBot" in their user agent string.

Ah yeah, that's a good example

It'll still trickle down to people using the system and waste people's time (from development to the people working to produce all the things used by these companies and mopping up the impact this has to eventually the users) but that would definitely resolve one of my concerns

Re: FOSS infrastructure is under attack by AI companies

#524

Post by Miraste »

bigfudge wrote: post #508
lukevp wrote: post #155

Earlier quoted context omitted.

Capitalism is an unthinking, unfeeling force. The writing is on the wall that AI is coming, and being altruistic about it doesn’t do jack to keep others from the land grab. Their thinking is, might as well join the rush and hope they’re one of the winners. Every one of us sitting on the sidelines will be impacted in some way or the other. So who’re the smart ones, the ones who grab shovels and start digging, or the o…

This is a bleak view. How about the ones who work hard to shape the way we adopt the technology to societal benefit ?

There are a few people like that, but not many. And certainly none in the AI space.

Re: FOSS infrastructure is under attack by AI companies

#525

Post by moooo99 »

bigfudge wrote: post #508
lukevp wrote: post #155

Earlier quoted context omitted.

Capitalism is an unthinking, unfeeling force. The writing is on the wall that AI is coming, and being altruistic about it doesn’t do jack to keep others from the land grab. Their thinking is, might as well join the rush and hope they’re one of the winners. Every one of us sitting on the sidelines will be impacted in some way or the other. So who’re the smart ones, the ones who grab shovels and start digging, or the o…

This is a bleak view. How about the ones who work hard to shape the way we adopt the technology to societal benefit ?

> How about the ones who work hard to shape the way we adopt the technology to societal benefit ?

they are heavily outnumbered and "outfunded"

Re: FOSS infrastructure is under attack by AI companies

#526

Post by prmoustache »

sir-alien wrote: post #103
lelanthran wrote: post #42

The big takeaway here is that Google's (and advertisement in general) dominance over the web is going away. This is because the only way to stop the bots is with a captcha, and this also stops search indexers from indexing your site. This will result in search engines not indexing sites, and hence providing no value anymore. There's probably going to be a small lag as the current knowledge in the current LLMs dry up…

Can we not just have a whitelist for allowed crawlers and ban the rest by default? Then places like DuckDuckGo and Google can provide a list of IP addresses that their crawlers will come from. Then simply just don't include major LLM providers like OpenAI

I am pretty sure a number of crawlers are running inside mobile apps of mobile phone users so they can get residential ip pools.

Re: FOSS infrastructure is under attack by AI companies

#527

Post by moooo99 »

Ray20 wrote: post #432
sozforex wrote: post #259

Earlier quoted context omitted.

At some point in the future, if you aren't AI, you won't be able to compete in the job market.

That's a laughable idea. Job market is formed by the presence of needs and the presence of the ability to satisfy them. AI - does not reduce the ability to satisfy needs, so only possible situation where you won't be able to compete - is either the socialists will seize power and ban competition, or all the needs will be met in some other ways. In any other situation - there will be job market and the people will com…

> there will be job market and the people will compete in it

maybe there will be. I'm sure there also is a market for Walkman somewhere, its just exceedingly small.

The proclaimed goal is to displace workers on a grand scale. This is basically the vision of any AI company and literally the only way you could even remotely justify their valuations given the heavy losses they incur right now.

> Job market is formed by the presence of needs and the presence of the ability to satisfy them

The needs of a job market are largely shaped by the overall economy. Many industrial nations are largely service based economies with a lot of white collar jobs in particular. These white collar jobs are generally easier to replace with AI than blue collar jobs because you don't have to deal with pesky things like the real, physical world. The problem is: if white collar workers are kicked out of their jobs en masse, it also negatively affects the "value" of the remaining people with employment (exhibit A: tech job marker right now).

> is either the socialists will seize power and ban competition,

I am really having a hard time understanding where this obsession with mythical socialism comes from. The reality we live in is largely capitalistic and a striving towards a monopoly - i.e. a lack of competition - is basically the entire purpose of a corporation, which is only kept in check by government regulations.

Re: FOSS infrastructure is under attack by AI companies

#528

Post by steveklabnik »

devit wrote: post #495

Well, looking at the SourceHut code, it's written in Python and handles git by spawning a "git" process. In other words, it was written with no consideration for performance at all. A competent engineer would use Rust or C++ with an in-process git library, perhaps rewrite part of the git library or git storage system if necessary for high performance, and would design a fast storage system with SSDs, and rate-limit s…

> A competent engineer would use Rust or C++ with an in-process git library,

The issue is, there aren't any fully featured ones of these yet. Sure, they do exist, but you run into issues. Spawning a git process isn't about not considering performance, it's about correctness. You simply won't be able to support a lot of people if you don't just spawn a git process.

Re: FOSS infrastructure is under attack by AI companies

#529

Post by Ndymium »

devit wrote: post #516
Ndymium wrote: post #215

I was also under attack recently [0]. The little Forgejo instance where I host my code (of several open source packages so it needs to be open) was run into the ground and the disk was filled with generated zip archives. I'm not the only one who has suffered the same fate. For me, the attacks subsided (for now) when I banned Alibaba Cloud's IP range. If you are hosting a Forgejo instance, I strongly recommend setting…

Or perhaps switch to well-engineered software actually properly designed to be served on the public Internet. Clearly generating zip files, writing them fully to disk and then sending them to the client all at once is a completely awful and unusable design, compared to the proper design of incrementally generating and transmitting them to the client with minimal memory consumption and no disk usage at all. The fact t…

Just noting that the archives are written to disk on purpose, as they are cached for 24 hours (by default). But when you have a several thousand commit repository, and the bots tend to generate all the archive formats for every commit…

But Forgejo is not the only piece of software that can have CPU intensive endpoints. If I can't fence those off with robots.txt, should I just not be allowed to have them in the open? And if I forced people to have an account to view my packages, then surely I'd have close to 0 users for them.

Re: FOSS infrastructure is under attack by AI companies

#530

Post by consumer451 »

I was wondering... Common Crawl [0] exists, why do "AI companies" do this on their own?

Is Common Crawl data not fresh enough? Is there some other deficiency?

Whatever the problem is, a single crawler which every AI company can reference seems like a compromise to solve this issue, doesn't it?

[0] https://commoncrawl.org/

Post reply on HN