Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FOSS infrastructure is under attack by AI companies

thelibre.news

Post reply on HN
Print view View original

591–600 of 631 posts

Re: FOSS infrastructure is under attack by AI companies

#591

Post by PaulDavisThe1st »

pabs3 wrote: post #579
PaulDavisThe1st wrote: post #417

Earlier quoted context omitted.

We do precisely this ... and we're still dealing with the load issues. Currently I have fail2ban doing a 10 day block on any IP addr that hits our read only http-git endpoint twice in 30 mins. The problem with this is that the default implementation of iptables doesn't scale well to 100k blocked addresses.

Does that not cause your devs, or people doing contributions, or browsing commits via the web, to get blocked?

Yes, it does mean that anyone using the web-based interface to git gets blocked.

We mirror to github for public access; our developers all use git itself, not the web interface, for interacting with the repo.

How/what github et al. are doing to deal with this, I do not know.

Re: FOSS infrastructure is under attack by AI companies

#592

Post by kotenok2000 »

eb0la wrote: post #339
xena wrote: post #317

Earlier quoted context omitted.

Author of Anubis here. If that happens, I win.

If that happens, count with me to use Anubis to factor large primes or whatever science needs as a background task.

Shouldn't you factor composite numbers? Factoring prime numbers is pointless.

Re: FOSS infrastructure is under attack by AI companies

#593

Post by biophysboy »

xena wrote: post #521
biophysboy wrote: post #513

Earlier quoted context omitted.

If its for training data, why are they straining FOSS so much? Is there thousands of actors repeatedly making training data all the time? I thought it was a sort of one-off thing w/ the big tech players.

Git forges are some of the worst case for this. The scrapers click on every link on every page. If you do this to a git forge, it gets very O(scary) very fast because you have to look at data that is not frequently looked at and will NOT be cached. Most of how git forges are fast is through caching. The thing about AI scrapers is that they don't just do this once. They do this every day in case every file in a glibc…

That's very strange to me that they do it everyday. I thought training runs took months. Do they throw away the vast majority of their training attempts (e.g. one had suboptimal hyperparameters, etc)?

Re: FOSS infrastructure is under attack by AI companies

#594

Post by pipe01 »

xena wrote: post #198
hmry wrote: post #168

Earlier quoted context omitted.

I hope lots of websites adopt this, mainly because I want to see more happy jackal girls while browsing.

My monetization strategy is unironically to offer a de-anime'd version under the name Techaro BotStopper or something.

That's kinda what the company Friendly Captcha does

Re: FOSS infrastructure is under attack by AI companies

#595

Post by joquarky »

xyzal wrote: post #159

In case anyone is interested in a tiny bit of sabotage, I am under the impression I managed to 'drown' true information on my microblog by generating contradicting posts with LLaMa (tens of them for each real post) and invisibly linking them, so a human would not click through. You know, flood the zone with s***, Bannon-style ...

When I read this yesterday, I was contemplating one possible way to mitigate this at a larger scale is if websites could create random virtual paths/endpoints that drive the bot into a locally served Library of Babel[0] that poisons the spiders with lots of useless text.

It won't work for well-structured sites where the bots know the exact endpoint they want to scrape, but might slow down the more exploratory spider threads.

[0] https://libraryofbabel.info/

Re: FOSS infrastructure is under attack by AI companies

#596

Post by joquarky »

noveltyaccount wrote: post #358
xena wrote: post #142

Earlier quoted context omitted.

Hi! I do this! See https://github.com/TecharoHQ/anubis for more info!

Does the PoW make money via crypto mining? Or is it just to waste the caller's CPU cycles? If you could monetize the PoW then you could re-challenge at an interval tuned so that the caller pays for their usage.

This feels like one of the few ways to potentially avoid what seems like the inevitability of attestation.

Re: FOSS infrastructure is under attack by AI companies

#597

Post by joepie91_ »

Ray20 wrote: post #445
lgeek wrote: post #258

Earlier quoted context omitted.

> One crawler downloaded 73 TB of zipped HTML files in May 2024 [...] This cost us over $5,000 in bandwidth charges I had to do a double take here. I run (mostly using dedicated servers) infrastructure that handles a few hundred TB of traffic per month, and my traffic costs are on the order of $0.50 to $3 per TB (mostly depending on the geographical location). AWS egress costs are just nuts.

I think uncontrolled price of cloud traffic - is a real fraud and way bigger problem then some AI companies that ignore robot.txt. One time we went over limit on Netlify or something, and they charged over thousand for a couple TB.

> I think uncontrolled price of cloud traffic - is a real fraud

Yes, it is.

> and way bigger problem then some AI companies that ignore robot.txt.

No, it absolutely is not. I think you underestimate just how hard these AI companies hammer services - it is bringing down systems that have weathered significant past traffic spikes with no issues, and the traffic volumes are at the level where literally any other kind of company would've been banned by their upstream for "carrying out DDoS attacks" months ago.

Re: FOSS infrastructure is under attack by AI companies

#598

Post by joepie91_ »

k1tanaka wrote: post #571
xena wrote: post #395

Earlier quoted context omitted.

The magic of proof of work is that it's something that's really hard to do but easy to validate. Anubis' proof of work works like this: A sha256 hash is a bunch of bytes like this: 394d1cc82924c2368d4e34fa450c6b30d5d02f8ae4bb6310e2296593008ff89f We usually write it out in hex form, but that's literally what the bytes in ram look like. In a proof of work validation system, you take some base value (the "challenge") an…

I am sorry if this question is dumb, but how does proof of work deter bots/scrappers from accessing a website? I imagine it costs more resource to access the protected website but would this stop the bots? Wouldn't they be able to pass the challenge and scrap the data after? Or normal scrapbots usually timeout after a small amount of time/ resources is used?

There are a few ways in which bots can fail to get past such challenges, but the most durable one (ie. the one that you cannot work around by changing the scraper code) is that it simply makes it much more expensive to make a request.

Like spam, this kind of mass-scraping only works because the cost of sending/requesting is virtually zero. Any cost is going to be a massive increase compared to 'virtually zero', at the kind of scale they operate at, even if it would be small to a normal user.

Re: FOSS infrastructure is under attack by AI companies

#600

Post by joepie91_ »

bitmasher9 wrote: post #308
Cthulhu_ wrote: post #249

Back when search engines caused this, the industry made an agreement and designed the robots.txt spec in order to avoid legal frameworks being made to stop them. Because of that, legal frameworks weren't being made. Now there's a new generation of hungry hungry hippo indexers that didn't agree to that and who feel intense pressure from competition to scoop up as much data as they can, who just ignore it. Legislation…

I’m not sure that I like this plan. We shouldn’t let the illegal AIs gain more knowledge and usefulness than the legal ones. There’s other options besides a blanket ban.

None of the AIs have any 'knowledge' to begin with, so that's an easy one to satisfy.
Post reply on HN