Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FOSS infrastructure is under attack by AI companies

thelibre.news

Post reply on HN
Print view View original

571–580 of 631 posts

Re: FOSS infrastructure is under attack by AI companies

#571

Post by k1tanaka »

xena wrote: post #395
reginald78 wrote: post #377

Earlier quoted context omitted.

Maybe I'm missing something, but doesn't this mean the work has to be done by the client AND the server every time a challenge is issued? I think ideally you'd want work that was easy for the server and difficult for the server . And what is to stop being DDoS'd by clients that are challenged but neglect to perform the challenge? Regardless, I think something like this is the way forward if one doesn't want to throw…

The magic of proof of work is that it's something that's really hard to do but easy to validate. Anubis' proof of work works like this: A sha256 hash is a bunch of bytes like this: 394d1cc82924c2368d4e34fa450c6b30d5d02f8ae4bb6310e2296593008ff89f We usually write it out in hex form, but that's literally what the bytes in ram look like. In a proof of work validation system, you take some base value (the "challenge") an…

I am sorry if this question is dumb, but how does proof of work deter bots/scrappers from accessing a website?

I imagine it costs more resource to access the protected website but would this stop the bots? Wouldn't they be able to pass the challenge and scrap the data after? Or normal scrapbots usually timeout after a small amount of time/ resources is used?

Re: FOSS infrastructure is under attack by AI companies

#572

Post by olleromam91 »

rectang wrote: post #307
WhyOhWhyQ wrote: post #277

Earlier quoted context omitted.

I've pointed this out to a few people in this space. They tend to suggest that the value in AI is so great this means we should get rid of copyright law entirely.

AI is worthless without training data. If all content becomes AI generated because AI outcompetes original content then there will be no data left to train on. When Google first came out in 1998, it was amazing, spooky how good it was. Then people figured out how to game pagerank and Google's accuracy cratered. AI is now in a similar bubble period. Throwing out all of copyright law just for the benefit of a few oliga…

Are we not past past critical mass though? The velocity at which these things can out compete human labor is astonishing, any future human creations or original content will already have lost the battle the moment it goes online and gets cloned by AI.

Re: FOSS infrastructure is under attack by AI companies

#573

Post by dbmnt »

k1tanaka wrote: post #571
xena wrote: post #395

Earlier quoted context omitted.

The magic of proof of work is that it's something that's really hard to do but easy to validate. Anubis' proof of work works like this: A sha256 hash is a bunch of bytes like this: 394d1cc82924c2368d4e34fa450c6b30d5d02f8ae4bb6310e2296593008ff89f We usually write it out in hex form, but that's literally what the bytes in ram look like. In a proof of work validation system, you take some base value (the "challenge") an…

I am sorry if this question is dumb, but how does proof of work deter bots/scrappers from accessing a website? I imagine it costs more resource to access the protected website but would this stop the bots? Wouldn't they be able to pass the challenge and scrap the data after? Or normal scrapbots usually timeout after a small amount of time/ resources is used?

Put simply, most bots just aren't designed to solve such challenges.

Re: FOSS infrastructure is under attack by AI companies

#574

Post by DidYaWipe »

joquarky wrote: post #539
DidYaWipe wrote: post #503

Earlier quoted context omitted.

What an infantile worldview.

> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes. https://news.ycombinator.com/newsguidelines.html

OK. To be clear, that wasn't about the OP, but rather the alleged people promoting the abolition of copyright... which would significantly hurt open source.

The people agitating for such things are usually leeches who want everything free and do, in fact, hold an infantile worldview that doesn't consider how necessary remuneration is to whatever it is they want so badly (media pirates being another example).

Not that I haven't "pirated" media, but this is usually the result of it not being available for purchase or my already having purchased it.

Re: FOSS infrastructure is under attack by AI companies

#575

Post by pabs3 »

xena wrote: post #33

It's really surreal to see my project in the preview image like this. That's wild! If you want to try it: https://github.com/TecharoHQ/anubis . So far I've noticed that it seems to actually work. I just deployed it to xeiaso.net as a way to see how it fails in prod for my blog.

Could you add an option for non-JS users? Maybe a Linux command-line we can paste the output of into a form.

Re: FOSS infrastructure is under attack by AI companies

#576

Post by dbmnt »

boris wrote: post #493
dbaio wrote: post #373

This is how we handled this in the FreeBSD infrastructure. https://blog.sysopscafe.com/posts/ai-crawlers-hammering-git-... It has been running this way for a month, and the workload is fine. I hope it continues this way.

Thanks for sharing. If I understood correctly, you have rate-limiter specific urls (those with commit ids) that are infrequently requested by users but frequently by bots. Which means, provided the bots continue trying to request them, any user request will most likely end up being denies. In this case a simpler solution might be to just block such urls outright. The only advantage of your more complex solution that…

My guess after reading the same -- the bot traffic comes in bursts and targets a specific commit hash for a while. Users are unlikely to need that specific commit, and even less likely to need it at the same time a bot is bursting requests for it. There's probably a small risk of denying a real user, but there's a large reduction in traffic from the bots making it to git; a worthwhile trade.

Re: FOSS infrastructure is under attack by AI companies

#577

Post by pabs3 »

madeofpalk wrote: post #388
Suppafly wrote: post #331

Earlier quoted context omitted.

>Most normies just think everyone has a Facebook/Instagram account and can't even imagine a world where that is not the case. The people without a basic internet presence aren't likely to be customers anyway so it's not a huge loss. It's trivial to setup a basic account for any site that doesn't contain any personal data you want to keep hidden, if you aren't willing to do that, you're in a tiny minority.

I don't have a Facebook or Instagram account, but I definitely eat tacos and I was put off when I couldn't see a new taco place's opening hours without an instagram accoutn. I'm not sure why you think why people who don't have a Facebook account wouldn't eat at restaurants

There are ways to access public Instagram content without an account, gallery-dl for eg works most of the time. Tweeper can also be useful.

https://github.com/mikf/gallery-dl https://git.ao2.it/tweeper.git

Re: FOSS infrastructure is under attack by AI companies

#578

Post by pabs3 »

3029aGfqgaF wrote: post #178

Open source licenses are no longer strong enough. The copyright laundering use case was not been anticipated when the licenses were written. All licenses need a clause like the following: This software is for humans. AI training is prohibited and carries a default penalty of $1 trillion.

Such a license would not meet the Open Source Iniative's Open Source Definition:

https://opensource.org/osd/

Re: FOSS infrastructure is under attack by AI companies

#579

Post by pabs3 »

PaulDavisThe1st wrote: post #417
diggan wrote: post #398

Earlier quoted context omitted.

Offering read-only mirrors via git+ http:// might be a solution then, at least to shed the load if anything. It does remind me a bit about companies complaining about being scraped and trying to prevent it, instead of offering a API so no one would have to scrape them.

We do precisely this ... and we're still dealing with the load issues. Currently I have fail2ban doing a 10 day block on any IP addr that hits our read only http-git endpoint twice in 30 mins. The problem with this is that the default implementation of iptables doesn't scale well to 100k blocked addresses.

Does that not cause your devs, or people doing contributions, or browsing commits via the web, to get blocked?

Re: FOSS infrastructure is under attack by AI companies

#580

Post by pabs3 »

diggan wrote: post #402
myaccountonhn wrote: post #397

Earlier quoted context omitted.

There is nothing that says you can't charge money for FOSS software. FOSS is more about having the ability to inspect and freely change your software to your use-cases.

> There is nothing that says you can't charge money for FOSS software Well, yes and no. If you had a cost to access the source code, I'm pretty sure I'd stop calling that FOSS. If you only have a price for downloading binaries, sure, still FOSS, since we're talking source code licensing. > Nothing should be $$ free I took this statement at face value, and assumed parent argued for basically eliminating FOSS.

Something can be FOSS even if it isn't released to the public, just amongst friends. Its only about every user being able to access the source.
Post reply on HN