Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FOSS infrastructure is under attack by AI companies

thelibre.news

Post reply on HN
Print view View original

251–260 of 631 posts

Re: FOSS infrastructure is under attack by AI companies

#251

Post by TuringNYC »

ericholscher wrote: post #3

Yep -- our story here: https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse... (quoted in the OP) -- everyone I know has a similar story who is running large internet infrastructure -- this post does a great job of rounding a bunch of them up in 1 place. I called it when I wrote it, they are just burning their goodwill to the ground. I will note that one of the main startups in the space worked with us direct…

>> which I then emailed 3x and never got a reply.

At which point does the crawling cease to be a bug/oversight and constitute a DDOS?

Re: FOSS infrastructure is under attack by AI companies

#252

Post by rectang »

UncleMeat wrote: post #30
pjc50 wrote: post #10

Earlier quoted context omitted.

> just burning their goodwill to the ground AI firms seem to be leading from a position that goodwill is irrelevant: a $100bn pile of capital, like an 800lb gorilla, does what it wants. AI will be incorporated into all products whether you like it or not; it will absorb all data whether you like it or not.

Yep. And it is much more far reaching than that. Look at the primary economic claim offered by AI companies: to end the need for a substantial portion of all jobs on the planet. The entire vision is to remake the entire world into one where the owners of these companies own everything and are completely unconstrained. All intellectual property belongs to them. All labor belongs to them. Why would they need good will…

> Look at the primary economic claim offered by AI companies: to end the need for a substantial portion of all jobs on the planet.

And this is why AI training is not "fair use". The AI companies seek to train models in order to compete with the authors of the content used to train the models.

A possible eventual downfall of AI is that the risk of losing a copyright infringement lawsuit is not going away. If a court determines that the AI output you've used is close enough to be considered a derivative work, it's infringement.

Re: FOSS infrastructure is under attack by AI companies

#253

Post by ThinkBeat »

Banning Edge to deal with the problem is probably not ideal.

What I am thinking about may even be less idea.

For people actively working on these projects how about puptting the git server on a private net with VPN or SSH access.

Use a seperate read only static git server to the net.

Re: FOSS infrastructure is under attack by AI companies

#254

Post by xena »

isoprophlex wrote: post #241
xena wrote: post #233

Earlier quoted context omitted.

There's no way to really make a progress bar make sense, it's a luck-based mechanic.

It'll be somewhat involved, but based on the difficulty vs the clients hashing speed you could say something probabilistic like "90% of the time, this window will be gone in xyz seconds from now"?

Yeah, I have to get the data for that though! I'm gonna add that to the list.

Re: FOSS infrastructure is under attack by AI companies

#255

Post by insane_dreamer »

sir-alien wrote: post #103
lelanthran wrote: post #42

The big takeaway here is that Google's (and advertisement in general) dominance over the web is going away. This is because the only way to stop the bots is with a captcha, and this also stops search indexers from indexing your site. This will result in search engines not indexing sites, and hence providing no value anymore. There's probably going to be a small lag as the current knowledge in the current LLMs dry up…

Can we not just have a whitelist for allowed crawlers and ban the rest by default? Then places like DuckDuckGo and Google can provide a list of IP addresses that their crawlers will come from. Then simply just don't include major LLM providers like OpenAI

Or an open list of IPs that are identified as AI companies that is updated regularly and firewalls can be easily updated with? (Same idea as open source AV)

Re: FOSS infrastructure is under attack by AI companies

#258

Post by lgeek »

ericholscher wrote: post #3

Yep -- our story here: https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse... (quoted in the OP) -- everyone I know has a similar story who is running large internet infrastructure -- this post does a great job of rounding a bunch of them up in 1 place. I called it when I wrote it, they are just burning their goodwill to the ground. I will note that one of the main startups in the space worked with us direct…

> One crawler downloaded 73 TB of zipped HTML files in May 2024 [...] This cost us over $5,000 in bandwidth charges

I had to do a double take here. I run (mostly using dedicated servers) infrastructure that handles a few hundred TB of traffic per month, and my traffic costs are on the order of $0.50 to $3 per TB (mostly depending on the geographical location). AWS egress costs are just nuts.

Re: FOSS infrastructure is under attack by AI companies

#259

Post by sozforex »

billy99k wrote: post #197
lukevp wrote: post #155

Earlier quoted context omitted.

Capitalism is an unthinking, unfeeling force. The writing is on the wall that AI is coming, and being altruistic about it doesn’t do jack to keep others from the land grab. Their thinking is, might as well join the rush and hope they’re one of the winners. Every one of us sitting on the sidelines will be impacted in some way or the other. So who’re the smart ones, the ones who grab shovels and start digging, or the o…

I already started incorporating AI into my workflow. It's definitely helped with productivity. At some point in the future, if you aren't using AI, you won't be able to compete in the job market.

At some point in the future, if you aren't AI, you won't be able to compete in the job market.

Re: FOSS infrastructure is under attack by AI companies

#260

Post by pca006132 »

nottorp wrote: post #213
falcor84 wrote: post #122

Earlier quoted context omitted.

I would absolutely love this as an alternative to the ad-based model

That is called micropayments and doesn't need to have anything to do with blockchain. All you need is a central clearing house service that can handle billions of 0.000001 transactions per day. Incidentaly, I doubt the bitcoin chain could handle that...

There are off-chain solutions to handle most of the payment, and only put a summary on-chain. I think there are already micropayments in Brave or something.
Post reply on HN