FOSS infrastructure is under attack by AI companies • Hacker News Board

Re: FOSS infrastructure is under attack by AI companies

#61

Post by nicce » Thu, Mar 20, 2025, 1:32 PM UTC

lelanthran wrote: post #42

The big takeaway here is that Google's (and advertisement in general) dominance over the web is going away. This is because the only way to stop the bots is with a captcha, and this also stops search indexers from indexing your site. This will result in search engines not indexing sites, and hence providing no value anymore. There's probably going to be a small lag as the current knowledge in the current LLMs dry up…

> The big takeaway here is that Google's (and advertisement in general) dominance over the web is going away.

AI companies with best anti-captcha mechanics will win and will inject ads to LLM output in more sophisticated way.

Re: FOSS infrastructure is under attack by AI companies

#62

Post by biophysboy » Thu, Mar 20, 2025, 1:32 PM UTC

Can someone with more experience developing AI tools explain what these bots are mostly doing? Are they collecting data for training, or are they for the more recent search functionality? Or are they enhancing responses with links?

Re: FOSS infrastructure is under attack by AI companies

#63

Post by piokoch » Thu, Mar 20, 2025, 1:33 PM UTC

Copyright the content and sue those who use it for AI training. I believe there is a lot of low hanging fruit for lawyers here. I would be surprised if they weren't preparing to hit Open AI and alikes. Very badly. Google get away with deep linking issues as publishers after all had some interest in being linked from the search engine, here publishers see zero value.

Re: FOSS infrastructure is under attack by AI companies

#64

Post by MrArthegor » Thu, Mar 20, 2025, 1:33 PM UTC

FeepingCreature wrote: post #41

To be clear, this is not an attack in the deliberate sense, and has nothing to do with AI except in that AI companies want to crawl the internet. This is more "FOSS sites damaged by extreme incompetence and unaccountability." The crawlers could just as well be search engine startups.

Yes, any crawler or some traffic peak can do this if your infrastructure is not well engineered

Re: FOSS infrastructure is under attack by AI companies

#65

Post by johnnyanmac » Thu, Mar 20, 2025, 1:33 PM UTC

FeepingCreature wrote: post #41

To be clear, this is not an attack in the deliberate sense, and has nothing to do with AI except in that AI companies want to crawl the internet. This is more "FOSS sites damaged by extreme incompetence and unaccountability." The crawlers could just as well be search engine startups.

In the same way a drunk driver isn't deliberately trying to run over pedestrians, I suppose. I think gross negligence is in many ways worse than malice. A malicious actor can at least be somewhat reasoned with.

Re: FOSS infrastructure is under attack by AI companies

#66

Post by pfoof » Thu, Mar 20, 2025, 1:33 PM UTC

totetsu wrote: post #2

Could one put a mangler on the responses to suspected bots to poison their data sets with nonsense code.. :/

a proper CSS or comments in HTML might do this

Re: FOSS infrastructure is under attack by AI companies

#67

Post by ferguess_k » Thu, Mar 20, 2025, 1:34 PM UTC

ericholscher wrote: post #3

Yep -- our story here: https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse... (quoted in the OP) -- everyone I know has a similar story who is running large internet infrastructure -- this post does a great job of rounding a bunch of them up in 1 place. I called it when I wrote it, they are just burning their goodwill to the ground. I will note that one of the main startups in the space worked with us direct…

Maybe just feed them dynamically generated garbage information? More fun than no information.

Re: FOSS infrastructure is under attack by AI companies

#68

Post by ziddoap » Thu, Mar 20, 2025, 1:34 PM UTC

piokoch wrote: post #63

Copyright the content and sue those who use it for AI training. I believe there is a lot of low hanging fruit for lawyers here. I would be surprised if they weren't preparing to hit Open AI and alikes. Very badly. Google get away with deep linking issues as publishers after all had some interest in being linked from the search engine, here publishers see zero value.

>Copyright the content and sue those who use it for AI training.

This costs money, time, and ongoing commitment. FOSS isn't typically known for being overflowing with cash.

Re: FOSS infrastructure is under attack by AI companies

#69

Post by sir-alien » Thu, Mar 20, 2025, 1:35 PM UTC

napolux wrote: post #35

sir-alien wrote: post #20

It's going to get to the point where everything will be put behind a login to prevent LLM scrapers scanning a site. Annoying but the only option I can think of. If they use an account for scraping you just ban the account.

and then they'll login there too...

logins are more easily banned, and highly complex captchas for signup needs a human to signup and solve. As long as it's easier to get banned than it is to signup it will at least deter.

Re: FOSS infrastructure is under attack by AI companies

#70

Post by breckenedge » Thu, Mar 20, 2025, 1:35 PM UTC

lelanthran wrote: post #42

The big takeaway here is that Google's (and advertisement in general) dominance over the web is going away. This is because the only way to stop the bots is with a captcha, and this also stops search indexers from indexing your site. This will result in search engines not indexing sites, and hence providing no value anymore. There's probably going to be a small lag as the current knowledge in the current LLMs dry up…

Google is already scraping your site and presenting answers directly in search results. If I cared about traffic (hence selling ad space), why would I want my site indexed by Google at all anymore? Lots of advertising-supported sites are going to go dark because only bots will visit them.