FOSS infrastructure is under attack by AI companies • Hacker News Board

Re: FOSS infrastructure is under attack by AI companies

#321

Post by b112 » Thu, Mar 20, 2025, 3:03 PM UTC

pjc50 wrote: post #10

ericholscher wrote: post #3

Yep -- our story here: https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse... (quoted in the OP) -- everyone I know has a similar story who is running large internet infrastructure -- this post does a great job of rounding a bunch of them up in 1 place. I called it when I wrote it, they are just burning their goodwill to the ground. I will note that one of the main startups in the space worked with us direct…

> just burning their goodwill to the ground AI firms seem to be leading from a position that goodwill is irrelevant: a $100bn pile of capital, like an 800lb gorilla, does what it wants. AI will be incorporated into all products whether you like it or not; it will absorb all data whether you like it or not.

Yes, like the Pixel camera app, which mangles photos with AI processing, and users complain that it won't let people take pics.

One issue was a pic with text in it, like a store sign. Users were complaining that it kept asking for better focus on the text in the background, before allowing a photo. Alpha quality junk.

Which is what AI is, really.

Re: FOSS infrastructure is under attack by AI companies

#322

Post by PaulDavisThe1st » Thu, Mar 20, 2025, 3:03 PM UTC

> They do so using random User-Agents from tens of thousands of IP addresses,

"tens of thousands" ? I think not:

  % sudo fail2ban-client status gitbots | more
  Status for the jail: gitbots
  |- Filter
  |  |- Currently failed: 0
  |  |- Total failed: 573555
  |  `- File list: /var/log/nginx/gitea_access.log
  `- Actions
     |- Currently banned: 78671
     |- Total banned: 573074

Re: FOSS infrastructure is under attack by AI companies

#323

Post by aspir » Thu, Mar 20, 2025, 3:03 PM UTC

Aachen wrote: post #275

RVuRnvbM2e wrote: post #250

Earlier quoted context omitted.

Were the walls you hit caused by Fastly's bot detection? I've found it to be quite accurate. On the other hand CloudFlare and Akamai mistakenly block me all the damn time.

It's not like they say, but it's at least three different implementations and I don't think any were cloudflare because I've been running into those pages for years and they've got captchas (functional or not). One of them was Akamai I think indeed

Yeah, I definitely don't want to pivot this thread into a product pitch, as the important thing is helping the open-source projects, but we can work with the maintainers to tune the systems to be as strict/lax as preferred. I'm sure the other services can too, to be fair.

Re: FOSS infrastructure is under attack by AI companies

#324

Post by corytheboyd » Thu, Mar 20, 2025, 3:04 PM UTC

I never thought about it until now, but it’s insane that the companies who offer both LLM products and cloud compute services are double dipping— they get the LLM product to sell, as well as the elevated load egress (and compute, etc.) money. When you look at it that way, where’s the incentive to even care about inefficient LLM scraping, leaving it terrible makes you money from your other empire, cloud storage egress costs.

Re: FOSS infrastructure is under attack by AI companies

#325

Post by GTP » Thu, Mar 20, 2025, 3:05 PM UTC

blibble wrote: post #240

01HNNWZ0MV43FF wrote: post #105

Earlier quoted context omitted.

It'll still keep your site from getting hammered

until some drone working for the parasites (google/facebook/openai) sees this post and writes 5 lines of code to defeat it and now you have an experience where the bots have it easier time accessing your content than legitimate visitors

How would those 5 lines of code look like? The base of this solution is that it asks to solve a computationally-intensive problem whose solution, once provided, isn't computationally-intensive to check. How would those 5 lines of code change this?

Re: FOSS infrastructure is under attack by AI companies

#326

Post by lacksconfidence » Thu, Mar 20, 2025, 3:06 PM UTC

what wrote: post #270

regularfry wrote: post #166

Earlier quoted context omitted.

How are the crawlers identifying themselves? If it's user agent strings then they can be faked. If it's cryptographically secured then you create a situation where newcomers can't get into the market.

Google publishes the ip addresses that google bot uses. If someone claims to be google bot but is not from one of those addresses, it’s a fake.

How about if they claim to be google chrome running on windows 11, from a residential IP address? Is that a human or an AI bot?

Re: FOSS infrastructure is under attack by AI companies

#327

Post by mycall » Thu, Mar 20, 2025, 3:06 PM UTC

Sharlin wrote: post #172

QuadrupleA wrote: post #46

Isn't this just poor, sloppy crawler implementation? You shouldn't need to fetch a repo more than once to add it to a training set.

The brute-force solution to the first hard problem in computer science is to not have a cache at all.

Yet in these cases mentioned in the article, if they had an HTTP static cache version of each page, using git hooks to refresh them, the bots would be negligable to their services. That is assuming the bots use HTTP 80/443 instead of git 9418

Re: FOSS infrastructure is under attack by AI companies

#328

Post by blibble » Thu, Mar 20, 2025, 3:06 PM UTC

GTP wrote: post #325

blibble wrote: post #240

Earlier quoted context omitted.

until some drone working for the parasites (google/facebook/openai) sees this post and writes 5 lines of code to defeat it and now you have an experience where the bots have it easier time accessing your content than legitimate visitors

How would those 5 lines of code look like? The base of this solution is that it asks to solve a computationally-intensive problem whose solution, once provided, isn't computationally-intensive to check. How would those 5 lines of code change this?

nice try, Google employee

Re: FOSS infrastructure is under attack by AI companies

#329

Post by yetihehe » Thu, Mar 20, 2025, 3:06 PM UTC

xena wrote: post #77

biophysboy wrote: post #62

Can someone with more experience developing AI tools explain what these bots are mostly doing? Are they collecting data for training, or are they for the more recent search functionality? Or are they enhancing responses with links?

AI expert here. It's probably for collecting training data and the crawlers are probably very unsupervised. I'd guess that they're literally the most simplistic crawler code you can imagine combined with parallelism across machines. The good news is that it's easy to disrupt these crawlers with some easy hacks. Tactical reverse slowloris is probably gonna make a comeback.

Zipbombing rogue AI's will be the new hotness.

Re: FOSS infrastructure is under attack by AI companies

#330

Post by smashah » Thu, Mar 20, 2025, 3:07 PM UTC

harhargange wrote: post #55

Can IPFS or torrent and large local databases decentralised by people be a solution to this? I personally have the resources to share and host TBs of data but didn't find a good use to it.

The real solution to this is for cloud fees to fall by 80%.