AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

Post by majkinetor » Thu, Aug 21, 2025, 5:03 PM UTC

rnhmjoj wrote: post #54

pjc50 wrote: post #49

Place alongside https://news.ycombinator.com/item?id=44962529 "Why are anime catgirls blocking my access to the Linux kernel?". This is why. AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher…

I'm far from being an AI enthusiast as anyone can be, but this issue has nothing to do with AI specifically. It's just that some greedy companies are writing incredibly shitty crawlers that don't follow any of the enstablished conventions (respecting robots.txt, using a proper UA string, rate limiting, whatever). This situation could have easily happened earlier than the AI boom, for different reasons.

Post by AutoDunkGPT » Thu, Aug 21, 2025, 5:04 PM UTC

tehwebguy wrote: post #44

This is a feature! If half the internet is nuked and the other half put up fences there is less readily available training data for competitors.

Post by lionkor » Thu, Aug 21, 2025, 5:09 PM UTC

mediumsmart wrote: post #36

lionkor wrote: post #34

Earlier quoted context omitted.

The second I get hit with bot traffic that makes my server heat up, I would just slam some aggressive anti bot stuff infront. Then you, my friend, are getting nothing with your fancy AI agent.

so the fancy AI agent will have to get really fancy and mimic human traffic and all is good until the server heats up from all those separate human trafficionados - then what?

Post by discardedrefuse » Thu, Aug 21, 2025, 5:17 PM UTC

mediumsmart wrote: post #36

lionkor wrote: post #34

Earlier quoted context omitted.

The second I get hit with bot traffic that makes my server heat up, I would just slam some aggressive anti bot stuff infront. Then you, my friend, are getting nothing with your fancy AI agent.

so the fancy AI agent will have to get really fancy and mimic human traffic and all is good until the server heats up from all those separate human trafficionados - then what?

Post by GuB-42 » Thu, Aug 21, 2025, 5:24 PM UTC

bakugo wrote: post #24

internet_points wrote: post #10

They mention anubis, cloudflare, robots.txt – does anyone have experiences with how much any of them help?

robots.txt is obviously only effective against well-behaved bots. OpenAI etc are usually well behaved, but there's at least one large network of rogue scraping bots that ignores robots.txt, fakes the user-agent (usually to some old Chrome version) and cycles through millions of different residential proxy IPs. On my own sites, this network is by far the worst offender and the "well-behaved" bots like OpenAI are barel…