Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devs say AI crawlers dominate traffic, forcing blocks on entire countries

arstechnica.com

Post reply on HN
Print view View original

111–120 of 291 posts

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#111

Post by seabird »

throwaway2037 wrote: post #101
rco8786 wrote: post #65

I got DoSed by ClaudeBot (Anthropic) just last week. Hitting a website I manage 700,000 times in one month and tripping our bandwidth limit with our hosting provider. What a PITA to have to investigate that, figure it out, block the user agent, and work with hosting provider support to get the limit lifted as a courtesy. Noticed that the ChatGPT bot was 2nd in traffic to this site, just not enough to cause trouble.

robots.txt did not work?

Of course it didn't work. At best, the dorks doing this think there's a gamechanging LLM application to justify the insane valuations right around the corner if they just scrape every backwater site they can find. At worst, they're doing it because it's paying good money. Either way, they don't care, they're just going to ignore robots.txt.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#113

Post by ndiddy »

Nckpz wrote: post #106

I recently started a side-project with a "code everything in prod" approach for fun. I've done this many times over the past 20 years and the bot traffic is usually harmless, but this has been different. I haven't advertised the hostname anywhere, and in less than 24 hours I had a bunch of spam form submissions. I've always expected this after minor publicity, but not "start server, instantly get raided by bots perfo…

This is yet another great example of the innovation that the AI industry is delivering. Why just limit your scraper bots to GET requests when there might be some juicy data to train on hidden behind that form? There's a reason why the a16z funded cracked vibe coder ninjas are taking over software engineering, they're full of wonderful ideas like this.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#115

Post by threatofrain »

cyanydeez wrote: post #46
randmeerkat wrote: post #20

Earlier quoted context omitted.

> Why? What is the goal of a scraper, and how does disabling the source of the data benefit them? The next scraper doesn’t get the data. People don’t realize we’re not compute limited for ai, we’re data limited. What we’re watching is the “data war”.

at this point we're _good data_ limited, which has little to do with scraping.

Scraping social media is good data, even without ML. The fact that something is "happening" to people in a social space inherently has importance to people. The specter of law is more threatening to whether companies can get their hands on good data.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#116

Post by [deleted] »

userbinator wrote: post #88

All these JS-heavy "anti bot" measures do is further entrench the browser monopoly, making it much harder for the minority of independents, while those who pay big $$$ can still bypass them. Instead I recommend a simple HTML form that asks questions with answers that LLMs cannot yet figure out or get consistently wrong. The more related to the site's content the questions are, the better; I remember some electronics…

[deleted]

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#118

Post by Terr_ »

userbinator wrote: post #88

All these JS-heavy "anti bot" measures do is further entrench the browser monopoly, making it much harder for the minority of independents, while those who pay big $$$ can still bypass them. Instead I recommend a simple HTML form that asks questions with answers that LLMs cannot yet figure out or get consistently wrong. The more related to the site's content the questions are, the better; I remember some electronics…

Or require every fresh "unique" visitor to run some JS that takes X seconds to compute.

It's not nice for visitors using a very old smartphone, but it's arguably less-exclusionary than some of the tests and third-party gatekeepers that exist now.

In many cases we don't actually care about telling if someone is truly a human alone, as much as ensuring that they aren't a throwaway sockpuppet of a larger automated system that doesn't care about good behavior because a replacement is so easy to make.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#119

Post by mrheosuper »

saganus wrote: post #78
kevin_thibedeau wrote: post #75

Earlier quoted context omitted.

The problem is that the server also has to do the work. Fine for an infrequent auth challenge. Not so fine for every single data request.

Maybe there is a way for the server to ask the client to do the work? Something similar to proof-of-work but on a much smaller scale than Bitcoin.

just add some delay to your response, we don't have to waste any more energy on meaningless calculation.
Post reply on HN