Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devs say AI crawlers dominate traffic, forcing blocks on entire countries

arstechnica.com

Post reply on HN
Print view View original

91–100 of 291 posts

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#91

Post by timewizard »

DrFalkyn wrote: post #84
cyanydeez wrote: post #46

Earlier quoted context omitted.

at this point we're _good data_ limited, which has little to do with scraping.

Why kind of data that isn’t public would be so valuable for AI training? Seems like there’s a fuck ton. All of Wikipedia, GitHub for code, etc. I can understand targeting certain sites like Reddit, etc. but not random websites

It's to rip off copyrighted content and profit from it instead of the original authors. It's like every other low rent and highly automated scam that finds it's way onto the internet.

If you look closely even Google does this. This is probably why many popular sites started getting down ranked in the last 2 years. Now they're below the fold and Google can present their content as their own through the AI box.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#92

Post by ordersofmag »

CyberDildonics wrote: post #70

Can't you just rate limit beyond what a person would ever notice and do it by slowing the response?

Not if they hop to a different IP address every few requests. And they generally aren't bothered slow responses. It's not like they have to wait for one request to finish before they make another one (especially if they are making requests from thousands of machines).

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#93

Post by emseetech »

timewizard wrote: post #90
emseetech wrote: post #76

Earlier quoted context omitted.

Nobody has unlimited resources. Everything is a cost-benefit analysis. For highly valuable information, they might throw the GDP of a small country at scraping your site. But most information isn't worth that. And there are a lot of bad actors who don't have the resources you're thinking of that are trying to compete with the big guys on a budget. This would cut them out of the equation.

Make all websites intentionally waste energy as a strategy to defeat unscrupulous operators has negative costs and marginal benefits.

Resources applied to prevent bad actors from degrading or destroying the commons has always been the cost of civilization.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#95

Post by mvdtnz »

userbinator wrote: post #88

All these JS-heavy "anti bot" measures do is further entrench the browser monopoly, making it much harder for the minority of independents, while those who pay big $$$ can still bypass them. Instead I recommend a simple HTML form that asks questions with answers that LLMs cannot yet figure out or get consistently wrong. The more related to the site's content the questions are, the better; I remember some electronics…

How does JS entrench a browser monopoly? If you're not using vendor-specific JS extensions or non-standard APIs any browser should be able to execute your JS. Like most web developers I don't have a lot of patience for the people who refuse to run JS on their clients.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#96

Post by Retr0id »

userbinator wrote: post #89
tschwimmer wrote: post #77

Earlier quoted context omitted.

Check out Anubis - it's not quite what you're suggesting but similar in concept: https://anubis.techaro.lol/

I just hit a site with this --- and hit the back button immediately.

Better than a 500 error

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#97

Post by throwaway2037 »

timewizard wrote: post #91
DrFalkyn wrote: post #84

Earlier quoted context omitted.

Why kind of data that isn’t public would be so valuable for AI training? Seems like there’s a fuck ton. All of Wikipedia, GitHub for code, etc. I can understand targeting certain sites like Reddit, etc. but not random websites

It's to rip off copyrighted content and profit from it instead of the original authors. It's like every other low rent and highly automated scam that finds it's way onto the internet. If you look closely even Google does this. This is probably why many popular sites started getting down ranked in the last 2 years. Now they're below the fold and Google can present their content as their own through the AI box.

Please remember that Google only needs to be marginally better than the competition. And, of course, their primary biz is ads, not serving great results; that is a distant second priority.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#98

Post by fsckboy »

tedunangst wrote: post #82
kevin_thibedeau wrote: post #75

Earlier quoted context omitted.

The problem is that the server also has to do the work. Fine for an infrequent auth challenge. Not so fine for every single data request.

Tons of problems are easier to verify than to solve.

do any of them involve mining bitcoin?

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#99

Post by ronsor »

mbesto wrote: post #73
nosianu wrote: post #11

Earlier quoted context omitted.

You mean like this? [2025-03-19] https://blog.cloudflare.com/ai-labyrinth/ > Trapping misbehaving bots in an AI Labyrinth > Today, we’re excited to announce AI Labyrinth, a new mitigation approach that uses AI-generated content to slow down, confuse, and waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives.

Wait, what happens when a Cloudflare Worker AI meets an AI Labyrinth?!

Cloudflare deletes itself.
Post reply on HN