Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devs say AI crawlers dominate traffic, forcing blocks on entire countries

arstechnica.com

Post reply on HN
Print view View original

151–160 of 291 posts

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#151

Post by MathMonkeyMan »

throwaway2037 wrote: post #97
timewizard wrote: post #91

Earlier quoted context omitted.

It's to rip off copyrighted content and profit from it instead of the original authors. It's like every other low rent and highly automated scam that finds it's way onto the internet. If you look closely even Google does this. This is probably why many popular sites started getting down ranked in the last 2 years. Now they're below the fold and Google can present their content as their own through the AI box.

Please remember that Google only needs to be marginally better than the competition. And, of course, their primary biz is ads, not serving great results; that is a distant second priority.

Their biz is ads, but since search is winner takes all they need only be marginally better than the competition... twenty years ago.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#152

Post by 01HNNWZ0MV43FF »

timewizard wrote: post #90
emseetech wrote: post #76

Earlier quoted context omitted.

Nobody has unlimited resources. Everything is a cost-benefit analysis. For highly valuable information, they might throw the GDP of a small country at scraping your site. But most information isn't worth that. And there are a lot of bad actors who don't have the resources you're thinking of that are trying to compete with the big guys on a budget. This would cut them out of the equation.

Make all websites intentionally waste energy as a strategy to defeat unscrupulous operators has negative costs and marginal benefits.

I'll add a link on my site recommending that visitors petition their elected officials for a pollution tax

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#153

Post by zzo38computer »

My issue is not to prevent others from obtaining copies of the files, using Lynx or curl, disabling JavaScripts and CSS and pictures, etc. It is to prevent others from overloading the server due to badly behaved software.

I had briefly set up port knocking for the HTTP server (and only for HTTP; other protocols are accessible without port knocking), but due to a kernel panic I removed it and now the HTTP server is not accessible. (I may later put it back on once I can fix this problem.)

As far as I can tell, the LLM scrapers do not attempt to be "smart" about it at this time; if they do in future, you might try to take advantage of that somehow.

However, even if they don't, there are probably things that can be done. For example, check that if the declared user-agent declares things that it isn't doing, and display an error message if so (users who use Lynx will then remain unaffected and will still be able to access it). Another possibility is to try to confuse the scrapers however they are working, e.g. invalid redirects, valid redirects (e.g. to internal API functions of the companies that made them), invalid UTF-8, invalid compressed data, ZIP bombs (you can use the compression functions of HTTP to serve a small file that is too big when decompressed), EICAR test files, reverse pings (if you know who they really are), etc. What will work and what doesn't work depends on what software they are using.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#154

Post by nukem222 »

karlgkk wrote: post #123
rfurmani wrote: post #121

After I opened up https://sugaku.net to be usable without login, it was astounding how quickly the crawlers started. I'd like the site to be accessible to all, but I've had to restrict most of the dynamic features to logged in users, restrict robots.txt, use cloudflare to block AI crawlers and bad bots, and I'm still getting ~1M automated requests per day (compared to ~1K organic), so I think I'll need to restrict th…

One thing that worked well for me was layering obstacles It really sucks that this is the way things are, but what I did was 10 requests for pages in a minute, you get captchad (with a little apology and the option to bypass it by logging in). asset loads don’t count After a captcha pass, 100 requests in an hour gets you auth walled It’s really shitty but my industry is used to content scraping. This allows legit use…

> This allows legit users to get what they need.

Of course they could have just used the site directly.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#155

Post by kristiandupont »

userbinator wrote: post #124
Terr_ wrote: post #118

Earlier quoted context omitted.

Or require every fresh "unique" visitor to run some JS that takes X seconds to compute. It's not nice for visitors using a very old smartphone, but it's arguably less-exclusionary than some of the tests and third-party gatekeepers that exist now. In many cases we don't actually care about telling if someone is truly a human alone, as much as ensuring that they aren't a throwaway sockpuppet of a larger automated syste…

that takes X seconds to compute. Those who have the computing resources to do commercial scraping will easily get past that. In contrast, there are still many questions which a human can easily answer, but even the best LLMs currently can't.

>there are still many questions which a human can easily answer, but even the best LLMs currently can't.

I am genuinely curious: what is an example of such a question, if it's for a person you don't know (i.e. where you cannot rely on inside knowledge)?

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#156

Post by pabs3 »

ANarrativeApe wrote: post #71

Excuse my ignorance, but is it time to update the open source licenses in the light of this behavior? If so, what should the evolved license wording be? I appreciate that this could be easily circumvented by a 'bad actor', but it would make this abuse overt...

That wouldn't be an Open Source Definition compliant license. More in this subthread:

https://news.ycombinator.com/item?id=43423595

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#157

Post by xyzal »

I think the proper answer is to aim for the bots to get _negative_ utility value from visiting our sites, that is poisoning their well, not just zero value, that is to block them.

Did you try to GET a canary page forbidden in robots.txt? Very well, have a bucket load of articles on the benefits of drinking bleach.

Is your user-agent too suspicious? No problem, feel free to scrape my insecure code (google "emergent misalignment" for more info).

A request rate too inhuman? Here, take those generated articles about positive effect of catching measles on performance in bed.

And so on, and so forth ...

Nepenthes is nice, but word salad can be detected easily. It needs a feature that pre-generates linguistically plausible but factically garbage text via open models

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#158

Post by williamdclt »

bakugo wrote: post #142
Spivak wrote: post #131

Earlier quoted context omitted.

[flagged]

[flagged]

I’m no anime fan, but this sort of judgement on normalcy leaves me with a very sour impression about whoever says this sort of thing.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#159

Post by fc417fc802 »

nosianu wrote: post #11
edoloughlin wrote: post #6

I'm being trite, but if you can detect an AI bot, why not just serve them random data? At least they'll be sharing some of the pain they inflict.

You mean like this? [2025-03-19] https://blog.cloudflare.com/ai-labyrinth/ > Trapping misbehaving bots in an AI Labyrinth > Today, we’re excited to announce AI Labyrinth, a new mitigation approach that uses AI-generated content to slow down, confuse, and waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives.

> No real human would go four links deep into a maze of AI-generated nonsense.

... I would. Out of curiosity and amusement I would most definitely do that. Not every time, and not many times, but I would definitely do that one or a few times.

Guess I'm getting added to (yet another) Cloudflare naughty list.

> It is important to us that we don’t generate inaccurate content that contributes to the spread of misinformation on the Internet, so the content we generate is real and related to scientific facts, just not relevant or proprietary to the site being crawled.

In that case wouldn't it be faster and easier to restyle the CSS of wikipedia pages?

Post reply on HN