Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devs say AI crawlers dominate traffic, forcing blocks on entire countries

arstechnica.com

Post reply on HN
Print view View original

271–280 of 291 posts

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#271

Post by kragen »

Retric wrote: post #126
kragen wrote: post #37

Earlier quoted context omitted.

"You're cut off from access to knowledge because you live in the same country as AI researchers"?

A DoS attack is a DoS attack even if someone is pretending to be a “Researcher.” People in Iran, Russia, etc get annoyed with sanctions but that’s kind of the point. If your government isn’t responding appropriately, yes you’ll get shafted it’s what you do after that which solves the problem.

Me, I prefer to relate to people as individuals rather than, as you are advocating, interchangeable representatives of their area of residence. If what you want is World War III, this is how you get it.

In particular, universal access to knowledge is a fundamental principle of liberalism.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#272

Post by userbinator »

Terr_ wrote: post #181
userbinator wrote: post #124

Earlier quoted context omitted.

that takes X seconds to compute. Those who have the computing resources to do commercial scraping will easily get past that. In contrast, there are still many questions which a human can easily answer, but even the best LLMs currently can't.

It doesn't have to be bulletproof, it just has to create a cost that doesn't scale economically for them.

Computing power is cheap, and getting cheaper for the big guys. Real humans are not.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#273

Post by GoblinSlayer »

everdrive wrote: post #204

One more reason we're moving away from privacy. Didn't load all the javascript domains? You're probably a bot. Not signed in? You're probably a bot. The web we knew is dying step by step. One interesting thought: do we know if these AI crawlers intentionally avoid certain topics? Is pornography totally left unscathed by these bots? How about extreme political opinions?

No, https://www.heise.de/en/news/Poisoning-training-data-Russian...

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#274

Post by GoblinSlayer »

userbinator wrote: post #88

All these JS-heavy "anti bot" measures do is further entrench the browser monopoly, making it much harder for the minority of independents, while those who pay big $$$ can still bypass them. Instead I recommend a simple HTML form that asks questions with answers that LLMs cannot yet figure out or get consistently wrong. The more related to the site's content the questions are, the better; I remember some electronics…

The algorithm is inspired by hashcash https://raw.githubusercontent.com/TecharoHQ/anubis/refs/head...

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#275

Post by GoblinSlayer »

mbesto wrote: post #73
nosianu wrote: post #11

Earlier quoted context omitted.

You mean like this? [2025-03-19] https://blog.cloudflare.com/ai-labyrinth/ > Trapping misbehaving bots in an AI Labyrinth > Today, we’re excited to announce AI Labyrinth, a new mitigation approach that uses AI-generated content to slow down, confuse, and waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives.

Wait, what happens when a Cloudflare Worker AI meets an AI Labyrinth?!

Rise of the machines.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#276

Post by GoblinSlayer »

karlgkk wrote: post #123
rfurmani wrote: post #121

After I opened up https://sugaku.net to be usable without login, it was astounding how quickly the crawlers started. I'd like the site to be accessible to all, but I've had to restrict most of the dynamic features to logged in users, restrict robots.txt, use cloudflare to block AI crawlers and bad bots, and I'm still getting ~1M automated requests per day (compared to ~1K organic), so I think I'll need to restrict th…

One thing that worked well for me was layering obstacles It really sucks that this is the way things are, but what I did was 10 requests for pages in a minute, you get captchad (with a little apology and the option to bypass it by logging in). asset loads don’t count After a captcha pass, 100 requests in an hour gets you auth walled It’s really shitty but my industry is used to content scraping. This allows legit use…

Or just buy cloudflare :)

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#277

Post by Retric »

kragen wrote: post #271
Retric wrote: post #126

Earlier quoted context omitted.

A DoS attack is a DoS attack even if someone is pretending to be a “Researcher.” People in Iran, Russia, etc get annoyed with sanctions but that’s kind of the point. If your government isn’t responding appropriately, yes you’ll get shafted it’s what you do after that which solves the problem.

Me, I prefer to relate to people as individuals rather than, as you are advocating, interchangeable representatives of their area of residence. If what you want is World War III, this is how you get it. In particular, universal access to knowledge is a fundamental principle of liberalism.

I’d personally love for someone to hand me a billion dollars no strings attached.

That’s got nothing to do with solving the issues created by these people, but if you’re going to toss out meaningless non sequitur’s then I figure I might as well join in on the fun.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#278

Post by kristiandupont »

userbinator wrote: post #261
kristiandupont wrote: post #155

Earlier quoted context omitted.

>there are still many questions which a human can easily answer, but even the best LLMs currently can't. I am genuinely curious: what is an example of such a question, if it's for a person you don't know (i.e. where you cannot rely on inside knowledge)?

This was a notable example in that category: https://news.ycombinator.com/item?id=41058318 Also https://news.ycombinator.com/item?id=38766512

I just tried these with ChatGPT (4o) and it got both of them right. That's not to say that you won't be able to find something that still works but I think that particular hole is closing fast.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#279

Post by egypturnash »

misnome wrote: post #55
egypturnash wrote: post #31

This is 100% off-topic but: Is it just me or does Ars keep on serving videos about the sound design of Callisto Protocol in the middle of everything? Why do they keep on promoting these videos about a game from 2022? They've been doing this for months now.

I also get this for 1+ years, and just assume that it’s a symptom of my adblockers/pihole working and screwing up something in their auto ad targeting.

I’m glad to know it’s not just me! It’s such a weird and specific failure state.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#280

Post by brookst »

i5heu wrote: post #256
brookst wrote: post #248

Earlier quoted context omitted.

You can claim whatever you want, but actually litigation is expensive and it’s not at all a sure thing that “I made a publicly available resource and they used it too much” is going to win damages. Maybe? Maybe not?

So I could ddos anyone legally as long I have a some reason?

Sure. Courts tend to care about intent, but your are welcome to try to change that.
Post reply on HN