Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devs say AI crawlers dominate traffic, forcing blocks on entire countries

arstechnica.com

Post reply on HN
Print view View original

261–270 of 291 posts

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#261

Post by userbinator »

kristiandupont wrote: post #155
userbinator wrote: post #124

Earlier quoted context omitted.

that takes X seconds to compute. Those who have the computing resources to do commercial scraping will easily get past that. In contrast, there are still many questions which a human can easily answer, but even the best LLMs currently can't.

>there are still many questions which a human can easily answer, but even the best LLMs currently can't. I am genuinely curious: what is an example of such a question, if it's for a person you don't know (i.e. where you cannot rely on inside knowledge)?

This was a notable example in that category: https://news.ycombinator.com/item?id=41058318

Also https://news.ycombinator.com/item?id=38766512

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#262

Post by sigmoid10 »

soco wrote: post #215
sigmoid10 wrote: post #193

Earlier quoted context omitted.

This relies a lot on being able to detect bots. Everything you said could be easily bypassed with a small to moderate amount of effort on the side of crawler's creators. Distinguishing genuine traffic has always been hard and it will not get easier in the age of AI.

But the very comment you answered explains how to do it: a page forbidden in robots.txt. Does this method need explanation why it's ideal for sorting humans and google, from malicious crawlers?

The detection and bypass is trivial: Access the site from two IPs, one disrespecting robots.txt. If the content changes, you know it's garbage.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#263

Post by mostlysimilar »

rustc wrote: post #228
hec126 wrote: post #208

Earlier quoted context omitted.

You can sprinkle your site with almost-invisible hyperlinks. Bots will follow, humans will not.

This would be terrible for accessibility for users using a screen reader.

So would the site shutting down because AI bots are too much traffic.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#264

Post by WesolyKubeczek »

CaptainFever wrote: post #213
WesolyKubeczek wrote: post #165

Earlier quoted context omitted.

What do you think a search engine’s crawler bot is doing exactly? I could sure be wrong, but I have a hunch that “downloading content and paraing the HTML tags in a programmatic manner” describes it.

Yes, but the difference is that the term "scraping" also targets things like automatically generating RSS feeds from HTML pages, which is not covered by robots.txt.

I thought robots.txt covered all automated, programmatic access by third parties where a bot slurps stuff and follows links, without splitting hairs about it.

But what do I know, the young whippersnappers will just word lawyer me to death, so I better shut up and go away.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#265

Post by grotorea »

DrFalkyn wrote: post #84
cyanydeez wrote: post #46

Earlier quoted context omitted.

at this point we're _good data_ limited, which has little to do with scraping.

Why kind of data that isn’t public would be so valuable for AI training? Seems like there’s a fuck ton. All of Wikipedia, GitHub for code, etc. I can understand targeting certain sites like Reddit, etc. but not random websites

Discord I guess would be quite valuable, even the de facto public servers.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#267

Post by zzo38computer »

voidUpdate wrote: post #222
delichon wrote: post #217

Earlier quoted context omitted.

Yes, please explain. How does an entry in robots.txt distinguish humans from bots that ignore it?

When was the last time you looked at robots.txt to find a page that wasn't linked anywhere else?

It was a while ago, and it was not deliberate (wget downloaded robots.txt as well as the files I requested, and I was able to find many other files due to that, some of which could not be accessed due to requiring a password, but some were interesting (although I did not use wget to copy those other files; I only wanted to copy the files I originally requested)).

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#268

Post by aorth »

sokoloff wrote: post #201
TonyTrapp wrote: post #196

Earlier quoted context omitted.

We're affected by this. The only thing that would realistically work is the first suggestion. The most unscrupulous AI crawlers distribute their inhuman request rate over dozens of IPs, so every IP just makes 1-2 requests in total. And they use real-world browser user agents, so blocking those could lock out real users. However, sometimes they claim to be using really old Chrome versions, so I feel less bad about loc…

> dozens of IPs, so every IP just makes 1-2 requests in total Dozens of IPs making 1-2 requests per IP hardly seems like something to spend time worrying about.

Parent probably meant hundreds or thousands of IPs.

Last week I had a web server with a high load. After some log analysis I found 66,000 unique IPs from residential ISPs in Brazil had made requests to the server in a few hours. I have broad rate limits on data center ISPs, but this kinda shocked me. Botnet? News coverage of the site in Brazil? No clue.

Edit: LOL didn't read the article unity after posting—they mention the Fedora Pagure server getting this traffic from Brazil last week too!

Rate limiting vast swathes of Google Cloud, Amazon EC2, Digital Ocean, Hetzner, Huawei, Alibaba, Tencent, and a dozen other data center ISPs by subnet has really helped keep the load on my web servers down.

Last year I had one incident with 14,000 unique IPs in Amazon Singapore making requests in one day. What the hell is that?

I don't even bother trusting user agents any more. My nginx config has gotten too complex over the years and I wish I didn't need all this arcane mapping and whatnot.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#270

Post by PufPufPuf »

Figs wrote: post #147
xena wrote: post #4

Wow it is so surreal to see a project of mine on Ars Technica! It's such an honor!

Hmm. Instead of requiring JS on the client, why don't you add a delay on the server side (e.g. 1 second default, adjustable by server admin) for requests that don't have a session cookie? For each session keep a counter and a timestamp. Every time you get a request from a session, look up the tracked entry, increment the counter (or initialize it if not found) and update the timestamp. If the counter is greater than…

In a "denial of service prevention" scenario, you need your cost to be lower than the cost of the attacker. "Delay on the server side" means keeping a TCP connection open for that long, and that's a limited resource.
Post reply on HN