Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devs say AI crawlers dominate traffic, forcing blocks on entire countries

arstechnica.com

Post reply on HN
Print view View original

211–220 of 291 posts

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#211

Post by TonyTrapp »

sokoloff wrote: post #201
TonyTrapp wrote: post #196

Earlier quoted context omitted.

We're affected by this. The only thing that would realistically work is the first suggestion. The most unscrupulous AI crawlers distribute their inhuman request rate over dozens of IPs, so every IP just makes 1-2 requests in total. And they use real-world browser user agents, so blocking those could lock out real users. However, sometimes they claim to be using really old Chrome versions, so I feel less bad about loc…

> dozens of IPs, so every IP just makes 1-2 requests in total Dozens of IPs making 1-2 requests per IP hardly seems like something to spend time worrying about.

If you are serving a Git repository browser and all of those IPs are hitting all the expensive endpoints such as git blame, it becomes something to worry about very quickly.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#212

Post by gchamonlive »

xyzal wrote: post #157

I think the proper answer is to aim for the bots to get _negative_ utility value from visiting our sites, that is poisoning their well, not just zero value, that is to block them. Did you try to GET a canary page forbidden in robots.txt? Very well, have a bucket load of articles on the benefits of drinking bleach. Is your user-agent too suspicious? No problem, feel free to scrape my insecure code (google "emergent mi…

If I'm hosting my site independently, with a rented machine and a cloudflare CDN hosting my code on a self managed gitlab instance, how should I go about implementing this? Is there something plug and play I can drop into nginx that would do this work for me of serving bogus content and leaving my gitlab instance unscathed by bots?

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#213

Post by CaptainFever »

WesolyKubeczek wrote: post #165
CaptainFever wrote: post #109

Earlier quoted context omitted.

Close, robots.txt was originally for web crawlers, to reduce accidental denial-of-service attacks. It had nothing to do with the scraping (i.e. downloading content and parsing the HTML tags in a programmatic manner).

What do you think a search engine’s crawler bot is doing exactly? I could sure be wrong, but I have a hunch that “downloading content and paraing the HTML tags in a programmatic manner” describes it.

Yes, but the difference is that the term "scraping" also targets things like automatically generating RSS feeds from HTML pages, which is not covered by robots.txt.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#215

Post by soco »

sigmoid10 wrote: post #193
xyzal wrote: post #157

I think the proper answer is to aim for the bots to get _negative_ utility value from visiting our sites, that is poisoning their well, not just zero value, that is to block them. Did you try to GET a canary page forbidden in robots.txt? Very well, have a bucket load of articles on the benefits of drinking bleach. Is your user-agent too suspicious? No problem, feel free to scrape my insecure code (google "emergent mi…

This relies a lot on being able to detect bots. Everything you said could be easily bypassed with a small to moderate amount of effort on the side of crawler's creators. Distinguishing genuine traffic has always been hard and it will not get easier in the age of AI.

But the very comment you answered explains how to do it: a page forbidden in robots.txt. Does this method need explanation why it's ideal for sorting humans and google, from malicious crawlers?

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#216

Post by soulofmischief »

bakugo wrote: post #168
williamdclt wrote: post #158

Earlier quoted context omitted.

I’m no anime fan, but this sort of judgement on normalcy leaves me with a very sour impression about whoever says this sort of thing.

[flagged]

[flagged]

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#217

Post by delichon »

soco wrote: post #215
sigmoid10 wrote: post #193

Earlier quoted context omitted.

This relies a lot on being able to detect bots. Everything you said could be easily bypassed with a small to moderate amount of effort on the side of crawler's creators. Distinguishing genuine traffic has always been hard and it will not get easier in the age of AI.

But the very comment you answered explains how to do it: a page forbidden in robots.txt. Does this method need explanation why it's ideal for sorting humans and google, from malicious crawlers?

Yes, please explain. How does an entry in robots.txt distinguish humans from bots that ignore it?

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#218

Post by kh_hk »

pdw wrote: post #214

I wonder how long until we'll see DNS blocklists to blackhole IP addresses associated with scrapers. It seems like the logical evolution.

There are ip blocklist. Ranging from free, to paid, from text files to apis. Some of the sites offering IP blacklists also need to protect themselves from automated crawls too. It all goes full circle

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#220

Post by kh_hk »

Proof of work is sufficient (although easy to bypass on targeted crawls) for protecting endpoints that are accessed via browsers, but plain public APIs have to resort to other more primitive methods like rate limiting.

Blocking by UA is stupid, an by country kind of wrong. I am currently exploring ja4 fingerprints, that together with other metrics (country, Arn, block list), might give me a good tool to stop malicious usage.

My point is, this is a lot of work, and it takes time off the budget you give to side projects.

Post reply on HN