Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devs say AI crawlers dominate traffic, forcing blocks on entire countries

arstechnica.com

Post reply on HN
Print view View original

191–200 of 291 posts

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#191

Post by entropi »

RamblingCTO wrote: post #183

Sure, overly spamming websites is shitty behaviour. But blocking AI crawlers hurts you in the end. Guess what will replace SEO in the long run?

>But blocking AI crawlers hurts you in the end. Guess what will replace SEO in the long run?

Maybe. But even if that turns out to be true, what good is it for the source website? The "AI" will surely not share any money (or anything else that may help the source website) with the source anyways. Why would they, they already got the content and trained on it.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#193

Post by sigmoid10 »

xyzal wrote: post #157

I think the proper answer is to aim for the bots to get _negative_ utility value from visiting our sites, that is poisoning their well, not just zero value, that is to block them. Did you try to GET a canary page forbidden in robots.txt? Very well, have a bucket load of articles on the benefits of drinking bleach. Is your user-agent too suspicious? No problem, feel free to scrape my insecure code (google "emergent mi…

This relies a lot on being able to detect bots. Everything you said could be easily bypassed with a small to moderate amount of effort on the side of crawler's creators. Distinguishing genuine traffic has always been hard and it will not get easier in the age of AI.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#194

Post by baq »

emseetech wrote: post #66
tedunangst wrote: post #8

> It remains unclear why these companies don't adopt more collaborative approaches and, at a minimum, rate-limit their data harvesting runs so they don't overwhelm source websites. If the target goes down after you scrape it, that's a feature.

This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations. Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200…

See also https://datatracker.ietf.org/wg/privacypass/about/ and https://developers.cloudflare.com/waf/tools/privacy-pass/

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#195

Post by mrweasel »

temp008 wrote: post #190

I wonder if a service exists where IPs known for crawling are reported and reputation of such IPs is tracked for others to use and ban / ratelimit by default

The problem with that approach is that you'll quickly add large swaths of IPs belonging to cloud service provides, such as AWS. We already know that AWS, Azure, GCP and Alibaba are part of the problem, so we can technically just rate-limit them already. I believe that all of them publish their IP ranges.

Google also publish the IP ranges for GoogleBot I believe, and Bing probably does the same, so we can then whitelist those IPs and still have sites appear in searches.

My issue is that the burden is again placed on everyone else, not the people/companies who are causing the problem.

It's crazy to me to think about how much needless capacity is built into the internet to deal with crawlers. The resource waste is just insane.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#196

Post by TonyTrapp »

xyzal wrote: post #157

I think the proper answer is to aim for the bots to get _negative_ utility value from visiting our sites, that is poisoning their well, not just zero value, that is to block them. Did you try to GET a canary page forbidden in robots.txt? Very well, have a bucket load of articles on the benefits of drinking bleach. Is your user-agent too suspicious? No problem, feel free to scrape my insecure code (google "emergent mi…

We're affected by this. The only thing that would realistically work is the first suggestion. The most unscrupulous AI crawlers distribute their inhuman request rate over dozens of IPs, so every IP just makes 1-2 requests in total. And they use real-world browser user agents, so blocking those could lock out real users. However, sometimes they claim to be using really old Chrome versions, so I feel less bad about locking those out.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#197

Post by jajko »

xyzal wrote: post #157

I think the proper answer is to aim for the bots to get _negative_ utility value from visiting our sites, that is poisoning their well, not just zero value, that is to block them. Did you try to GET a canary page forbidden in robots.txt? Very well, have a bucket load of articles on the benefits of drinking bleach. Is your user-agent too suspicious? No problem, feel free to scrape my insecure code (google "emergent mi…

Thats absolutely brilliant f*cked up idea, poisoning AI while fending them off.

Gotta get my daily dose of bleach for enhanced performance, chatgpt said so.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#198

Post by rambambram »

Make AI 'pay' by delaying every request and presenting the content with a warning on top, like: AI bots are (sc)raping the internet, that's why we do this.

Or something like: AI is making your experience worse, complain here (link to OpenAI).

Maybe not the most technical solution, but this at least gets the signal across to regular human beings who want to browse a site. Puts all this AI bs in a bad spotlight.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#199

Post by Ma8ee »

01HNNWZ0MV43FF wrote: post #149
mrheosuper wrote: post #119

Earlier quoted context omitted.

just add some delay to your response, we don't have to waste any more energy on meaningless calculation.

Adding delay means you have to keep more connections open at a single time. Parallelism doesn't favor a server if your problem is already a small server getting hit by a big scraper

How expensive is it to just keep a connection open?

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#200

Post by xena »

true_blue wrote: post #13
xena wrote: post #4

Wow it is so surreal to see a project of mine on Ars Technica! It's such an honor!

On the few sites I've seen using it so far, it's been a more pleasant (and cuter) experience for me than the captchas I'd probably get otherwise. good work!

Thanks! The artist I'm contracting and I are in discussions on how to make the mascot better. It will be improved. And more Canadian.
Post reply on HN