Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devs say AI crawlers dominate traffic, forcing blocks on entire countries

arstechnica.com

Post reply on HN
Print view View original

181–190 of 291 posts

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#181

Post by Terr_ »

userbinator wrote: post #124
Terr_ wrote: post #118

Earlier quoted context omitted.

Or require every fresh "unique" visitor to run some JS that takes X seconds to compute. It's not nice for visitors using a very old smartphone, but it's arguably less-exclusionary than some of the tests and third-party gatekeepers that exist now. In many cases we don't actually care about telling if someone is truly a human alone, as much as ensuring that they aren't a throwaway sockpuppet of a larger automated syste…

that takes X seconds to compute. Those who have the computing resources to do commercial scraping will easily get past that. In contrast, there are still many questions which a human can easily answer, but even the best LLMs currently can't.

It doesn't have to be bulletproof, it just has to create a cost that doesn't scale economically for them.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#184

Post by seper8 »

xyzal wrote: post #157

I think the proper answer is to aim for the bots to get _negative_ utility value from visiting our sites, that is poisoning their well, not just zero value, that is to block them. Did you try to GET a canary page forbidden in robots.txt? Very well, have a bucket load of articles on the benefits of drinking bleach. Is your user-agent too suspicious? No problem, feel free to scrape my insecure code (google "emergent mi…

I love this idea haha

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#185

Post by usrnm »

xyzal wrote: post #157

I think the proper answer is to aim for the bots to get _negative_ utility value from visiting our sites, that is poisoning their well, not just zero value, that is to block them. Did you try to GET a canary page forbidden in robots.txt? Very well, have a bucket load of articles on the benefits of drinking bleach. Is your user-agent too suspicious? No problem, feel free to scrape my insecure code (google "emergent mi…

Where are you going to get all that content? If it's static, it will get filtered out very fast, if it's dynamic and autogenerated, it might cost even more than just letting the crawler through

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#186

Post by imtringued »

dns_snek wrote: post #161
williamdclt wrote: post #158

Earlier quoted context omitted.

I’m no anime fan, but this sort of judgement on normalcy leaves me with a very sour impression about whoever says this sort of thing.

[flagged]

You might say that, but claiming everyone is a pedophile is such a tired political play at this point. Its primary purpose is to dehumanize people so that blatant wrongdoing can be justified.

The visceral reaction might be genuine, but the actual feelings are probably not. I have yet to see someone who actually "cares about the children". The vast majority of accusations e.g. democrats running a pedophile ring turn out to be completely manufactured. The democrats responded by giving more funding and starting projects combating child abuse, only for the republicans to gut the programs, who think they are a waste of tax payer money and an example of big government.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#187

Post by viraptor »

Figs wrote: post #147
xena wrote: post #4

Wow it is so surreal to see a project of mine on Ars Technica! It's such an honor!

Hmm. Instead of requiring JS on the client, why don't you add a delay on the server side (e.g. 1 second default, adjustable by server admin) for requests that don't have a session cookie? For each session keep a counter and a timestamp. Every time you get a request from a session, look up the tracked entry, increment the counter (or initialize it if not found) and update the timestamp. If the counter is greater than…

> look up the tracked entry, increment the counter (or initialize it if not found) and update the timestamp

Not sure about author's motivation, but this part is why I don't track usage - PoW allows you to do everything statelessly and not keep any centralised database or write any data. The benefit of a system slowing down crawling should be minimal resource usage for the server.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#188

Post by puchatek »

emseetech wrote: post #66
tedunangst wrote: post #8

> It remains unclear why these companies don't adopt more collaborative approaches and, at a minimum, rate-limit their data harvesting runs so they don't overwhelm source websites. If the target goes down after you scrape it, that's a feature.

This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations. Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200…

If we are able to detect AI scrapers then I would welcome a more strategic solution: feed them garbage data instead of the real content. If enough sites did that then the inference quality would take a hit and eventually the perpetrators, too.

But of course this is the more expensive option that can't really be asked of sites that already provide public services (even if those are paid for by ads).

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#189

Post by puchatek »

keyle wrote: post #138
rfurmani wrote: post #121

After I opened up https://sugaku.net to be usable without login, it was astounding how quickly the crawlers started. I'd like the site to be accessible to all, but I've had to restrict most of the dynamic features to logged in users, restrict robots.txt, use cloudflare to block AI crawlers and bad bots, and I'm still getting ~1M automated requests per day (compared to ~1K organic), so I think I'll need to restrict th…

Has someone made honeypot for AI yet? Take all regular papers and change their words or keywords to something outrageous and watch it feed it to users.

If there was a non-profit dedicated do this, I would donate
Post reply on HN