Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devs say AI crawlers dominate traffic, forcing blocks on entire countries

arstechnica.com

Post reply on HN
Print view View original

221–230 of 291 posts

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#221

Post by BiteCode_dev »

everdrive wrote: post #204

One more reason we're moving away from privacy. Didn't load all the javascript domains? You're probably a bot. Not signed in? You're probably a bot. The web we knew is dying step by step. One interesting thought: do we know if these AI crawlers intentionally avoid certain topics? Is pornography totally left unscathed by these bots? How about extreme political opinions?

That's an interesting thing: put things about all dictators known currently in power that they don't want to hear, and maybe they will back off.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#222

Post by voidUpdate »

delichon wrote: post #217
soco wrote: post #215

Earlier quoted context omitted.

But the very comment you answered explains how to do it: a page forbidden in robots.txt. Does this method need explanation why it's ideal for sorting humans and google, from malicious crawlers?

Yes, please explain. How does an entry in robots.txt distinguish humans from bots that ignore it?

When was the last time you looked at robots.txt to find a page that wasn't linked anywhere else?

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#223

Post by BiteCode_dev »

EVa5I7bHFq9mnYK wrote: post #160

Why can't Cloudflare do it? They know in real time which IPs spew millions of scrape requests to various sites, so they can classify them as AI bots and allow site owners to block them.

Some people have issues with the idea that the entire web will eventually depend of a single point of failure and private for provide entity like cloudflare.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#224

Post by banq »

I have blocked these ip from the country: 190.0.0.0/8 207.248.0.0/16 177.0.0.0/8 200.0.0.0/8 201.0.0.0/8 145.0.0.0/8 168.0.0.0/8 187.0.0.0/8 186.0.0.0/8 45.0.0.0/8 131.0.0.0/16 191.0.0.0/8 160.238.0.0/16 179.0.0.0/8 186.192.0.0/10 187.0.0.0/8 189.0.0.0/8

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#225

Post by j-bos »

xyzal wrote: post #157

I think the proper answer is to aim for the bots to get _negative_ utility value from visiting our sites, that is poisoning their well, not just zero value, that is to block them. Did you try to GET a canary page forbidden in robots.txt? Very well, have a bucket load of articles on the benefits of drinking bleach. Is your user-agent too suspicious? No problem, feel free to scrape my insecure code (google "emergent mi…

This makes tons of sense, the AI trainers spend endless resources aligning their llms, the least we could do ia spend a few minutes aligning their owners. Fixing things at the incentive level.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#226

Post by swiftcoder »

Ma8ee wrote: post #199
01HNNWZ0MV43FF wrote: post #149

Earlier quoted context omitted.

Adding delay means you have to keep more connections open at a single time. Parallelism doesn't favor a server if your problem is already a small server getting hit by a big scraper

How expensive is it to just keep a connection open?

About 20 kilobytes of socket + TLS state, if you've really optimised it down to the minimum. Most server software isn't that lean, of course, so pick a framework designed for running a million or so concurrent connections on a single server (i.e. something like Nginx)

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#227

Post by zkmon »

An opensource repo asking who is responsible for this AI invasions? Well, it is you, who is responsible for all this. What did you think when you helped tech to advance so rapidly, over-pacing the needs of humans? Read about panchatantra story of 4 brothers who got a dead tiger alive, just to boast of their skill and greatness.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#228

Post by rustc »

hec126 wrote: post #208
sigmoid10 wrote: post #193

Earlier quoted context omitted.

This relies a lot on being able to detect bots. Everything you said could be easily bypassed with a small to moderate amount of effort on the side of crawler's creators. Distinguishing genuine traffic has always been hard and it will not get easier in the age of AI.

You can sprinkle your site with almost-invisible hyperlinks. Bots will follow, humans will not.

This would be terrible for accessibility for users using a screen reader.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#229

Post by sharlos201068 »

voidUpdate wrote: post #222
delichon wrote: post #217

Earlier quoted context omitted.

Yes, please explain. How does an entry in robots.txt distinguish humans from bots that ignore it?

When was the last time you looked at robots.txt to find a page that wasn't linked anywhere else?

Crawlers aren't interested in fake pages that aren't linked to anywhere, they're crawling the same pages your users are viewing.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#230

Post by brookst »

voidUpdate wrote: post #222
delichon wrote: post #217

Earlier quoted context omitted.

Yes, please explain. How does an entry in robots.txt distinguish humans from bots that ignore it?

When was the last time you looked at robots.txt to find a page that wasn't linked anywhere else?

robots.txt is not a sitemap. If it worked that way you could just make a 5TB file linking to a billion pages that look like static links but are dynamically generated.
Post reply on HN