Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devs say AI crawlers dominate traffic, forcing blocks on entire countries

arstechnica.com

Post reply on HN
Print view View original

51–60 of 291 posts

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#51

Post by cyanydeez »

j45 wrote: post #38

Surprised at how crawling as a whole seems to have taken a sustained step backwards with old best practices that are solved being new to devs today? I can't help but wonder if big AI crawlers belonging to the LLMs wouldn't be doing some amount of local caching with Squid or something. Maybe it's beneficial somehow to let the websites tarpit them or slow down requests to use more tokens.

I'm guessing its the rise of the AI agents that just go out and download "research" and not necessarily building LLMs.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#52

Post by XorNot »

cyanydeez wrote: post #46
randmeerkat wrote: post #20

Earlier quoted context omitted.

> Why? What is the goal of a scraper, and how does disabling the source of the data benefit them? The next scraper doesn’t get the data. People don’t realize we’re not compute limited for ai, we’re data limited. What we’re watching is the “data war”.

at this point we're _good data_ limited, which has little to do with scraping.

Honestly it's hard to tell how much more value the LLM people are going to get out of another copy of the internet.

It feels a lot like they're stuck for improvements but management doesn't want to hear it.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#53

Post by xena »

charcircuit wrote: post #16
noirscape wrote: post #9

Earlier quoted context omitted.

Bandwidth isn't free, not at the volume these crawlers scrape at; serving them random data (for example by leading them down an endless tarpit of links that no human would end up visiting) would still incur bandwidth fees. Also it's not identifiable AI bot traffic that's detected (they mask themselves as regular browsers and hop between domestic IP addresses when blocked), it's just really obviously AI scraper traffi…

>Bandwidth isn't free Via peering agreements it is.

What button do I click in the AWS panel for that?

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#55

Post by misnome »

egypturnash wrote: post #31

This is 100% off-topic but: Is it just me or does Ars keep on serving videos about the sound design of Callisto Protocol in the middle of everything? Why do they keep on promoting these videos about a game from 2022? They've been doing this for months now.

I also get this for 1+ years, and just assume that it’s a symptom of my adblockers/pihole working and screwing up something in their auto ad targeting.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#57

Post by alabastervlog »

noirscape wrote: post #10
ggm wrote: post #2

Entire country blocks are lazy, and pragmatic. The US armed forces at one point blocked AU/NZ on 202/8 and 203/8 on a misunderstanding about packets from China, also from these blocks. Not so useful for military staff seconded into the region seeking to use public internet to get back to base. People need to find better methods. And, crawlers need to pay a stupidity tax or be regulated (dirty word in the tech sector)

They can absolutely work if you aren't expecting any traffic from those countries whatsoever. I don't expect any international calls... ever, so I block international calling numbers on my phone (since they are always spam calls) and it cuts down on the overwhelming majority of them. Don't see why that couldn't apply to websites either.

Country or region blocks based on IPs used to (c. ~2000) be pretty standard. Blackhole the blocks associated with China, Russia, and maybe Africa, and your failed-login logs drop from scrolling so fast you can't read them, to a handful of lines per minute. Almost all the traffic was from those blocks, and was malicious. Meanwhile, for many sites, your odds (especially back then) of getting legitimate traffic from, say, China, was nearly zero, so the cost of blocking them was effectively nothing.

Cloudflare is basically still just this, but with more steps.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#58

Post by randmeerkat »

eevilspock wrote: post #41

The irony is that the people and employees of the AI companies will vehemently defend the morality of capitalism, private property and free markets. Their robber baron behavior reveals their true values and the reality of capitalism.

> Their robber baron behavior reveals their true values and the reality of capitalism.

This is rather reductionist… By your same logic I could say that Stalin and Mao revealed the true values and reality of communism.

Let’s not elaborate on it further though and just leave this as a simple argument. Free market capitalism has led us to the most prosperous, peaceful, and advanced society humanity has ever ventured to create. Communism threatened that prosperity and peace with atrocities on a scale that exists beyond human comprehension. Capitalism, even with all of its faults, is the obvious choice.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#59

Post by charcircuit »

xena wrote: post #53
charcircuit wrote: post #16

Earlier quoted context omitted.

>Bandwidth isn't free Via peering agreements it is.

What button do I click in the AWS panel for that?

There is no button. AWS is where you go to light money on fire.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#60

Post by charcircuit »

rcxdude wrote: post #28
charcircuit wrote: post #16

Earlier quoted context omitted.

>Bandwidth isn't free Via peering agreements it is.

Not something available to smaller sites

Yes, it is. They transitively get it via the agreements the smaller site's host's host makes. Or via services like Cloudflare.
Post reply on HN