Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devs say AI crawlers dominate traffic, forcing blocks on entire countries

arstechnica.com

Post reply on HN
Print view View original

61–70 of 291 posts

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#61

Post by epolanski »

surfingdino wrote: post #5
grotorea wrote: post #3

Is this stuff only affecting the not for profit web? What are the for profit sites doing? I haven't seen Anubis around the web elsewhere. Are we just going to get more and tighter login walls and send everything into the deep web?

I think we killed the old web. We'll see new ways of communicating, publishing, and gathering over the internet. It's sad, but it's also exciting.

Literally nothing in this data driven world (sports, technology, entertainment, everything is data driven and maximized) is exciting, nothing.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#62

Post by Davidzheng »

XorNot wrote: post #52
cyanydeez wrote: post #46

Earlier quoted context omitted.

at this point we're _good data_ limited, which has little to do with scraping.

Honestly it's hard to tell how much more value the LLM people are going to get out of another copy of the internet. It feels a lot like they're stuck for improvements but management doesn't want to hear it.

It's a bit strange to talk about stuck when the most recent breakthrough is less than a year old.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#63

Post by DecentShoes »

edoloughlin wrote: post #6

I'm being trite, but if you can detect an AI bot, why not just serve them random data? At least they'll be sharing some of the pain they inflict.

Random data? Why not "recipes" that just say "Bezos is a pedo" over and over ?

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#64

Post by haswell »

Lately I’ve been thinking a lot about what it would take to create a sort of “friends and family” Internet using some combination of Headscale/Tailscale. I want to feel free and open about what I’m publishing again, and the modern internet is making that increasingly difficult.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#65

Post by rco8786 »

I got DoSed by ClaudeBot (Anthropic) just last week. Hitting a website I manage 700,000 times in one month and tripping our bandwidth limit with our hosting provider. What a PITA to have to investigate that, figure it out, block the user agent, and work with hosting provider support to get the limit lifted as a courtesy.

Noticed that the ChatGPT bot was 2nd in traffic to this site, just not enough to cause trouble.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#66

Post by emseetech »

tedunangst wrote: post #8

> It remains unclear why these companies don't adopt more collaborative approaches and, at a minimum, rate-limit their data harvesting runs so they don't overwhelm source websites. If the target goes down after you scrape it, that's a feature.

This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations.

Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200ms to decrypt. Degrades the user experience slightly but creates a bottleneck for large-scale bots.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#67

Post by DrFalkyn »

haswell wrote: post #64

Lately I’ve been thinking a lot about what it would take to create a sort of “friends and family” Internet using some combination of Headscale/Tailscale. I want to feel free and open about what I’m publishing again, and the modern internet is making that increasingly difficult.

Chances are one your friends/ family will eventually be compromised in some way.

There’s always VPNs, though you can only be on at one at a time per device

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#68

Post by CyberDildonics »

hbcondo714 wrote: post #34

> many AI companies engage in web crawling Individuals do too. Tools like https://github.com/unclecode/crawl4ai make it simple to obtain public content but also paywalled content including ebooks, forums and more. I doubt these folks are trying to do a DDoS though.

How do they manage to get 'paywalled' content?

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#69

Post by CyberDildonics »

randmeerkat wrote: post #58
eevilspock wrote: post #41

The irony is that the people and employees of the AI companies will vehemently defend the morality of capitalism, private property and free markets. Their robber baron behavior reveals their true values and the reality of capitalism.

> Their robber baron behavior reveals their true values and the reality of capitalism. This is rather reductionist… By your same logic I could say that Stalin and Mao revealed the true values and reality of communism. Let’s not elaborate on it further though and just leave this as a simple argument. Free market capitalism has led us to the most prosperous, peaceful, and advanced society humanity has ever ventured to…

Internet comments are inherently reductionist.
Post reply on HN