Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devs say AI crawlers dominate traffic, forcing blocks on entire countries

arstechnica.com

Post reply on HN
Print view View original

81–90 of 291 posts

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#81

Post by emseetech »

tschwimmer wrote: post #77
emseetech wrote: post #66

Earlier quoted context omitted.

This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations. Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200…

Check out Anubis - it's not quite what you're suggesting but similar in concept: https://anubis.techaro.lol/

Interesting! Not how I'd approach it but certainly thinking along the same lines.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#82

Post by tedunangst »

kevin_thibedeau wrote: post #75
emseetech wrote: post #66

Earlier quoted context omitted.

This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations. Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200…

The problem is that the server also has to do the work. Fine for an infrequent auth challenge. Not so fine for every single data request.

Tons of problems are easier to verify than to solve.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#83

Post by kazinator »

I've been seeing crawlers which report an Agent string like this:

  Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3405.80 Safari/537.36
Everything but the Chrome/ is the same. They come from different IP addresses and make two hit-and-run requests. The different IPs always use a different Chrome string. Always some two digit main version like 69, 70. Then a .0. and then some funny minor and build numbers: typically a four digit minor.

When I was hit with a lot of these a couple of weeks ago, I put in a custom rewrite rule to redirect them to the honeypot.

The attack quickly abated.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#84

Post by DrFalkyn »

cyanydeez wrote: post #46
randmeerkat wrote: post #20

Earlier quoted context omitted.

> Why? What is the goal of a scraper, and how does disabling the source of the data benefit them? The next scraper doesn’t get the data. People don’t realize we’re not compute limited for ai, we’re data limited. What we’re watching is the “data war”.

at this point we're _good data_ limited, which has little to do with scraping.

Why kind of data that isn’t public would be so valuable for AI training?

Seems like there’s a fuck ton. All of Wikipedia, GitHub for code, etc.

I can understand targeting certain sites like Reddit, etc. but not random websites

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#85

Post by johnnyanmac »

eevilspock wrote: post #41

The irony is that the people and employees of the AI companies will vehemently defend the morality of capitalism, private property and free markets. Their robber baron behavior reveals their true values and the reality of capitalism.

Insert the Sinclair quote here. Anything to drive up the stock, no matter how immoral or illegal.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#86

Post by johnnyanmac »

randmeerkat wrote: post #58
eevilspock wrote: post #41

The irony is that the people and employees of the AI companies will vehemently defend the morality of capitalism, private property and free markets. Their robber baron behavior reveals their true values and the reality of capitalism.

> Their robber baron behavior reveals their true values and the reality of capitalism. This is rather reductionist… By your same logic I could say that Stalin and Mao revealed the true values and reality of communism. Let’s not elaborate on it further though and just leave this as a simple argument. Free market capitalism has led us to the most prosperous, peaceful, and advanced society humanity has ever ventured to…

It's rather strawman to bring up communism in a conversation that talked nothing about it, except that Capitalism is clearly flawed.

Capitalism without law ends up with the same kind of authoritariasm as communism without law. Some Rich Guy ends up telling everyone what to do as a ruler with loose rules that no longer resemble the economic model. That's what people complain about when they bring up terms like "late stage capitalism".

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#88

Post by userbinator »

All these JS-heavy "anti bot" measures do is further entrench the browser monopoly, making it much harder for the minority of independents, while those who pay big $$$ can still bypass them. Instead I recommend a simple HTML form that asks questions with answers that LLMs cannot yet figure out or get consistently wrong. The more related to the site's content the questions are, the better; I remember some electronics forums would have similar "skill-testing" questions on their registration forms, and while some of them may be LLM'able now, I suspect many of them are still really CAPTCHAs that only humans can solve.

IMHO the fact that this shows up at a time when the Ladybird browser is just starting to become a serious contender is suspicious.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#89

Post by userbinator »

tschwimmer wrote: post #77
emseetech wrote: post #66

Earlier quoted context omitted.

This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations. Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200…

Check out Anubis - it's not quite what you're suggesting but similar in concept: https://anubis.techaro.lol/

I just hit a site with this --- and hit the back button immediately.

Re: Devs say AI crawlers dominate traffic, forcing blocks on entire countries

#90

Post by timewizard »

emseetech wrote: post #76
koakuma-chan wrote: post #72

Earlier quoted context omitted.

Companies running those bots have more than enough resources

Nobody has unlimited resources. Everything is a cost-benefit analysis. For highly valuable information, they might throw the GDP of a small country at scraping your site. But most information isn't worth that. And there are a lot of bad actors who don't have the resources you're thinking of that are trying to compete with the big guys on a budget. This would cut them out of the equation.

Make all websites intentionally waste energy as a strategy to defeat unscrupulous operators has negative costs and marginal benefits.
Post reply on HN