Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

101–110 of 686 posts

Re: Creepy Crawlies

#101

Post by semiquaver »

smallerize wrote: post #92
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

But the scraper is making way more requests and is paying for all that compute.

no, a single solve returns a token which entitles the scraper to access the site as much as it wants for one week.

Re: Creepy Crawlies

#102

Post by Symbiote »

rcxdude wrote: post #40
Velocifyer wrote: post #18

But why don't they just git clone?

These crawlers (in contrast to e.g. googlebot and similar better behaved crawlers) are not very smart: they seem to make very little effort to avoiding crawling useless deep trees of generated pages. About the only thing they seem to put a lot of effort into is avoiding blocking. (I'll note that while these are generally attributed to AI data gathering because of the timing of when they took off, it's not actually ob…

Meta is an exception. They intentionally ignore robots.txt rules.

(No second chances, I blocked their whole IP range.)

Re: Creepy Crawlies

#103

Post by marginalia_nu »

dunder_cat wrote: post #24
nicman23 wrote: post #16

couldn't you have anubis on a dynamic difficulty? ie if a ip requests more than 1k pages per day +1 the difficulty ?

Yes, but the article (not to call you out - I just think it's a very important point!) points out that this type of throttling would not be effective: > Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the tim…

I've had a fair bit of success with increasing the bot mitigation based on a global rate limit. During periods of high request rates, I throw progressively more hurdles at the bots, and during periods of low request rates I disable them all.

Re: Creepy Crawlies

#104

Post by TZubiri »

Same problem we've been having for ages.

Using shared ip banlists is the best solution so far, like cloudflare. Sure maybe they hit your server for 5 seconds and then desist, but they'll attack someone else, and they'll eventually rotate.

I'm not sure if Anubis has a feature for centralized banlists, but I'm assuming since it's OS and privacy oriented, there isn't.

There's a tradeoff between privacy and abuse, you want privacy? You get abuse, you want to battle abuse? Gotta sacrifice privacy. Worth noting that unmarked vpn users (residential proxy or residential vpn users) use these proxies for privacy, and therefore give a reasonable alibi to abusers.

Re: Creepy Crawlies

#105

Post by ninglor »

znnajdla wrote: post #58

Put a CDN in front and let them absorb the load? Seriously, this is static content, which is so cheap to serve it should be free.

This is not meaningfully static content. Look at the charts in TFA. There is a combinatorial explosion of distinct URLs which the crawlers can and do request.

Re: Creepy Crawlies

#106

Post by TZubiri »

semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

I thought we learned this with Litecoin, it works only as a novelty factor, nothing fundamental.

When the reward to abuse crosses a certain threshold, a determined party will come along and completely dominate it.

Re: Creepy Crawlies

#107

Post by marginalia_nu »

FWIW, git hosts have always interacted very poorly with crawlers, to the point where you have to actively code in git host detection to avoid getting stuck in an accidental crawler trap if you want to run a well behaved crawler. Easiest is just to look for anything that looks like a commit hash in a path and drop those URLs from the crawl frontier.

Reason they interact so poorly is that is that git hosts generate a lot of links. One for each file in each commit, and a diff for each file appearing in a pair of commits. Even a small repo can have millions of viable links, and most of these are stupidly expensive to render for the git host. On top of this crawlers generally don't have a very deep understanding of what they are crawling, and can't meaningfully distinguish computationally expensive requests from cheap ones.

Re: Creepy Crawlies

#108

Post by DarmokTanagra »

lkbm wrote: post #15

> So, you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right? Clone the repos, walk every commit. Done. If repos like this were 10%+ of what they crawled, having a special case for clone-able repos would be smart, but if you're crawling everything, you're not going to do an efficiency tweak for each special case that has…

Most of these crawlers are probably AI generated in the first place, and AI generated code is garbage.

Thats a good enough explanation in my opinion.

Source: someone who reviews AI generated PR's on a daily basis

Re: Creepy Crawlies

#109

Post by TZubiri »

jdnier wrote: post #72

I really enjoyed the writing style in this article. And the bot progression from "alter user agent" to "change IP addresses" to providers having to ban whole subnets, whole ASNs, and realizing "proxy SDK monetization" is a thing mirrors threat actor progression from the time before LLMs.

It goes beyond mirrors, it's just something criminals have been doing since forever, to abuse all websites.

Re: Creepy Crawlies

#110

Post by NobodyNada »

semiquaver wrote: post #101
smallerize wrote: post #92

Earlier quoted context omitted.

But the scraper is making way more requests and is paying for all that compute.

no, a single solve returns a token which entitles the scraper to access the site as much as it wants for one week.

The token should only be valid for one IP, so the scraper has to keep solving challenges if it rotates IPs.
Post reply on HN