Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

411–420 of 694 posts

Re: Creepy Crawlies

#411

Post by emsign »

TonyTrapp wrote: post #264
mzajc wrote: post #201

> Why is git.kernel.org “interesting” to crawlers I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests. The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this mea…

Exactly my observation as well. They devour absolutely everything, no exceptions. No matter how stupid it might be to digest a source code repository via HTTP. They probably don't even recognize what's inside those pages and that there's an easier way to obtain the same result.

Makes me wonder how much garbage they actually collect across the web. That can't be good for the quality of the LLM.

Re: Creepy Crawlies

#412

Post by sgsjchs »

bjourne wrote: post #370

> They still do that — welcome to the wonderful world of “proxy SDK monetization.” It's big business, and your TV is probably doing it. I must be missing something. How can using peoples' TVs as bot farms be even remotely legal? Especially when the purpose is to avoid IP blocks?

The people "consented" to this when they clicked OK on the user agreement.

Re: Creepy Crawlies

#413

Post by emsign »

inigyou wrote: post #240
atq2119 wrote: post #168

Earlier quoted context omitted.

It feels like this progression of increasingly drastic measures to circumvent the protections of a computer system ought to be enough to establish criminal intent and get some of the people running those crawlers into prison.

It does. It's literally a felony but for some reason not a single person has pressed charges.

It's because the companies doing this bought the current government.

Re: Creepy Crawlies

#415

Post by slipknotfan »

sgsjchs wrote: post #410

Ironically, defense by obscurity may be the way to go here. Fork Anubis. Slightly modify the hash function it computes. Deploy. Do not try to make your fork widely adopted. Do not even publish it. You've just defeated ASICs and any craweler that's special-cased Anubis (currently all of them). If enough people do this, the only recourse they will have is either genuinely executing served js code like a real user or bu…

> Fork Anubis. Slightly modify the hash function it computes. Deploy. Do not try to make your fork widely adopted. Do not even publish it.

A more robust solution would be to keep a few patches handy with different versions of the algorithm, and rotate which one is in use. This would keep the crawlers on their toes if they wise up to the changed algorithm. One could even imagine automatically rotating witch algorithm to use on a weekly basis.

Re: Creepy Crawlies

#416

Post by Self-Perfection »

bayindirh wrote: post #227
Self-Perfection wrote: post #116

Earlier quoted context omitted.

The better solution would be fix Anubis and optimise its calculations on iPhone. Maybe it needs WebAssembly hashing implementation instead of JS? But it would just make Anubis suck a little bit less Anubis "protection" model is wrong/broken on several levels. I suppose it does not even have ability to bypass hashing screen when server load is low.

> I suppose it does not even have ability to bypass hashing screen when server load is low. Anubis is not a load optimizer. It's sole purpose is to deny the bots you don't want on your site. It's on the same category as Iocaine and Go-Away, even if what they do is different in some cases.

>It's sole purpose is to deny the bots you don't want on your site.

This is fundamentally impossible.

In the end only pay per crawl will work. Like x402 but maybe something similar.

In other words: not only client has to spend some kind of resource for access, but it has to be a transaction, server must get something from serving the request.

Re: Creepy Crawlies

#417

Post by semiquaver »

dspillett wrote: post #223
semiquaver wrote: post #126

Earlier quoted context omitted.

OK, but that doesn't change anything. You have a large pool of IPs, each of which only needs to expend a handful of extra milliseconds of work to get unlimited access to the protected resource. Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a…

> In other words not a meaningful barrier. And yet many report it works, at least for now, and the excess load due to scraping activity falls precipitously when Anubis or similar solutions are used. Maybe once this sort of challenge is used almost everywhere we'll see concerted attempts to get around it, but for now it is easier for them to just move on to another target. > a $5000 ASIC-based device can hash SHA256 a…

You don’t need to spend $5000 to obtain the hash rate of a $5000 device on a rental basis. You may have heard of this thing called “the cloud”. Obtaining very high hash rates is effectively free, largely as a side effect of the crypto bust.

Not sure why anyone would characterize these scrapers im general as all being fly-by-night operations that don’t have two cents to scrape together.

Re: Creepy Crawlies

#418

Post by lmz »

It's funny how some people say "AI bad, datacenters waste energy" then other people say "AI bad, going to make humans and their phones waste energy".

Re: Creepy Crawlies

#419

Post by physicalecon »

embedding-shape wrote: post #337
wongarsu wrote: post #100

Earlier quoted context omitted.

Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them Fo…

> which takes ~180sec for my iPhone 17 to solve at ~100KH/s > so I really don't see why Safari on a modern iPhone should be so slow at this FWIW, my iPhone 12 Mini also does ~110KH/s with Anubis on lists.ffmpeg.org, so seems fairly likely that Safari somehow here isn't working as expected.

iPhone Air is roughly 90KH/s. I bet it’s a matter of the site not being “trusted” and the JIT being disabled.

Re: Creepy Crawlies

#420

Post by darkmighty »

tptacek wrote: post #335
xena wrote: post #325

Earlier quoted context omitted.

The WebAssembly challenge (as of yet unreleased, it's been in dev for a year and I don't want to enable it by default until I know it won't blow up horribly) is bit-wise instead of nibble-wise, meaning that admins have better challenge granularity: https://anubis.techaro.lol/docs/admin/configuration/challeng...

What I'm hearing is that you spent a year designing a more complicated proof-of-work challenge rather than moving to a challenge system that actually impacts scrapers. Consider revisiting this decision!

> rather than moving to a challenge system that actually impacts scrapers

At this point isn't it basically auth-only?

Rant: (genuinely wondering too, and RFC, request for conversation) at this point don't we have Google, etc. basically doing Real World ID Verification, but without an open protocol backing, using it to corral users into their ecosystem and gather data, and leaving us without some open and distributed alternative like the (fairly well working it seems to me) Public Key infrastructure? (PKI)

I know Sam Altman (with Proof of Human or w/e) and others have suggested and also have their own alternative (which I think is basically what Google already does), but I think this should simply be built based on an open protocol and then hopefully not-too-large (and with a healthy competition) local companies could do it for you.

Post reply on HN