Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

581–590 of 711 posts

Re: Creepy Crawlies

#581

Post by johneth »

zbentley wrote: post #450
johneth wrote: post #143

Earlier quoted context omitted.

If there was a way of detecting the proxy SDKs, it would be a simple matter of banning the device. Or at least naming and shaming the apps that use them.

I'm sure there are ways of detecting residential proxy clients that are reasonably effective. I'm equally sure that the people in possession of those techniques (thinking CDN providers, big social networks with public links, newsrooms) are unlikely to share their secret sauce in public. Doing that loses a competitive edge in cost-of-hosting, and as soon as you go public the residential proxy makers can adapt.

There's JA4 fingerprinting (and the JA4+ suite), but still not a surefire way of detecting. Plus the metadata for the signatures has to be constantly maintained and updated.

Re: Creepy Crawlies

#582

Post by oasisbob »

stickfigure wrote: post #479
oasisbob wrote: post #185

Earlier quoted context omitted.

I think you underestimate the difficulty in effectively gating user registration. If the anti-bot efforts in general don't work for other site pages, they won't work for signup functionality either.

I've been on the other side of this kind of thing (hero rather than villain, though I'm sure someone out there disagrees). You can fairly arbitrarily increase the difficulty of user registration, far beyond what users will tolerate for viewing an individual page. You can exploit this; a bot needs to make many accounts for the activity desired, and it's not hard to make "generate account" more expensive than it's wort…

I think this asymmetry is a fantasy.

For many sites, your hypothetical penny to create an account is a cost an attacker would gladly pay.

Re: Creepy Crawlies

#583

Post by crote »

jacobvosmaer wrote: post #54
easton wrote: post #37

Side note: why are shallow clones evil? I always thought they were cheaper, but I guess that’s really just for my disk space. (since the server has to compute what blobs to give you instead of just “everything”?)

Normal clones can reuse delta-compressed data the server stored on disk. Shallow clones impose a negative constraint: do not transfer data outside the requested commit depth. Pre-computed delta chains than contain unrequested data become unusable and the server must do delta compression on the fly to satsify the shallow clone.

So why not introduce a semi-shallow clone option?

If I'm doing a shallow clone it isn't because I only want to receive a specific commit, it's because I don't want to burn a giant amount of disk space and network traffic on a full history.

In most use cases it would be perfectly acceptable for the server to send additional data. The client doesn't care about it because it is meaningless to them, but if it results in a significant load reduction on the server's side they don't really mind receiving it either. A 100MB shallow checkout coming with 400MB of garbage still beats cloning an entire 5GB history!

Re: Creepy Crawlies

#584

Post by GoblinSlayer »

darkmighty wrote: post #420
tptacek wrote: post #335

Earlier quoted context omitted.

What I'm hearing is that you spent a year designing a more complicated proof-of-work challenge rather than moving to a challenge system that actually impacts scrapers. Consider revisiting this decision!

> rather than moving to a challenge system that actually impacts scrapers At this point isn't it basically auth-only? Rant: (genuinely wondering too, and RFC, request for conversation) at this point don't we have Google, etc. basically doing Real World ID Verification, but without an open protocol backing, using it to corral users into their ecosystem and gather data, and leaving us without some open and distributed…

Botnets will just borrow your id.

Re: Creepy Crawlies

#585

Post by nxobject »

trashb wrote: post #575

Yea these AI bots are getting out of hand. The article mentions that some requested pages are less likely to be legit traffic (old commits) and more likely to have bot activity. Perhaps they could increase difficulty on those pages for the proof of work. Keeping the "current" at a lower difficulty allows most normal users to use the pages as normal, while penalizing the bots. One other way I have been thinking of is…

> One other way I have been thinking of is just delay the delivery of the pages, either limit bandwidth or just wait for a bit until you deliver the page. For one user a (lets say max)3s delay on some pages is not a huge deal, however at scale that adds up and means the client can't gather other pages in the meantime

Or some kind of vintage “set up a request in a form and press a submit form, and we’ll pretend to take a while to put things together.”

Re: Creepy Crawlies

#586

Post by throwawayffffas »

Stop rendering html on your side. So simple, just add a js dependency that does the rendering on the frontend. You just serve flat files and the diffs. All the rendering happens on the users device, legitimate users will probably not even notice, bots won't notice either, your cpu usage will go way down.

It's like anubis but instead of doing useless math, you will be moving the legitimate cpu work to their side.

Re: Creepy Crawlies

#587

Post by GoblinSlayer »

crote wrote: post #568
teravor wrote: post #483

Earlier quoted context omitted.

anubis was a joke on arrival. PoW for this purpose is plain stupid. the only feasible CAPTCHA at this moment is probably hCaptcha. they began to offer animated challenges. it's probably the best you can do, using an LLM to solve them would be a bit expensive for now.

Yes, but it actually works . As the article mentions: Anubis blocked virtually all bots for a couple of months, and even now it is still blocking 67% of requests. Without Anubis they'd have to double the number of CPU cores, solely to serve crawler traffic! It's an arms race. The best solution is the one you can plug in right now - even if it is stupid. hCaptcha might have a better blocking rate at the moment, but it…

That's because those 67% of bots are the most stupid and don't run any javascript at all.

Re: Creepy Crawlies

#589

Post by GoblinSlayer »

semiquaver wrote: post #112
wongarsu wrote: post #100

Earlier quoted context omitted.

Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them Fo…

> I really don't see why Safari on a modern iPhone should be so slow at this me neither, but I don't think it changes the argument. There's always going to be someone on a low-end device. Your adversaries already have superhuman coding ability and infinite patience. Why would you expect the long-term advantage to be with the defenders?

What phone users do on git.kernel.org? In the worst case they can login (if they get to the login page).

Re: Creepy Crawlies

#590

Post by throwawayffffas »

> worth spending a ton of cycles to calculate the Anubis challenge.

Not sure how difficult Anubis is, but I would not be surprised if parsing the request through the LLM costs multiple orders of magnitude more compute than the Anubis challenge.

Post reply on HN