Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

561–570 of 717 posts

Re: Creepy Crawlies

#561

Post by rtpg »

sgsjchs wrote: post #410

Ironically, defense by obscurity may be the way to go here. Fork Anubis. Slightly modify the hash function it computes. Deploy. Do not try to make your fork widely adopted. Do not even publish it. You've just defeated ASICs and any craweler that's special-cased Anubis (currently all of them). If enough people do this, the only recourse they will have is either genuinely executing served js code like a real user or bu…

I was thinking about this. Would it be possible for Anubis to have a code gen process that would make every deployment of itself sufficiently unique such that it's "annoying" to work around?

I do wonder about the premise as well: are people special casing for Anubis?

Re: Creepy Crawlies

#562

Post by miki123211 »

paytonjjones wrote: post #157
semiquaver wrote: post #112

Earlier quoted context omitted.

> I really don't see why Safari on a modern iPhone should be so slow at this me neither, but I don't think it changes the argument. There's always going to be someone on a low-end device. Your adversaries already have superhuman coding ability and infinite patience. Why would you expect the long-term advantage to be with the defenders?

In this case, because there's a vastly more efficient economic path for the adversaries (cloning). They're not trying to engage in an arms race, they're trying to channel a racing river into its natural course.

I highly doubt the adversaries actually care about kernel.org in particular.

It's more likely that they're trying to scrape the entire internet, and cgit is just unfortunate enough to have autogenerated pages, where each page seems to contain text entirely distinct from its neighbors. This probably bypasses the scrapers' heuristics, which normally prevent them from getting trapped in a maze of twisty little autogenerated pages, all alike.

The way you normally deal with datasets at that scale is to move postprocessing (particularly expensive postprocessing) as late in the pipeline as possible.

Re: Creepy Crawlies

#563

Post by miki123211 »

paytonjjones wrote: post #203
pbronez wrote: post #193

Earlier quoted context omitted.

It’s more economical at a compute level, but not at the developer level. The moment you start customizing your crawler to use protocol X for site Y your scale story collapses.

It's a good point, but in practice it depends on how easy those customizations are to implement / maintain, and how much money and effort you save. At some point the compute cost can disrupt even the nicest scale story. I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and sc…

The temptation to offer different / inferior / limited content to scrapers will be too strong, so such solutions are doomed to fail.

This would likely work as "Cloudflare SideChannel", a (hypothetical) Cloudflare product that would let scrapers download the pages that humans actually visit, as they are added to the CF cache. It wouldn't work for the non-Cloudflare part of the internet where humans connect directly to the servers that have their content.

Re: Creepy Crawlies

#564

Post by philippta »

miki123211 wrote: post #557
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

And if you're a bot operating at scale, you can use different devices for crawling and challenge solving. Do you have an underpowered TV box that would need hours to solve difficulty 6? Do you also have a Windows box where somebody installed gta6_leak_full.iso? Great, have the TV send their challenges to the Windows box for solving. If you're operating at scale, you probably have hyper-optimized Anubis kernels, E.G.…

I think TVs are just used as a residential proxies. No remote controlled chrome is running there.

Re: Creepy Crawlies

#565

Post by miki123211 »

inigyou wrote: post #230
paytonjjones wrote: post #203

Earlier quoted context omitted.

It's a good point, but in practice it depends on how easy those customizations are to implement / maintain, and how much money and effort you save. At some point the compute cost can disrupt even the nicest scale story. I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and sc…

Why aren't we litigating against scrapers, anyway? DDoS is a felony.

Because they're all in countries who would ignore such litigation.

Re: Creepy Crawlies

#567

Post by danieltanfh95 »

lxgr wrote: post #130
nneonneo wrote: post #59

Earlier quoted context omitted.

I disagree. The kernel finds it effective - 66% of scrapers are turned away directly. The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site. If each new site they hit forces…

> 66% of scrapers are turned away directly. Until they discover this neat trick [1] and solve challenges orders of magnitudes more efficiently than legitimate users. The game theory of Anubis is not sound. It makes fundamentally less sense than Captchas, and even those have been on the way out for a while. [1] https://news.ycombinator.com/item?id=49500040

it's not just "not sound", it is outright wrong.

Re: Creepy Crawlies

#568

Post by crote »

teravor wrote: post #483
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

anubis was a joke on arrival. PoW for this purpose is plain stupid. the only feasible CAPTCHA at this moment is probably hCaptcha. they began to offer animated challenges. it's probably the best you can do, using an LLM to solve them would be a bit expensive for now.

Yes, but it actually works.

As the article mentions: Anubis blocked virtually all bots for a couple of months, and even now it is still blocking 67% of requests. Without Anubis they'd have to double the number of CPU cores, solely to serve crawler traffic!

It's an arms race. The best solution is the one you can plug in right now - even if it is stupid. hCaptcha might have a better blocking rate at the moment, but it's also $0.99/1K evals, and that "animated challenge" will be impossible to solve for visually-impaired users: for many people that cost is significantly higher than spinning up an extra server or two to feed the crawlers.

Re: Creepy Crawlies

#569

Post by dTP90pN »

Does cgit not support a .patch or .diff URL suffix? How hard could it be for AI companies to add a preference for such URLs into their models or system prompts? Wouldn't that be obvious improvement for any model?

Re: Creepy Crawlies

#570

Post by mattacular »

afdbcreid wrote: post #464
sgsjchs wrote: post #410

Ironically, defense by obscurity may be the way to go here. Fork Anubis. Slightly modify the hash function it computes. Deploy. Do not try to make your fork widely adopted. Do not even publish it. You've just defeated ASICs and any craweler that's special-cased Anubis (currently all of them). If enough people do this, the only recourse they will have is either genuinely executing served js code like a real user or bu…

It's not really defense by obscurity (the JavaScript is public), more like defense by... being different?

Snowflake defense
Post reply on HN