Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

461–470 of 691 posts

Re: Creepy Crawlies

#461

Post by fc417fc802 »

chr15m wrote: post #426
stjo wrote: post #386

Earlier quoted context omitted.

> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. Lots of work has been done to do just that by monero’s proof of work algorithm. They achieved ASIC and GPU resistance by making you generate entire programs (with quite a bit of RAM usage) and finding one that produces a desired output. Then you submit the seed used for the program (a serie…

Monero is fine with making people wait, that's literally the whole basis of mining blocks in "crypto currency". It's tuned to the time required. Anubis wants "slow for bots but fast for humans" which is not possible if somebody has a farm of devices that costs them basically zero.

> Anubis wants "slow for bots but fast for humans"

That's not true. The goal is to raise the cost for bulk scrapers to the extent possible while minimizing any inconvenience to regular users. It's not about being slow for bots but rather reducing the asymmetry of the problem (from the perspective of the server) by imposing a (minor) cost on anyone requesting a page.

Re: Creepy Crawlies

#462

Post by Shank »

physicalecon wrote: post #419
embedding-shape wrote: post #337

Earlier quoted context omitted.

> which takes ~180sec for my iPhone 17 to solve at ~100KH/s > so I really don't see why Safari on a modern iPhone should be so slow at this FWIW, my iPhone 12 Mini also does ~110KH/s with Anubis on lists.ffmpeg.org, so seems fairly likely that Safari somehow here isn't working as expected.

iPhone Air is roughly 90KH/s. I bet it’s a matter of the site not being “trusted” and the JIT being disabled.

Apple dynamically disables JIT based on website trust?

Re: Creepy Crawlies

#463

Post by fc417fc802 »

wbl wrote: post #434
applfanboysbgon wrote: post #270

Earlier quoted context omitted.

Okay. That's good enough. The problem people using Anubis have is not "I'm being targeted for a DDoS" and is actually "my site is being incidentally DDoS'd because scrapers are DDoSing the entire internet". A solution to the latter is what people need . Every single one of these threads comes up with hypotheticals about how useless Anubis is and how it can't possibly work and yet it does work for the real-world probl…

If every site does it the scrapers will adapt. The scrapers will always have a cost advantage over real users for this.

Most of the problem in this case is that the scrapers are being incredibly lazy and not investing even minimal effort in their approach. If this forces them to revisit how they go about things then mission accomplished.

Re: Creepy Crawlies

#464

Post by afdbcreid »

sgsjchs wrote: post #410

Ironically, defense by obscurity may be the way to go here. Fork Anubis. Slightly modify the hash function it computes. Deploy. Do not try to make your fork widely adopted. Do not even publish it. You've just defeated ASICs and any craweler that's special-cased Anubis (currently all of them). If enough people do this, the only recourse they will have is either genuinely executing served js code like a real user or bu…

It's not really defense by obscurity (the JavaScript is public), more like defense by... being different?

Re: Creepy Crawlies

#465

Post by zbentley »

calvinmorrison wrote: post #310
inigyou wrote: post #266

Earlier quoted context omitted.

It's running a diff between two arbitrary blobs of text. Do you actually have a solution or are you just saying to remove the feature from the site entirely? > Us PHP developers I can tell.

it's not an insult. yes. cache heavily. shitty php stacks serving trillions of dollars of ecommerce sales have managed to do it for a long time.

Those ecommerce stacks serve a large fixed number of pages. cgit does not.

Imagine if WooCommerce had a route "/product//compare/" which displayed an auto-generated comparison between any two product pages.

Now imagine running a million-SKU WooCommerce site, where each product page was 100kb of text.

Now imagine scrapers are permuting those URLs.

How would you cache that?

That's what cgit/kernel.org and many other Git forges are dealing with. These aren't static websites, even if the underlying Git repo is largely static; they're rendering arbitrary diffs and other generated-on-the-fly views into Git history. The ability to do that is a large part of the value of a Git UI.

Re: Creepy Crawlies

#466

Post by QuantumNomad_ »

Barbing wrote: post #393
inigyou wrote: post #255

Earlier quoted context omitted.

What's actually stopping them isn't the PoW, it's the customisation effort. If one site has Anubis nothing scrapes it. If many sites have Anubis they write counter scrapers. Today if you make a slight change to the Anubis algorithm on your site, they'll burn CPU endlessly computing hashes with the original algorithm and submitting wrong ones. The author of Anubis hates this fact and will ban you if you mention it, so…

Searching around for this, found "make users click the mouse three times"[1] as an anti-bot idea. Generally, makes sense that smaller site owners can make small customizations to existing anti-bot tech and see positive results until they're either (1) deemed valuable enough to receive custom attention or (2) the scrapers include LLM-based anti-antibot methods. >will ban you if you mention it Even if mentioned really…

Ask your visitors “what does it feel like being you?”

Anyone who responds with something like:

> I don’t have subjective feelings, consciousness, or an inner point of view, so there isn’t a sensation of “being me.”

is definitely an LLM.

Re: Creepy Crawlies

#467

Post by zbentley »

calvinmorrison wrote: post #349
riknos314 wrote: post #320

Earlier quoted context omitted.

The amount of memory required to cache all possible diffs (defined as an ordered pair of commits) would likely be in the exobytes. At current ram prices that's easily a trillion dollars of ram to run that cache lol. Git focuses on making diff calculations efficient largely because the space of possible diffs is very expensive to enumerate. The following from Claude: """ A diff between two randomly chosen commits usua…

cache all of them? no... but you could cache some of them, likely the newer are accessed more frequently.

Per the article, the vast majority of traffic is scrapers that are generating effectively random, permuted URLs. Even if they're already caching their popular-with-real-humans most recent data, that efficiency gain is a tiny drop in a very large bucket.

Re: Creepy Crawlies

#468

Post by TZubiri »

inigyou wrote: post #294
TZubiri wrote: post #104

Same problem we've been having for ages. Using shared ip banlists is the best solution so far, like cloudflare. Sure maybe they hit your server for 5 seconds and then desist, but they'll attack someone else, and they'll eventually rotate. I'm not sure if Anubis has a feature for centralized banlists, but I'm assuming since it's OS and privacy oriented, there isn't. There's a tradeoff between privacy and abuse, you wa…

Cloudflare doesn't block bots.

That's the whole raison d'etre for CloudFlare, it was originally a DDoS protection layer, which, as the article mentions, is the final form of malicious traffic, being distributed and hard to attribute traffic to an identity.

If you know CloudFlare as anything else, it speaks to how successfully it has grown and marketed itself into other areas.

Re: Creepy Crawlies

#469

Post by zbentley »

greatgib wrote: post #407

I have the feeling that the hate might be misplaced. For a shopping website or user generated content website, I might understand the terrible load of crawlers that are trying to "steal" the data. But for the kernel, what's the purpose? Are you that "no human" are seeing your page or its content? Maybe we should investigate more the usage being this "bots". I don't buy the explanation that there are millions LLM that…

> the C10k challenge is solved since a long time

This has nothing to do with that.

Any Node.JS application will happily accept 100K connections. They'll all wait for the under-resourced database behind it. That application "solved" the C10K challenge, but it's still overwhelmed.

> it might be possible that the "LLM agent" is requesting the commit contents to "understand" or refer or explain them. Is it a bad thing if it helps users?

The article describes random algorithmically-generated traffic arriving in batched waves from laundered residential proxy IP addresses, a few unrelated hits in a group then gone. That's not the pattern you'd see if end users were asking their agents for help.

> it is a shame that such talented people would not be able to have a proper optimization

It's mostly not static content in the sense that you're implying.

Routes that access a single commit can be cached. But most of the routes scrapers are hitting are e.g. computing diffs between arbitrary pairs of commits, or other computed-on-the-fly views into history.

I'm sure they're already caching their useful-to-real-humans data. As the article said, the vast majority of their traffic is bots hitting those arbitrary, permuted URLs. So whatever cache they're using is probably a) missed almost every time, and b) constantly getting evicted to make room for data served to bots (unless they eschew caching to avoid this--fair--and are thus back to the original issue regardless).

There is no "proper optimization" here. It's not slow to go compute the diff between a random pair of refs, render that into pretty HTML, and serve it. But it costs something more than a cache hit, and doing that dozens-to-hundreds of times a second constantly consumes resources.

Re: Creepy Crawlies

#470

Post by phyzome »

lmz wrote: post #418

It's funny how some people say "AI bad, datacenters waste energy" then other people say "AI bad, going to make humans and their phones waste energy".

This is a silly take. AI is why they're having to waste energy.
Post reply on HN