Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

321–330 of 691 posts

Re: Creepy Crawlies

#321

Post by brokensegue »

DenisM wrote: post #289
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

Memory-hard hash functions maybe? Like, you must dedicate 4gb of ram to compute the function. Not a problem for a one-off, but is a problem when reading lots of pages at once. Or… the site will serve a random seed and the device must compute 4gb of pseudo-random data, then supply a value at a random server-demanded offset.

Do you want your phone to allocate 4GB to read a mailing list?

Re: Creepy Crawlies

#323

Post by riknos314 »

riknos314 wrote: post #320
calvinmorrison wrote: post #249

I love to see the 'leet kernel hackers and maintainers' struggling with basic volume. Each page load should cost you near nil. Us lowly PHP developers have been caching shit for close to twenty years. Learn how to cache your application and your cpu usage should be almost zero. In fact basically any read should cost nothing in comparison to writes.

The amount of memory required to cache all possible diffs (defined as an ordered pair of commits) would likely be in the exobytes. At current ram prices that's easily a trillion dollars of ram to run that cache lol. Git focuses on making diff calculations efficient largely because the space of possible diffs is very expensive to enumerate. The following from Claude: """ A diff between two randomly chosen commits usua…

I'm not sure if the website allows for diffs against arbitrary tree states, but if it does than the diff space is completely unbounded, and the ram demand is theoretically infinite.

Re: Creepy Crawlies

#324

Post by wiredfool »

inigyou wrote: post #260
wiredfool wrote: post #90

Earlier quoted context omitted.

Seeing the exact same thing on (somewhat high profile) open data sites I run. The crawlers get stuck in a loop requesting the dataset listing page with every. single. combination. of. facets. At essentially as fast as it can be pumped out or blocked. Had one bot super interested in a single organization to the tune of 1000 r/min for 24+ hours. Realistic rotating user agents, realistic sec-* headers, no ip address see…

Once detected, don't block them because they'll just change strategies automatically, but you can toy with them, like returning a page full of random numbers instead of real data.

I’d love to, but I’m a bit limited in what I can do from a reputational damage POV. They’re my sites, in that I’m responsible, but they aren’t something where I can return incorrect responses.

Re: Creepy Crawlies

#325

Post by xena »

semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

The WebAssembly challenge (as of yet unreleased, it's been in dev for a year and I don't want to enable it by default until I know it won't blow up horribly) is bit-wise instead of nibble-wise, meaning that admins have better challenge granularity: https://anubis.techaro.lol/docs/admin/configuration/challeng...

Re: Creepy Crawlies

#326

Post by ynniv »

inigyou wrote: post #255
ynniv wrote: post #74

this is an increasingly common situation. it goes something like: - i have a free, niche resource - it becomes too popular - i make it more efficient - now it's really popular, and people are "abusing" it - let's make them proof-of-work - ... and proof-of-work harder - but now "legitimate" users can't use it - ??? the core problem is that the average person uses a mobile device where work is expensive, and the "attac…

What's actually stopping them isn't the PoW, it's the customisation effort. If one site has Anubis nothing scrapes it. If many sites have Anubis they write counter scrapers. Today if you make a slight change to the Anubis algorithm on your site, they'll burn CPU endlessly computing hashes with the original algorithm and submitting wrong ones. The author of Anubis hates this fact and will ban you if you mention it, so…

sounds right to me. ai will shred it

Re: Creepy Crawlies

#327

Post by NavinF »

inigyou wrote: post #240
atq2119 wrote: post #168

Earlier quoted context omitted.

It feels like this progression of increasingly drastic measures to circumvent the protections of a computer system ought to be enough to establish criminal intent and get some of the people running those crawlers into prison.

It does. It's literally a felony but for some reason not a single person has pressed charges.

turns out the overlap between "people who can't configure their webserver to serve at wire speed" and "people who can get law enforcement to take them seriously" is the empty set

Re: Creepy Crawlies

#328

Post by kdowns »

I made it to a third round interview at anthropic in 2024 and they had me build a web crawler as their programming test. Part way through I started on making it respect robots.txt and I could immediately tell they were no longer interested in me.

Re: Creepy Crawlies

#329

Post by duplessitous »

> Training an LLM on content produced by the LLM gives it the equivalent of a digital prion disease, so when a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data.

Except the majority of LLM training content nowadays is synthetically generated by LLMs. I wish people would stop making this statement, I don't know why this claim persists to this day. It wasn't true two years ago and it sure isn't true now

Re: Creepy Crawlies

#330

Post by colinsane »

Waterluvian wrote: post #75

> It was immediately extremely effective — the bots just gave up. For a few months, it was bliss: bots were blocked at the perimeter and gave up, moving on to easier targets; the users were mildly annoyed but tolerated it, and the Anubis stack was easy enough to deploy everywhere. As a tech person who works with tech people, I have become extremely sensitive to this kind of bias. Is this solution actually better? Is…

> Is the CPU cost actually worse than mildly annoying everyone

> What is the cost of 20% and is that cost worth “mildly annoying” everyone?

from the articled:

> With a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic — everything else are scrapers.

this is not some "CPU use is 20% higher than baseline" situation. it seems that people still do not understand the scale of these bad actors.

Post reply on HN