Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

81–90 of 684 posts

Re: Creepy Crawlies

#81

Post by [deleted] »

tptacek wrote: post #67
nneonneo wrote: post #59

Earlier quoted context omitted.

I disagree. The kernel finds it effective - 66% of scrapers are turned away directly. The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site. If each new site they hit forces…

It's weird to believe data center based, Internet-scale scraping operations will be less able to allocate compute to proof-of-work challenges than individual users. This is design problem with things like Anubis: proof-of-work depends on a cost asymmetry between attacker and defender. But in scraping, both legitimate users and scrapers get the same value out of a transaction.

[deleted]

Re: Creepy Crawlies

#82

Post by Lerc »

theandrewbailey wrote: post #39
petesergeant wrote: post #31

> Training an LLM on content produced by the LLM gives it the equivalent of a digital prion disease Is it foolish of me to have expected more from a blog post on kernel.org?

Are you trying to say that's bad writing? I think it's a good metaphor for a documented phenomenon: https://en.wikipedia.org/wiki/Model_collapse

As the article states, this phenomenon may be documented, but there is no consensus that it describes any practical reality.

The predicted consequences have now had time to manifest, and have not done so. This makes the claim either false or overstated. Perhaps there will be issues in the future, but to date there have been many claims that AI development will stall (for a variety of reasons). If they were the critical weaknesses they have been portrayed as, models would not have advanced to the level they are today.

If you have a hypothesis, make a clear prediction based upon it. If you start pushing the date forward after each failed prediction, you end up looking like a hapless doomsday cult. If your hypothesis is correct however, your prediction should actually happen. Then provided you have not made so many predictions to get one right by chance, people will take what you have to say seriously.

Re: Creepy Crawlies

#83

Post by iririririr »

delichon wrote: post #42

> Why is git.kernel.org “interesting” to crawlers Interesting to crawlers is not a narrow scope. We have the same problem on a B2B car wash site.

so true. the article authors wishing crawlers will use git instead is so funny because the crawlers don't care at all. they are scrapping everything with brute force. they don't care about your content or effective alternatives, and one more site driving their real users crazy with Anubis is nothing more than a new blip in their dashboard. the crawler operators will not even look at the url.

Re: Creepy Crawlies

#84

Post by anal_reactor »

Demiurge wrote: post #26

I maintain a formerly popular gaming website, and it used to have hundreds of legitimate requests per second. The load would be especially high during popular event times. So, it’s always been running on a dedicated server. It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and d…

> Overall, it seems like the LLM are really straining the internet economy, the openness of it

I feel like it's the same issue that public resources are facing.

Step 1: public resource is freely available

Step 2: asshole shows up, drains the resource completely just because they can

Step 3: either resource is taken away, or it has massive security behind it

Re: Creepy Crawlies

#85

Post by IshKebab »

MithrilTuxedo wrote: post #68
yellow_lead wrote: post #25

High Anubis difficulty is annoying the hell out of me for several sites. And it's starting to not block LLM bots anymore? > 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.

Has anyone considered having Anubis perform more valuable hashing? Like, maybe you can't stop the LLM bots, but you can use them as one-off Bitcoin pool mining pool participants. You have to assume that making them find hash values with N leading zeroes has led to finding hash values with more than N leading zeroes. Maybe run a Bitcoin node under there and let each visitor take a couple swings for you with their pick…

Yeah that has been around for many years. Usually by sketchy download sites.

It's not really going to help though because the scrapers using residential proxies aren't burning their own compute.

Re: Creepy Crawlies

#86

Post by BowBun »

Waterluvian wrote: post #75

> It was immediately extremely effective — the bots just gave up. For a few months, it was bliss: bots were blocked at the perimeter and gave up, moving on to easier targets; the users were mildly annoyed but tolerated it, and the Anubis stack was easy enough to deploy everywhere. As a tech person who works with tech people, I have become extremely sensitive to this kind of bias. Is this solution actually better? Is…

Not sure if you mean the solution, or the problem they were trying to solve.

The problem is the cost here is paid by a group of volunteers, no? It's objectively reducing capacity for a really important public project by 20%. If you're responsible for keeping a public good like this available, it seems obvious to me to want to prevent this overuse of resources. Only one of the 3 parties involved here is not willing to engage in good faith right now and causing harm. I don't understand a need to try and tolerate them.

Re: Creepy Crawlies

#87

Post by ipdashc »

rcxdude wrote: post #40
Velocifyer wrote: post #18

But why don't they just git clone?

These crawlers (in contrast to e.g. googlebot and similar better behaved crawlers) are not very smart: they seem to make very little effort to avoiding crawling useless deep trees of generated pages. About the only thing they seem to put a lot of effort into is avoiding blocking. (I'll note that while these are generally attributed to AI data gathering because of the timing of when they took off, it's not actually ob…

> it's not actually obvious who's running these bots

This is fascinating to me. It's a large enough phenomenon that it's affecting the entire Internet and yet nobody seems to know yet who's actually doing it. Which isn't surprising, of course, it's hard to trace back to a source through all these proxies and it's probably a bunch of distinct groups anyways, but still!

Personally I have to wonder how much of it is "scrapers for training data" vs just tool-use LLMs. Even if you use chatgpt in thinking mode you can clearly see it searching and visiting a bunch of different websites to answer a question, presumably faster than any human would. That's got to add up. It's got me wondering why everyone seemingly discounts that as an option

Re: Creepy Crawlies

#88

Post by iririririr »

anyone knows how Jwz solution is working?

dont click next link because he will show a nutsack image if the referrer contains hackernews. love the guy.

www.jwz.org/blog/2025/01/exterminate-all-rational-ai-scrapers/

basically, instead of blocking, he just poison it. and if a human sees it, it takes less effort to ignore the nonsense than it takes your pocket computer to deal with proof of work.

Re: Creepy Crawlies

#89

Post by semiquaver » 

  > because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices.

I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my iPhone 17 to solve at ~100KH/s, making the site unusable. So I spent ~10 minutes vibe coding a safari extension with a native bridge to an optimized C kernel using ARM SHA256H* instructions that can do 200+ MH/s on the same device. This solves Anubis difficulty level 6 in a handful of milliseconds.

Given the numbers and capabilities involved (a single $5K ASIC miner yields 200TH/s, a million times more hash rate than my optimized kernel running on an iPhone), I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won.

Edit: I encourage you to try this yourself. Here's a sample prompt that ought to one-shot the task:

  > Build an iOS Safari Web Extension that accelerates Anubis proof-of-work using a native C ARM64 SHA-256 kernel. Precompute the invariant 128-byte challenge prefix, search fixed-width decimal nonces with ARM SHA-2 intrinsics and two worker threads, and target difficulty-6 solves under one second. Relay challenges from a Safari content script through the background service worker to native code, then submit the valid nonce/hash through Anubis’s normal pass-challenge endpoint. Include a deterministic benchmark app, correctness tests against CryptoKit, bounded execution, and fallback to Anubis’s stock solver.

Re: Creepy Crawlies

#90

Post by wiredfool »

delichon wrote: post #42

> Why is git.kernel.org “interesting” to crawlers Interesting to crawlers is not a narrow scope. We have the same problem on a B2B car wash site.

Seeing the exact same thing on (somewhat high profile) open data sites I run.

The crawlers get stuck in a loop requesting the dataset listing page with every. single. combination. of. facets. At essentially as fast as it can be pumped out or blocked.

Had one bot super interested in a single organization to the tune of 1000 r/min for 24+ hours. Realistic rotating user agents, realistic sec-* headers, no ip address seen more than a couple times in 10 minutes. The only commonality was the route.

Post reply on HN