Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

591–600 of 688 posts

Re: Creepy Crawlies

#591

Post by inigyou »

LatticeAnimal wrote: post #316
inigyou wrote: post #198

Earlier quoted context omitted.

You can do this on yours. Just have the client and server add an extra "2" after the challenge key or something. A different client which extracts the challenge key and does its own processing will only generate invalid responses.

Cool, so then that invalidates the ASIC problem, right? My earlier idea was to imagine that each day Anubis picks an entirely different problem-class. Ex: one day it is Sha256, the next it is prime factorization, the next it is twin-prime-finding, the next it is cracking elliptic curves, the next it is some kind of sorting / information theory problem... All with the goal of adapting constantly so that scrapers have…

That doesn't work because there is a finite list to choose from.

Nobody is using ASICs on Anubis.

Re: Creepy Crawlies

#592

Post by inigyou »

semiquaver wrote: post #401
inigyou wrote: post #197

Earlier quoted context omitted.

The way they're internally implemented doesn't allow pinning an IP. They buy a rotating proxy service from a vendor, and don't get to choose their source IP.

Why are you and other defenders of the Anubis approach so fixated on this one specific limitation of a certain type of scraping architecture? It’s hardly an immutable characteristic. You say “they” as if all scrapers are a monolithic group with the same constraints and goals. Part of the problem is the massive diversity.

Why do you think anyone thinks it's immutable? Do you understand the basic concept of an arms race, or cat and mouse game?

Re: Creepy Crawlies

#593

Post by throwawayffffas »

bourse_lee wrote: post #233

What if Anubis computations were turned into a crypto-miner

The whole concept behind Anubis is flawed. It tries to block access by imposing a compute cost to block people that are using incredible amounts of compute just to generate and parse the requests.

The people that Anubis tries to block have tons of compute to spare.

For example at runpod, if you rent a container with just one B300, you get 32 cpu cores and 250 gbs or ram that are essentially just sitting there while the gpu does all the work. If you think you can impose a compute cost on these people without blocking everyone else, well think again.

Or in my case, I am running qwen3.8 at home on a couple of gpus, these are attached on 32 core epyc server with 128gb of ram I am pretty sure i have more compute than the typical dev laptop.

I am of course nice, and don't aggressively scrape peoples services.

Re: Creepy Crawlies

#594

Post by whstl »

robotmay wrote: post #169

I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…

My team runs a quite popular website, #1 or #2 in the market depending on the region. Several million visits per day.

Around June/July we got a 10x boost out of nowhere, and it started affecting performance for users, increased hosting costs, and random bursts would bring the website down.

We spent some time trying out solutions, from Cloudflare and Anubis to AWS, but it ended up affecting real users, and we got complaints on Reddit from day one, pretty much. At the same time, it got a bit expensive having a WAF.

In the end the only solution that worked was blocking every single Chinese IP.

Traffic instantly got down to 10x less, same as it was before. Same conversion percentage as before.

Re: Creepy Crawlies

#595

Post by desterothx »

Demiurge wrote: post #303
inigyou wrote: post #241

Earlier quoted context omitted.

[to prove you are not a crawler and read this comment, please deposit $2.00]

I think that to prove I am not a crawler I probably just need to supply a Safari user agent string. The crawlers on my sites actually self identify for the most part. The malicious ones are banned when they try to post spam. Anyway, what’s wrong with making 2 cents? :)

The troubling part for me is that a system like this could also just as easily be pawned onto the humans (I realize it technically already is with the subscription model internet, but there are still plenty of open sites). A feature like this seems to me will lead to an internet where no one can browse anything without having a wallet connected and paying for each attempted access. Exactly what the internet is not supposed to be.

Re: Creepy Crawlies

#596

Post by 8474_s »

The solution is allowing the convenient web interface only for trusted members, like e.g. 4 year old account can view all pages(with easy anubis setting) but guest/bots/everyone else has to git clone the thing and do it on their backend. IIRC old forums also limited 'content for registered/trusted/moderators/etc' content views decades ago, turns out this crap saves gigatons of traffic.

Re: Creepy Crawlies

#597

Post by kev009 »

I run a web to text usenet gateway and it has been an interesting challenge to scale it to deal with this. Article fetch and retrieval is extremely efficient but I apply JWZ threading and that can cause a single article to make a number of overview requests which are more expensive depending on the depth. I solve it currently with caching but will eventually implement a thread backend on the NNTP side to keep thread roots updated at insertion time and it will be a very cheap read request.

One persistent thought is what are people doing with this data? I get that people want to train models, but I also have a hard time believing there are more than a hundred companies with the resources to spider the web like this and actually do anything meaningful with all that data. Academics, researchers, and people working on lower level innovation are probably well off with CommonCrawl.. only people trying to make frontier models really need fresh and endless data right?

Re: Creepy Crawlies

#599

Post by shakna »

inigyou wrote: post #521
shakna wrote: post #379

Earlier quoted context omitted.

Because Anthropic already admitted it? [0] [0] https://www.ft.com/content/07611b74-3d69-4579-9089-f2fc2af61...

It says "accused of"

... It also has a quote from Anthropic, admitting they backed off their scraper after a robots.txt update.
Post reply on HN