Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

191–200 of 674 posts

Re: Creepy Crawlies

#191

Post by Kamq »

Arubis wrote: post #165
alkonaut wrote: post #161

Proof-of-humanity can’t come soon enough. We’re talking about privacy-preserving proof of age, but as we see here the real utility of such a system will be proof of humanity.

How do you define humanity? How do you ensure it includes every human? How do you ensure it doesn’t include every non-human? I’m not even asking about computation or algorithms. I straight up don’t think you can make a definition that isn’t a tautology or an approximation. Both of which are useful, but neither of which can fit a _proof_.

You're not wrong, but something can work well enough to still be useful despite falling short of the idea of a proof or any formal definition.

Let's say that 95% of individual humans can pass it and only 2% of bots. For someone maintaining a website, who has to decide between using this system and shutting down their site because of the increased costs, that may very well be good enough

Re: Creepy Crawlies

#192

Post by 6d6b73 »

Add a lot of random text to the html pages, preferably hidden to regular users, have the bots use lots of tokens to process it all.

Re: Creepy Crawlies

#193

Post by pbronez »

paytonjjones wrote: post #157
semiquaver wrote: post #112

Earlier quoted context omitted.

> I really don't see why Safari on a modern iPhone should be so slow at this me neither, but I don't think it changes the argument. There's always going to be someone on a low-end device. Your adversaries already have superhuman coding ability and infinite patience. Why would you expect the long-term advantage to be with the defenders?

In this case, because there's a vastly more efficient economic path for the adversaries (cloning). They're not trying to engage in an arms race, they're trying to channel a racing river into its natural course.

It’s more economical at a compute level, but not at the developer level. The moment you start customizing your crawler to use protocol X for site Y your scale story collapses.

Re: Creepy Crawlies

#194

Post by mbirth »

mattmcal wrote: post #181
afarah1 wrote: post #147

Why not aggressively rate limit? Legitimate use of HTML rendered commits should be largely unaffected, and crawlers slowed to a halt. You can even jail after a number of 429's...

There is a section in the article answering your question if you read it. > Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you.

I’d love a service like spamcop.net where I could submit my access_log and they lookup the abuse addresses and file abuse reports in my name. Maybe if people’s Internet access gets suspended they’ll think about installing random apps that work as a proxy in the background.

Re: Creepy Crawlies

#196

Post by inigyou »

rzmmm wrote: post #131
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

It does work, at least in my experience. The challenge may be cheap for your n-of-1 but keep in mind the scraper bots are crawling number of domains with very high concurrency.

And without control of their source IP addresses. They use a "rotating proxy" meta-proxy which sends each request through a different proxy, which invalidates the Anubis response and triggers a new challenge.

Re: Creepy Crawlies

#197

Post by inigyou »

semiquaver wrote: post #126
NobodyNada wrote: post #110

Earlier quoted context omitted.

The token should only be valid for one IP, so the scraper has to keep solving challenges if it rotates IPs.

OK, but that doesn't change anything. You have a large pool of IPs, each of which only needs to expend a handful of extra milliseconds of work to get unlimited access to the protected resource. Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a…

The way they're internally implemented doesn't allow pinning an IP. They buy a rotating proxy service from a vendor, and don't get to choose their source IP.

Re: Creepy Crawlies

#198

Post by inigyou »

LatticeAnimal wrote: post #152
semiquaver wrote: post #126

Earlier quoted context omitted.

OK, but that doesn't change anything. You have a large pool of IPs, each of which only needs to expend a handful of extra milliseconds of work to get unlimited access to the protected resource. Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a…

Some of the asymmetry might be regained if anubis had thousands of variations of PoW algorithms, each different enough that they must be solved independently. I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...

You can do this on yours. Just have the client and server add an extra "2" after the challenge key or something. A different client which extracts the challenge key and does its own processing will only generate invalid responses.

Re: Creepy Crawlies

#199

Post by __MatrixMan__ »

This is a fundamental flaw in the web. Since we treat a server's name as authoritative, anybody maintaining a replica needs to repeatedly hit that server to know if their remote version is up to date.

If we trusted digital signatures on content instead of server names, we could have a model where a single bit of server load propagates to millions of interested parties. As it is the server must do something distinct for each interested party.

CDNs mitigate this only partially, because mutable data means they have no good cache invalidation strategy.

There's got to be a solution that doesn't involve heaping even more burdensome requirements on those who would dare to publish.

Re: Creepy Crawlies

#200

Post by inigyou »

I also had this problem, but since nobody actually uses my gitea site besides crawlers, I just let a script run through my access log and ban every IP address who asked for a commit in the last 24 hours
Post reply on HN