Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

131–140 of 694 posts

Re: Creepy Crawlies

#131

Post by rzmmm »

semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

It does work, at least in my experience. The challenge may be cheap for your n-of-1 but keep in mind the scraper bots are crawling number of domains with very high concurrency.

Re: Creepy Crawlies

#132

Post by radlad »

semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

I'm reminded of LUKS Argon2 implementation, which is much slower than userspace. Therefore a sufficiently difficult challenge that may take less than a second in userspace Linux can take 20 or more seconds at boot.

Re: Creepy Crawlies

#133

Post by KomoD »

yellow_lead wrote: post #25

High Anubis difficulty is annoying the hell out of me for several sites. And it's starting to not block LLM bots anymore? > 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.

> High Anubis difficulty is annoying the hell out of me for several sites.

I just close the website if I see Anubis. Some have it set at reasonable difficulties (like 2)… others have it where I need to wait for like 30 seconds, I'm not wasting 30 seconds of my life for that.

Re: Creepy Crawlies

#135

Post by graemep »

nneonneo wrote: post #59
tptacek wrote: post #44

Tavis Ormandy called this, about Anubis, almost exactly a year ago: https://news.ycombinator.com/item?id=44962529 It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper…

I disagree. The kernel finds it effective - 66% of scrapers are turned away directly. The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site. If each new site they hit forces…

How s that measured? How do you count human users who have been turned away?

Re: Creepy Crawlies

#136

Post by jopsen »

znnajdla wrote: post #58

Put a CDN in front and let them absorb the load? Seriously, this is static content, which is so cheap to serve it should be free.

They allow you to diff commits, which is an awesome feature.

But if bots a crawling diffs between all possible commits it's crazy.

CDN will do nothing, because it's new urls each time. You can maybe find a CDN provider that block bots.

Re: Creepy Crawlies

#137

Post by timpera »

iririririr wrote: post #77
Demiurge wrote: post #26

I maintain a formerly popular gaming website, and it used to have hundreds of legitimate requests per second. The load would be especially high during popular event times. So, it’s always been running on a dedicated server. It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and d…

do not fall for cloudflare marketing. they have one of the worst bot detection in the industry. but because everyone uses them, their huge false positive numbers won't show up anywhere.

Cloudflare is also great at playing both sides, and they're trying pretty hard to push for pay-to-crawl because they'll probably get a 30% cut along the way.

Re: Creepy Crawlies

#138

Post by lxgr »

nneonneo wrote: post #59
tptacek wrote: post #44

Tavis Ormandy called this, about Anubis, almost exactly a year ago: https://news.ycombinator.com/item?id=44962529 It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper…

I disagree. The kernel finds it effective - 66% of scrapers are turned away directly. The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site. If each new site they hit forces…

Why would they even run a browser engine on the devices they're hosted on?

All they need to do is forward traffic and launder its IP origin. They don't even need to be able to (and would actually be well advised not to) decrypt TLS streams.

Re: Creepy Crawlies

#139

Post by jopsen »

I've seen this too.

I think it's a few bad actors really. Because nobody serious about indexing content will do what these crawlers are doing..

They are consume lots of content that is unoriginal or duplicate or duplicate with minor modifications.

Not sure how to block, but maybe a little bit of law enforcement could dramatically reduce the number of TVs being used a proxies.

Re: Creepy Crawlies

#140

Post by Demiurge »

timpera wrote: post #137
iririririr wrote: post #77

Earlier quoted context omitted.

do not fall for cloudflare marketing. they have one of the worst bot detection in the industry. but because everyone uses them, their huge false positive numbers won't show up anywhere.

Cloudflare is also great at playing both sides, and they're trying pretty hard to push for pay-to-crawl because they'll probably get a 30% cut along the way.

That’s definitely pretty annoying, it’s almost like the walled gardens scenario. I wish this could be more democratized.
Post reply on HN