Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

231–240 of 693 posts

Re: Creepy Crawlies

#231

Post by robotmay »

mlaretallack wrote: post #215
robotmay wrote: post #169

I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…

I hate to ask but what's a "sexy toasters."

Gemini's surprisingly bad at it but so far I have some voluptuous curvy toasters, toasters in the shape of a butt, and robot girls with toasters for tits. I have a Gemini subscription for the month after mistakenly thinking I'd get cheap Opencode usage through it, so gotta use it for something.

Re: Creepy Crawlies

#232

Post by wbl »

inigyou wrote: post #196
rzmmm wrote: post #131

Earlier quoted context omitted.

It does work, at least in my experience. The challenge may be cheap for your n-of-1 but keep in mind the scraper bots are crawling number of domains with very high concurrency.

And without control of their source IP addresses. They use a "rotating proxy" meta-proxy which sends each request through a different proxy, which invalidates the Anubis response and triggers a new challenge.

Real attackers fuck. If they have IP address control issues they will fix them. The bots being stopped are the silly ones.

Re: Creepy Crawlies

#234

Post by karlgkk »

semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won.

You yourself describe the solution.

Extensions or browser integrations of the Anubis task.

Re: Creepy Crawlies

#235

Post by vintermann »

mzajc wrote: post #201

> Why is git.kernel.org “interesting” to crawlers I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests. The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this mea…

A lot of work is apparently put into bypassing any kind of anti-scraping, no work is apparently put into figuring out if the site freely gives a way to get all that information in a less wasteful way.

Re: Creepy Crawlies

#236

Post by inigyou »

virgoerns wrote: post #214

I also run a public cgit instance and get over 1M hits every day, although my pet projects are nowhere near the size or impact of kernel. I had to block (via nginx conf) cgit endpoints for diffs, blame, snapshots and historical commits, because nothing else works. Now they return 402 (payment required). I consider this my total defeat and it's killing me inside, but it is what it is.

You could also publish a list of IP addresses.

Re: Creepy Crawlies

#237

Post by dpark »

karlgkk wrote: post #234
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won. You yourself describe the solution. Extensions or browser integrations of the Anubis task.

He didn’t describe a solution. He described a (crappy) workaround for humans. But the fact is that this cannot and will not stop bots. The people running bots can do the same, even faster.

Re: Creepy Crawlies

#238

Post by inigyou »

tptacek wrote: post #145
harshreality wrote: post #142

Earlier quoted context omitted.

If you want to argue that there's excessive enthusiasm about anubis, which is neither a permanent solution nor a solution against all bots run for all purposes (some bot authors are cleverer than others, and you don't have to block everything, just enough to reduce resource use to tolerable levels)... fine. However, the refrain from naysayers isn't that there's too much enthusiasm. It's that anubis doesn't make sense…

I'm not moralizing. The sites running Anubis are owned by people who like Anubis and are fully within their rights to run Anubis. I don't care how enthusiastic they are about it. It's their call. I'm just saying, it's bad computer science.

Which part of computer science is about getting DDoSed by billionaires who are above the law?

Re: Creepy Crawlies

#239

Post by inigyou »

wongarsu wrote: post #184
Y_Y wrote: post #64

Earlier quoted context omitted.

The implication here is that the proxy fridge forwards the Anubis challenge to a dedicated rig controlled by the scraper who efficiently solves it and returns the answer.

That's still a notable step up in completely and resource investment for the crawler See also how captchas continued being effective for years despite services like anti-captcha offering to solve them for you for a fifth of a cent each by farming the work out to India. It took advances in AI that made it viable to reliably solve them on-device to bring the end of the captcha

That's not a step up, that is how they work by default.

Re: Creepy Crawlies

#240

Post by inigyou »

atq2119 wrote: post #168
jdnier wrote: post #72

I really enjoyed the writing style in this article. And the bot progression from "alter user agent" to "change IP addresses" to providers having to ban whole subnets, whole ASNs, and realizing "proxy SDK monetization" is a thing mirrors threat actor progression from the time before LLMs.

It feels like this progression of increasingly drastic measures to circumvent the protections of a computer system ought to be enough to establish criminal intent and get some of the people running those crawlers into prison.

It does. It's literally a felony but for some reason not a single person has pressed charges.
Post reply on HN