Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

531–540 of 711 posts

Re: Creepy Crawlies

#531

Post by alok-g »

robotmay wrote: post #169

I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…

I had to look up what iocaine route is. Here's an article that explains well:

https://firesphere.dev/articles/iocaine-the-deadliest-poison...

Re: Creepy Crawlies

#532

Post by titularcomment »

apitman wrote: post #315
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

This comment captures the current AI zeitgeist about as well as anything I've seen.

I like that they supplied the prompt instead of the working code :)

Re: Creepy Crawlies

#533

Post by lxgr »

FridgeSeal wrote: post #471
lxgr wrote: post #130

Earlier quoted context omitted.

> 66% of scrapers are turned away directly. Until they discover this neat trick [1] and solve challenges orders of magnitudes more efficiently than legitimate users. The game theory of Anubis is not sound. It makes fundamentally less sense than Captchas, and even those have been on the way out for a while. [1] https://news.ycombinator.com/item?id=49500040

It doesn’t have to be fully sound, it’s there to provide _some_ degree of interim protection. Evidently, it makes _enough_ of a difference for the people running it. Criticising it is just letting the perfect be the enemy of the good. The alternative to Anubis isn’t “something better that isn’t being picked”, it’s “completely unprotected”, and the people criticising it for being imperfect could helpfully move the con…

Sure, but then you’re essentially just doing versions of the old “what is 2+2” text captchas with extra computational steps, so why not save yourself and your users those?

Re: Creepy Crawlies

#534

Post by yread »

We need a solid way to prove we are meatbags. How about a simple USB accessory that goes around your neck and measures your ECG. It could also give you subtle electrical shocks (in a unique pattern; a challenge) and measure how the ECG responds to that.

Re: Creepy Crawlies

#535

Post by robotmay »

charles_f wrote: post #400
hahnbee wrote: post #398

Earlier quoted context omitted.

what constitutes as a bad crawler and what information do you have that's attracting so many bots?

Not op, but a common tactic is to put honeypots in a page set to be forbidden in Robots.txt. Not respecting Robots.txt is making you a bad crawler. I have marginal blog, but its somewhat indexed and it gets tons of traffics from crawlers

Yep this is how I'm doing it. It's marked as my super exciting client archive full of high-res RAW photos that all bots are told not to index in robots.txt, only for some weird reason the link never gets rendered to the page :>

I also explicitly tell all crawlers not to index the images on my site, actually, so I feel even better about serving the ones that do complete junk.

Not sure why I get more crawler traffic than I do legitimate traffic tbh - it's a relatively new website that I've been trying to sort out the SEO for as I'm trying to get photography work, so it's probably started cropping up on the radar of scrapers looking for training data.

Re: Creepy Crawlies

#536

Post by fc417fc802 »

Dylan16807 wrote: post #504
fc417fc802 wrote: post #461

Earlier quoted context omitted.

> Anubis wants "slow for bots but fast for humans" That's not true. The goal is to raise the cost for bulk scrapers to the extent possible while minimizing any inconvenience to regular users. It's not about being slow for bots but rather reducing the asymmetry of the problem (from the perspective of the server) by imposing a (minor) cost on anyone requesting a page.

That's a better way to put the goal. But does it actually raise the cost or reduce asymmetry in a meaningful way? If it makes crawling 0-5% slower then it's really not worth the annoyance. I know having the Anubis javascript at all gets in the way of some bots. But is the calculation helping enough?

> does it actually raise the cost or reduce asymmetry in a meaningful way?

It's complicated. I think anubis currently only collects low hanging fruit so to speak but in practice that's the bulk of what was causing problems so it's good enough.

PoW in general absolutely yes - I think monero clearly demonstrates this. Last I checked it remained viable to mine monero on at least somewhat low end and at least somewhat outdated consumer CPUs. It seems to me that fact implies that a well constructed PoW scheme can in practice squash the impact of any performance differences to within at most a couple orders of magnitude.

Re: Creepy Crawlies

#537

Post by robotmay »

eloisius wrote: post #478
robotmay wrote: post #169

I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…

How do you reliably detect the bots? Instead of slowban, I would feed it something abhorrent like an archive of 4chan posts.

It's mostly about luring them into URLs they're explicitly told in robots.txt that they shouldn't index. I do some identification via reverse DNS of known crawlers I actually want like Googlebot, though they respect robots.txt, in case something goes wrong and they accidentally get flagged.

Re: Creepy Crawlies

#538

Post by panki27 »

xiconfjs wrote: post #530
mzajc wrote: post #201

> Why is git.kernel.org “interesting” to crawlers I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests. The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this mea…

Same here. Have a small gitea instance with cloned projects from github I‘m keeping in case the github version gets removed. Every few days there is an army of bots hammering my small vm with 40k req/min for 20 min straight. I had to install anubis to keep the instance online (while still slowed down).

I don't even have anything worthwile on my gitea instance, and the same thing happened to me.

The only thing that ended up helping was disabling access for anonymous users.

Re: Creepy Crawlies

#539

Post by robotmay »

someothherguyy wrote: post #334
robotmay wrote: post #169

I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…

wouldn't that just make your connection load worse?

Currently not really an issue on this site but connections aren't an issue on Elixir usually anyway, unless you get up to about 1 million on one machine IIRC.

Re: Creepy Crawlies

#540

Post by yuumei »

Here is an idea: instead of PoW do “proof of AI” each user who wants to access the page run AI/LLM prompts to undercut the corpos abusing the pages
Post reply on HN