Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anubis: Proof-of-work proxy to prevent AI crawlers

anubis.techaro.lol

Post reply on HN
Print view View original

31–40 of 68 posts

Re: Anubis: Proof-of-work proxy to prevent AI crawlers

#31

Post by jchw »

I'm really curious to see how this evolves as time goes on. Hashcash was originally conceived to stop e-mail SPAM, and a lot has changed since then, namely, compute has become absolutely dirt cheap. Despite that, PoW-based anti-bot remains somewhat enticing because it doesn't necessarily harm accessibility the way that solutions like Cloudflare or reCAPTCHA can: It should be possible to pass, even on a VPN or Tor, even on less used web browsers like Ladybird or Servo, and even if you're not on a super powerful device (provided you are willing to wait for the PoW challenge to pass, but as long as you don't have all of these conditions at once you should get an "easy" challenge and it should be quick.)

The challenge is definitely figuring out if this solution actually works at scale or not. I've played around with an implementation of Hashcash myself, using WebCrypto, but I worry because even using WebCrypto it is quite a lot slower than cracking hashes in native code. But seeing Anubis seemingly have some success makes me hopeful. If it gains broad adoption, it might just be enough of a pain in the ass for scrapers, while still being possible for automation to pass provided they can pay the compute toll (e.g. hopefully anything that's not terribly abusive.)

On a lighter note, I've found the reception of Anubis, and in particular the anime-style mascot, to be predictably amusing.

https://discourse.gnome.org/t/anime-girl-on-gnome-gitlab/276...

(Note: I'd personally suggest not going and replying here. Don't want to encourage brigading of any sort, just found this mildly amusing.)

Re: Anubis: Proof-of-work proxy to prevent AI crawlers

#32

Post by ronsor »

avodonosov wrote: post #28

Ideas: - Make it generate cryptucurrency, so that the work is not wasted. Either to compensate for server expences hosting the content, or for some noble non-profit cause - all installations would collect the currency to a single account. Wasting the work is worse than these both options. - An easy way for good crawlers (like internet archive) to authenticate themselves. E.g. TLS client side authentication or simply…

> Make it generate cryptucurrency, so that the work is not wasted. Either to compensate for server expences hosting the content, or for some noble non-profit cause - all installations would collect the currency to a single account. Wasting the work is worse than these both options.

The last time[0] we did this, everyone had a meltdown and blocked it.

[0] See Coinhive, which conspicuously lacks a Wikipedia page.

Re: Anubis: Proof-of-work proxy to prevent AI crawlers

#33

Post by Jotalea »

kmeisthax wrote: post #23

Is there a way to alter text to poison AI training sets? I know there's Glaze and Nightshade for images but I've heard of nothing to poison text models. To be clear, this wouldn't be a defensive measure to stop scraping; it'd be an offensive honeypot: you'd want to make pages that have the same text but mutated slightly differently each time, so that AI scrapers preferentially load up on your statistically different…

In theory you could "scramhe teble txt" (scramble the text) and show the proper one to the user, but I don't know how (in)efficient would that be.

Re: Anubis: Proof-of-work proxy to prevent AI crawlers

#36

Post by Trung0246 »

For no js solution, I think some sort of using optical illusion as captcha could works, especially https://en.wikipedia.org/wiki/Magic_Eye or something like https://www.youtube.com/watch?v=Bg3RAI8uyVw which could cleverly hide captcha answer within animated noise mess.

However these methods are not really accessibility-friendly tho.

Re: Anubis: Proof-of-work proxy to prevent AI crawlers

#37

Post by wslh »

ronsor wrote: post #32
avodonosov wrote: post #28

Ideas: - Make it generate cryptucurrency, so that the work is not wasted. Either to compensate for server expences hosting the content, or for some noble non-profit cause - all installations would collect the currency to a single account. Wasting the work is worse than these both options. - An easy way for good crawlers (like internet archive) to authenticate themselves. E.g. TLS client side authentication or simply…

> Make it generate cryptucurrency, so that the work is not wasted. Either to compensate for server expences hosting the content, or for some noble non-profit cause - all installations would collect the currency to a single account. Wasting the work is worse than these both options. The last time[0] we did this, everyone had a meltdown and blocked it. [0] See Coinhive, which conspicuously lacks a Wikipedia page.

It is mentioned at https://en.wikipedia.org/wiki/Cryptojacking>

Re: Anubis: Proof-of-work proxy to prevent AI crawlers

#38

Post by avodonosov »

wslh wrote: post #37
ronsor wrote: post #32

Earlier quoted context omitted.

> Make it generate cryptucurrency, so that the work is not wasted. Either to compensate for server expences hosting the content, or for some noble non-profit cause - all installations would collect the currency to a single account. Wasting the work is worse than these both options. The last time[0] we did this, everyone had a meltdown and blocked it. [0] See Coinhive, which conspicuously lacks a Wikipedia page.

It is mentioned at https://en.wikipedia.org/wiki/Cryptojacking >

And the mention links to https://en.m.wikipedia.org/wiki/Coinhive which redirects to https://en.m.wikipedia.org/wiki/Monero#Mining_malware

Re: Anubis: Proof-of-work proxy to prevent AI crawlers

#39

Post by avodonosov »

ronsor wrote: post #32
avodonosov wrote: post #28

Ideas: - Make it generate cryptucurrency, so that the work is not wasted. Either to compensate for server expences hosting the content, or for some noble non-profit cause - all installations would collect the currency to a single account. Wasting the work is worse than these both options. - An easy way for good crawlers (like internet archive) to authenticate themselves. E.g. TLS client side authentication or simply…

> Make it generate cryptucurrency, so that the work is not wasted. Either to compensate for server expences hosting the content, or for some noble non-profit cause - all installations would collect the currency to a single account. Wasting the work is worse than these both options. The last time[0] we did this, everyone had a meltdown and blocked it. [0] See Coinhive, which conspicuously lacks a Wikipedia page.

This time the site content will not be available if the challenge computation is blocked.
Post reply on HN