Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anubis: Proof-of-work proxy to prevent AI crawlers

anubis.techaro.lol

Post reply on HN
Print view View original

61–68 of 68 posts

Re: Anubis: Proof-of-work proxy to prevent AI crawlers

#61

Post by xg15 »

ronsor wrote: post #32
avodonosov wrote: post #28

Ideas: - Make it generate cryptucurrency, so that the work is not wasted. Either to compensate for server expences hosting the content, or for some noble non-profit cause - all installations would collect the currency to a single account. Wasting the work is worse than these both options. - An easy way for good crawlers (like internet archive) to authenticate themselves. E.g. TLS client side authentication or simply…

> Make it generate cryptucurrency, so that the work is not wasted. Either to compensate for server expences hosting the content, or for some noble non-profit cause - all installations would collect the currency to a single account. Wasting the work is worse than these both options. The last time[0] we did this, everyone had a meltdown and blocked it. [0] See Coinhive, which conspicuously lacks a Wikipedia page.

Mining crypto can lead to misaligned incentives. Suddenly, people would have motivation to run this thing even on benign requests, etc.

Re: Anubis: Proof-of-work proxy to prevent AI crawlers

#62

Post by theragra »

jeroenhd wrote: post #58
ranger_danger wrote: post #13

I would say it doesn't prevent anything, it just makes computers warm the planet more.

A quick PoW calculation per web session is probably not going to make a dent in pollution now that generative AI has become commonplace. It'll suck for battery life if you're often browsing random website after random website, or if you're a bot farm, but in practice I don't think most welcome users will notice a thing.

Other comment mentions it takes 30 seconds in mobile Firefox to pass, obviously too much

Re: Anubis: Proof-of-work proxy to prevent AI crawlers

#63

Post by kmeisthax »

akoboldfrying wrote: post #54
imtringued wrote: post #49

Earlier quoted context omitted.

You glossed over how it works. Bots don't maintain session cookies to avoid rate limits, so they will have to do the challenge over and over again, whereas humans keep the session cookie and amortise the cost of the challenge over multiple requests.

You're right, Anubis sets a week-long cookie: > exp: The token's expiry week after the token was issued This is surprising to me because it effectively nullifies Anubis. > Bots don't maintain session cookies to avoid rate limits Maybe they don't today, but there's absolutely nothing stopping them adding this, so if Anubis gets any traction, in two days' time they will.

You can turn Anubis into Proof of Storage by appending large amounts of random data to the session cookie and then hashing it on each request. Combined with per-session-cookie limits, you can effectively force the LLM bots to store all the data you want if they want to scrape your site.

Maybe suckerpinch can work it into a sequel to Harder Drive: Hard drives we didn't want or need[0].

[0] https://www.youtube.com/watch?v=JcJSW7Rprio

Re: Anubis: Proof-of-work proxy to prevent AI crawlers

#64

Post by Silphendio »

ronsor wrote: post #32
avodonosov wrote: post #28

Ideas: - Make it generate cryptucurrency, so that the work is not wasted. Either to compensate for server expences hosting the content, or for some noble non-profit cause - all installations would collect the currency to a single account. Wasting the work is worse than these both options. - An easy way for good crawlers (like internet archive) to authenticate themselves. E.g. TLS client side authentication or simply…

> Make it generate cryptucurrency, so that the work is not wasted. Either to compensate for server expences hosting the content, or for some noble non-profit cause - all installations would collect the currency to a single account. Wasting the work is worse than these both options. The last time[0] we did this, everyone had a meltdown and blocked it. [0] See Coinhive, which conspicuously lacks a Wikipedia page.

There's a German Wikipedia page: https://de.m.wikipedia.org/wiki/Coinhive

Re: Anubis: Proof-of-work proxy to prevent AI crawlers

#65

Post by rickydroll »

xena wrote: post #57
rickydroll wrote: post #56

Earlier quoted context omitted.

FWIW, I was part of the gang talking with Adam about using Hashcash as a postage stamp as an anti-spam measure. I implemented an extension to Postfix using Hashcash as an anti-spam method (camram/two penny blue). The compute inflationary pressure you mentioned was only one of the challenges in deploying such a system. FWIW, none of the challenges were barriers; they were the kind of challenges you get when you take s…

> I think Anubis is a good first start, but the fact that it uses a fixed size stamp, a small one, does not have a reputation database, and from what I can tell, does not have a robust detector of good versus bad players. These shortcomings will make it challenging to provide adequate protection without interfering with good players. That's what it's intended to be right now. I've been thinking out how to do a reputa…

I would focus on the bad player detector first, then move on to the dynamic pricing scheme. I wouldn't worry about cross-Anubis coordination until you've solved the database problem for the single node.

IMO, of course.

Re: Anubis: Proof-of-work proxy to prevent AI crawlers

#66

Post by akoboldfrying »

kmeisthax wrote: post #63
akoboldfrying wrote: post #54

Earlier quoted context omitted.

You're right, Anubis sets a week-long cookie: > exp: The token's expiry week after the token was issued This is surprising to me because it effectively nullifies Anubis. > Bots don't maintain session cookies to avoid rate limits Maybe they don't today, but there's absolutely nothing stopping them adding this, so if Anubis gets any traction, in two days' time they will.

You can turn Anubis into Proof of Storage by appending large amounts of random data to the session cookie and then hashing it on each request. Combined with per-session-cookie limits, you can effectively force the LLM bots to store all the data you want if they want to scrape your site. Maybe suckerpinch can work it into a sequel to Harder Drive: Hard drives we didn't want or need [0]. [0] https://www.youtube.com/wat…

Great clip! I love the idea of making internet miscreants reluctantly store your data!

Slightly more seriously though, I think for the Proof of Storage idea to pack enough punch to be a deterrent, you'd need the cookies to be quite large. Is there a way to avoid needing them to send you all the bytes each time? Because that will cost you (the site owner) too.

I had the idea of sending requestors a challenge like "Flip the bit at position i of the large chunk of stored data, hash the result and include the hash in your headers". Instead of the site owner keeping the full stored data, they would just keep the RNG seed used to generate it -- this saves on storage, though it still requires them to do the same time-consuming hash computation done by the requestor.

Re: Anubis: Proof-of-work proxy to prevent AI crawlers

#67

Post by TUSF »

kmeisthax wrote: post #23

Is there a way to alter text to poison AI training sets? I know there's Glaze and Nightshade for images but I've heard of nothing to poison text models. To be clear, this wouldn't be a defensive measure to stop scraping; it'd be an offensive honeypot: you'd want to make pages that have the same text but mutated slightly differently each time, so that AI scrapers preferentially load up on your statistically different…

One idea is to include a bunch of nonsense text into webpages, and use CSS to hide it from browsers. Depending on how aggressively you want to do it, this may affect some accessibility tooling.

Re: Anubis: Proof-of-work proxy to prevent AI crawlers

#68

Post by fc417fc802 »

akoboldfrying wrote: post #54
imtringued wrote: post #49

Earlier quoted context omitted.

You glossed over how it works. Bots don't maintain session cookies to avoid rate limits, so they will have to do the challenge over and over again, whereas humans keep the session cookie and amortise the cost of the challenge over multiple requests.

You're right, Anubis sets a week-long cookie: > exp: The token's expiry week after the token was issued This is surprising to me because it effectively nullifies Anubis. > Bots don't maintain session cookies to avoid rate limits Maybe they don't today, but there's absolutely nothing stopping them adding this, so if Anubis gets any traction, in two days' time they will.

Anubis remains effective because the token is subject to a rate limit and you could additionally limit token lifetime to some maximum number of requests if you wanted to.

All of these factors (total requests, rate of requests, associated IPs, associated browser fingerprints) tie in to detecting bad players, who should receive more frequent and larger challenges.

Post reply on HN