When I visited Xe-sann's page, I was curious to see Jackal-chann challenges in action. This is Anubis.
Anubis: Proof-of-work proxy to prevent AI crawlers
41–50 of 68 posts
Re: Anubis: Proof-of-work proxy to prevent AI crawlers
#42This reminded me of an article I printed (yes, with paper) at my college more than 20 years ago, titled Parasitic Computing. I don't remember where it was originally published but I do think I might have stumbled upon it via kuro5hin (maybe); a quick search resulted the publication from Nature (though paywalled).
Re: Anubis: Proof-of-work proxy to prevent AI crawlers
#43Is there a way to alter text to poison AI training sets? I know there's Glaze and Nightshade for images but I've heard of nothing to poison text models. To be clear, this wouldn't be a defensive measure to stop scraping; it'd be an offensive honeypot: you'd want to make pages that have the same text but mutated slightly differently each time, so that AI scrapers preferentially load up on your statistically different…
I don't think that type of adversarial stuff will work in general. A better idea is to just make the models learn stuff their owners don't want. Maybe put erotica (AI generated or not), malware code, and other kinds of offensive content to annoy the bot owners.
Re: Anubis: Proof-of-work proxy to prevent AI crawlers
#44Ideas: - Make it generate cryptucurrency, so that the work is not wasted. Either to compensate for server expences hosting the content, or for some noble non-profit cause - all installations would collect the currency to a single account. Wasting the work is worse than these both options. - An easy way for good crawlers (like internet archive) to authenticate themselves. E.g. TLS client side authentication or simply…
Maybe there should also be a protocol to support paying a small amount of crypto (on the order of 0.01-0.1 cents per request) to bypass the PoW. The bots can still scrape it by paying, but you now price in the externality.
Re: Anubis: Proof-of-work proxy to prevent AI crawlers
#45Ideas: - Make it generate cryptucurrency, so that the work is not wasted. Either to compensate for server expences hosting the content, or for some noble non-profit cause - all installations would collect the currency to a single account. Wasting the work is worse than these both options. - An easy way for good crawlers (like internet archive) to authenticate themselves. E.g. TLS client side authentication or simply…
Maybe there should also be a protocol to support paying a small amount of crypto (on the order of 0.01-0.1 cents per request) to bypass the PoW. The bots can still scrape it by paying, but you now price in the externality.
Yes, I was also thinking of this.
"proof of pay" or "proof of a transaction"
An advantage of this would be not wasting electricity for the proof of work computation.
Re: Anubis: Proof-of-work proxy to prevent AI crawlers
#46Ideas: - Make it generate cryptucurrency, so that the work is not wasted. Either to compensate for server expences hosting the content, or for some noble non-profit cause - all installations would collect the currency to a single account. Wasting the work is worse than these both options. - An easy way for good crawlers (like internet archive) to authenticate themselves. E.g. TLS client side authentication or simply…
Alternatively to cryptocurrency, some other useful c9mputation. Like contribution to BOINC. Only need to find a way to prove the computation.
Re: Anubis: Proof-of-work proxy to prevent AI crawlers
#47Regarding the problem of how to let "good" bots through: You could use PKI: Drop the PoW if the client provides a TLS client certificate chain that asserts that corresponds to a private key that is controlled by (or just by, say, , for Let's Encrypt-style automatable cert signing). This would be a slight hassle for good bot operators to set up, but not a very big deal. The result is that bad bots couldn't spoof good…
well, good bots usually publish their ip ranges, and that's way simpler
Yes, that's much simpler, but doesn't it mean that every site owner needs to manage a hardcoded set of "places to look" for these published IPs?
If I want to create a good bot tomorrow, where do I publish its IP addresses? IOW, how can I ensure that the world "notices"?
Re: Anubis: Proof-of-work proxy to prevent AI crawlers
#48Ideas: - Make it generate cryptucurrency, so that the work is not wasted. Either to compensate for server expences hosting the content, or for some noble non-profit cause - all installations would collect the currency to a single account. Wasting the work is worse than these both options. - An easy way for good crawlers (like internet archive) to authenticate themselves. E.g. TLS client side authentication or simply…
I’m not sure how you can generate cryptocurrency with just sha256 hashes lol. Plus, there’s already a configuration file that lets you determine what you’re looking out for in terms of scrapers. I need to look at it more myself especially because I plan on using this on my own website soon, but I hope there’s already a good way to whitelist archival sites.
Re: Anubis: Proof-of-work proxy to prevent AI crawlers
#49Is there a way to alter text to poison AI training sets? I know there's Glaze and Nightshade for images but I've heard of nothing to poison text models. To be clear, this wouldn't be a defensive measure to stop scraping; it'd be an offensive honeypot: you'd want to make pages that have the same text but mutated slightly differently each time, so that AI scrapers preferentially load up on your statistically different…
AFAICT this isn't possible, unless you're OK with showing everyone the poisoned text. TTBOMK there's nothing here that "detects botness" of an individual request, because in the limit, that's impossible -- if an attacker has access to many different IPs to make requests from (and many do), then any sequence of bot-generated requests from different IPs is indistinguishable from the same set of requests made by actual…
You glossed over how it works. Bots don't maintain session cookies to avoid rate limits, so they will have to do the challenge over and over again, whereas humans keep the session cookie and amortise the cost of the challenge over multiple requests.
Re: Anubis: Proof-of-work proxy to prevent AI crawlers
#50Is there a way to alter text to poison AI training sets? I know there's Glaze and Nightshade for images but I've heard of nothing to poison text models. To be clear, this wouldn't be a defensive measure to stop scraping; it'd be an offensive honeypot: you'd want to make pages that have the same text but mutated slightly differently each time, so that AI scrapers preferentially load up on your statistically different…
Generate lots (LOTS!) of bad, insecure or wrong code and link it somewhere on your page (see https://www.emergent-misalignment.com/ , Ideally, make your local LLaMa generate more examples like those in the linked site)