51–60 of 68 posts
What stops a scraper from detecting Anubis and just removing "Mozilla" from the user-agent string?
Is there a way to alter text to poison AI training sets? I know there's Glaze and Nightshade for images but I've heard of nothing to poison text models. To be clear, this wouldn't be a defensive measure to stop scraping; it'd be an offensive honeypot: you'd want to make pages that have the same text but mutated slightly differently each time, so that AI scrapers preferentially load up on your statistically different…
Generate lots (LOTS!) of bad, insecure or wrong code and link it somewhere on your page (see https://www.emergent-misalignment.com/ , Ideally, make your local LLaMa generate more examples like those in the linked site)
Doesn't seem to noticably slow down my test bot. Headful crawling already takes ~10sec/page so an extra 0.5sec is hardly that big a deal.
A small scraper may be able to afford spending the extra CPU cycles, but if you're like AI training bots, sometimes sending hundreds of browser instances at a time, the math becomes different.
From what I've read about the results, it seems like the approach is effective against the very worst scrapers and bots out there.
Earlier quoted context omitted.
AFAICT this isn't possible, unless you're OK with showing everyone the poisoned text. TTBOMK there's nothing here that "detects botness" of an individual request, because in the limit, that's impossible -- if an attacker has access to many different IPs to make requests from (and many do), then any sequence of bot-generated requests from different IPs is indistinguishable from the same set of requests made by actual…
You glossed over how it works. Bots don't maintain session cookies to avoid rate limits, so they will have to do the challenge over and over again, whereas humans keep the session cookie and amortise the cost of the challenge over multiple requests.
> exp: The token's expiry week after the token was issued
This is surprising to me because it effectively nullifies Anubis.
> Bots don't maintain session cookies to avoid rate limits
Maybe they don't today, but there's absolutely nothing stopping them adding this, so if Anubis gets any traction, in two days' time they will.
Earlier quoted context omitted.
Maybe there should also be a protocol to support paying a small amount of crypto (on the order of 0.01-0.1 cents per request) to bypass the PoW. The bots can still scrape it by paying, but you now price in the externality.
Yes, I was also thinking of this. "proof of pay" or "proof of a transaction" An advantage of this would be not wasting electricity for the proof of work computation.
I'm really curious to see how this evolves as time goes on. Hashcash was originally conceived to stop e-mail SPAM, and a lot has changed since then, namely, compute has become absolutely dirt cheap. Despite that, PoW-based anti-bot remains somewhat enticing because it doesn't necessarily harm accessibility the way that solutions like Cloudflare or reCAPTCHA can: It should be possible to pass, even on a VPN or Tor, ev…
Some may not be aware that Hashcash value is as a decentralized rate limiter that can be added to almost any protocol. Experience with Hashcash taught us that it's essential to have a dynamic pricing scheme based on the reputation of the contact originator. In the email context, when a message sender connects to a receiving server, the receiving server should be able to tell the sender the size of the stamp needed based on the reputation created by previous messages.
From my perspective, the two main challenges in rate-limiting HTTP requests are embedding the required Hashcash stamp size and measuring the reputation of the request initiator. I think Anubis is a good first start, but the fact that it uses a fixed size stamp, a small one, does not have a reputation database, and from what I can tell, does not have a robust detector of good versus bad players. These shortcomings will make it challenging to provide adequate protection without interfering with good players.
I'll spare you my design note rambling, but I think from 3-page requests, one can gather enough information to determine the size of the Hashcash stamp for future requests.
I'm really curious to see how this evolves as time goes on. Hashcash was originally conceived to stop e-mail SPAM, and a lot has changed since then, namely, compute has become absolutely dirt cheap. Despite that, PoW-based anti-bot remains somewhat enticing because it doesn't necessarily harm accessibility the way that solutions like Cloudflare or reCAPTCHA can: It should be possible to pass, even on a VPN or Tor, ev…
FWIW, I was part of the gang talking with Adam about using Hashcash as a postage stamp as an anti-spam measure. I implemented an extension to Postfix using Hashcash as an anti-spam method (camram/two penny blue). The compute inflationary pressure you mentioned was only one of the challenges in deploying such a system. FWIW, none of the challenges were barriers; they were the kind of challenges you get when you take s…
That's what it's intended to be right now. I've been thinking out how to do a reputation database; I'm half considering using a DHT like BitTorrent's for a cross-Anubis coordination (haven't filed an issue about this because I'm still noodling it out into a spec). I'm also working on more advanced risk calculation, but this kind of exploded out of nowhere for me.
I would say it doesn't prevent anything, it just makes computers warm the planet more.
It'll suck for battery life if you're often browsing random website after random website, or if you're a bot farm, but in practice I don't think most welcome users will notice a thing.
Is there a way to alter text to poison AI training sets? I know there's Glaze and Nightshade for images but I've heard of nothing to poison text models. To be clear, this wouldn't be a defensive measure to stop scraping; it'd be an offensive honeypot: you'd want to make pages that have the same text but mutated slightly differently each time, so that AI scrapers preferentially load up on your statistically different…
AFAICT this isn't possible, unless you're OK with showing everyone the poisoned text. TTBOMK there's nothing here that "detects botness" of an individual request, because in the limit, that's impossible -- if an attacker has access to many different IPs to make requests from (and many do), then any sequence of bot-generated requests from different IPs is indistinguishable from the same set of requests made by actual…