Earlier quoted context omitted.
I guess it could in principle be profitable for the website, not the client?
unfortunately at that point it would be indistinguishable from running cryptojacking on your website
No. The client is doing the work.
It took a year to ship WebAssembly in Anubis
81–90 of 223 posts
Re: It took a year to ship WebAssembly in Anubis
#82It's still very unclear to me how Anubis is supposed to work in the long term or even the medium term. The assumption is that scrappers, a.k.a. AI companies, a.k.a. those who hog all the RAM, are short of memory...?
[deleted]
Re: It took a year to ship WebAssembly in Anubis
#83Anubis is fine if you want to block bots that are bothering you, but I never understood why they don't have robots.txt at the root level too. I happily obey that for my private crawler. I suppose you don't want to duplicate implementations. Fair enough. I guess I should add an Anubis detector so I can just blacklist because that person clearly doesn't want bots.
One of the standard library rules enables robots.txt passthrough by default. I can't control people not choosing to use it.
That makes sense, Xe, but you don't enable it at:
- https://techaro.lol/robots.txt
- https://anubis.techaro.lol/robots.txt
and the latter (at least) is Anubis protected. It could be just an omission. But if it's intentional, I am just curious why so that as a bot author I can be well-behaved.
Re: It took a year to ship WebAssembly in Anubis
#84It's still very unclear to me how Anubis is supposed to work in the long term or even the medium term. The assumption is that scrappers, a.k.a. AI companies, a.k.a. those who hog all the RAM, are short of memory...?
The assumption is that there will always be some population distribution in terms of effort expended as well as in terms of problems caused. At present it's the lowest effort actors that are causing the highest number of problems. So an 80% (or even just a 20%) solution largely solves the practical problem faced by site operators. If against all odds it turns out that skiddies (or agents or AI billionaires or what ha…
Okay but it's like no RAM at all. Thats the problem: you've got stupid bots causing a bunch of issues - these can be stopped.
And then regular users: who now can't really browse your site.
And then smart bots - anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down.
The solution basically makes the favored customer well funded AI scrapers, not users.
Re: It took a year to ship WebAssembly in Anubis
#85The interesting part is that bots mitigation is actually an economic problem. You don't need to make scraping impossible, you only need to just make it expensive enough so that abusive traffic stops being worth it.
Scrapers scrape all sorts of nonsense, notably every possible git diff from git hosts. They clearly aren't concerned with rationality.
Scrapers scrape every link they can find.
The git host example is probably the one which Anubis is the dumbest defense for: the main reason to have those links is for easy machine interaction. So that's dead when you implement it.
So why even host the links? It's a git repo: send them a local got client and let them clone the repo in browser or something.
Re: It took a year to ship WebAssembly in Anubis
#86Earlier quoted context omitted.
The assumption is that there will always be some population distribution in terms of effort expended as well as in terms of problems caused. At present it's the lowest effort actors that are causing the highest number of problems. So an 80% (or even just a 20%) solution largely solves the practical problem faced by site operators. If against all odds it turns out that skiddies (or agents or AI billionaires or what ha…
Okay but it's like no RAM at all. Thats the problem: you've got stupid bots causing a bunch of issues - these can be stopped. And then regular users: who now can't really browse your site. And then smart bots - anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down. The solution basically makes the favored customer well funded AI scrapers, not users.
> And then regular users: who now can't really browse your site.
That's entirely up to how the site operator configures the PoW challenge.
> anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down
Do you have any evidence that it is the large well funded AI companies DoSing sites by (for example) scraping absurd numbers of permutations of cgit url parameters?
Even if it is (which I doubt), if all this accomplishes is getting them to spend 10 minutes optimizing their bots to not do that it will be a major win.
Re: It took a year to ship WebAssembly in Anubis
#87Earlier quoted context omitted.
Scrapers scrape all sorts of nonsense, notably every possible git diff from git hosts. They clearly aren't concerned with rationality.
Scrapers scrape every link they can find. The git host example is probably the one which Anubis is the dumbest defense for: the main reason to have those links is for easy machine interaction. So that's dead when you implement it. So why even host the links? It's a git repo: send them a local got client and let them clone the repo in browser or something.
The main reason to have those links is for people to click on them. No sensibly programmed machine is scraping diffs from a git web front end.
Re: It took a year to ship WebAssembly in Anubis
#88Anubis is fine if you want to block bots that are bothering you, but I never understood why they don't have robots.txt at the root level too. I happily obey that for my private crawler. I suppose you don't want to duplicate implementations. Fair enough. I guess I should add an Anubis detector so I can just blacklist because that person clearly doesn't want bots.
In this day and age, only good bots, the ones you actually want, obey robots.txt. You obtain no benefit by having a restrictive one.
That is true, but if you have Anubis you presumably do intend to `Disallow` all bots.
Re: It took a year to ship WebAssembly in Anubis
#89Re: It took a year to ship WebAssembly in Anubis
#90Earlier quoted context omitted.
The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute. Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amou…
> requiring 1 second of compute before allowing a page load After 1 solve, you get a cookie and can load tons of pages with it. Scrapers and user agents alike will need to spend the compute only once to get a valid session for themselves Seems to be working so far though. Fwiw, lots of silly things stop bots until someone puts their mind (or tokens) to bypassing it, and then the blocking mechanism has to adapt. We'll…
https://people.kernel.org/monsieuricon/creepy-crawlies reports anubis effectiveness falling off