221–230 of 707 posts
That’s not an option for the kernel. It’s hard to read the cat-and-mouse account with any hope today. I think the flood abates someday but not sure how it happens.
Put a CDN in front and let them absorb the load? Seriously, this is static content, which is so cheap to serve it should be free.
Did you read the article? “[…]because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git.”
> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
Earlier quoted context omitted.
The token should only be valid for one IP, so the scraper has to keep solving challenges if it rotates IPs.
OK, but that doesn't change anything. You have a large pool of IPs, each of which only needs to expend a handful of extra milliseconds of work to get unlimited access to the protected resource. Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a…
And yet many report it works, at least for now, and the excess load due to scraping activity falls precipitously when Anubis or similar solutions are used. Maybe once this sort of challenge is used almost everywhere we'll see concerted attempts to get around it, but for now it is easier for them to just move on to another target.
> a $5000 ASIC-based device can hash SHA256 at 200TH/s or more
Peanuts for the big players, but many (almost all?) running smaller scale scraping operations are going to find $5000 rather prohibitive, and they are unlikely to be able to integrate it as they are probably running a “stock” scraper that they didn't write themselves.
Earlier quoted context omitted.
But the scraper is making way more requests and is paying for all that compute.
This is the basic misconception behind the Anubis design. In an anti-spam system, there's a user/abuser reward asymmetry. Users make single requests and get some constant reward V. Abusers make millions of requests and also get some constant reward V. Proof-of-work exploits the asymmetry: costs are, by a factor of ~millions, disproportionately imposed on abusers. That's exactly not what happens with scrapers. Scraper…
Anubis doesn't really help against this.
I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…
Probably they must be deduplicating text they've seen before. The only punishment would be unique text that trains their models to be degenerate. And even then you'd probably have to serve across many domains.
The altered versions would be the page or paragraphs of it, with sentences and words swapped arbitrarily so that training on the data adds as many iffy token sequences as correct ones.
> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…
The better solution would be fix Anubis and optimise its calculations on iPhone. Maybe it needs WebAssembly hashing implementation instead of JS? But it would just make Anubis suck a little bit less Anubis "protection" model is wrong/broken on several levels. I suppose it does not even have ability to bypass hashing screen when server load is low.
Anubis is not a load optimizer. It's sole purpose is to deny the bots you don't want on your site.
It's on the same category as Iocaine and Go-Away, even if what they do is different in some cases.
from what I see there are 2 solutions: 1) ban TV-proxy-as-a-service - straight up go to every representative there is and start pushing and lobbying and everything to stop spammers from distributing over non-computer devices, especially legally 2) make old commits more expensive to access than new ones. Legit users are not going to access those much, so they can pay the time. I assume diverse (unpredictable?) difficu…
> straight up go to every representative there is In every country of the world? I think you might have a significantly easier time convincing Samsung and LG to ban this via their T&C.
Earlier quoted context omitted.
It’s more economical at a compute level, but not at the developer level. The moment you start customizing your crawler to use protocol X for site Y your scale story collapses.
It's a good point, but in practice it depends on how easy those customizations are to implement / maintain, and how much money and effort you save. At some point the compute cost can disrupt even the nicest scale story. I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and sc…