211–220 of 693 posts
I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…
> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…
I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…
I've seen this too. I think it's a few bad actors really. Because nobody serious about indexing content will do what these crawlers are doing.. They are consume lots of content that is unoriginal or duplicate or duplicate with minor modifications. Not sure how to block, but maybe a little bit of law enforcement could dramatically reduce the number of TVs being used a proxies.
I run a website with 10k unique pages. If I leave the gates open, Meta hits it 200.000 times per day. Every day. What are you paying developers $500k for Mark?
That's bad for static content.
Try adding search, with pagination and 16 filters that can toggled on off. And 1000 tags, give each page 5.
Earlier quoted context omitted.
OK, but that doesn't change anything. You have a large pool of IPs, each of which only needs to expend a handful of extra milliseconds of work to get unlimited access to the protected resource. Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a…
Some of the asymmetry might be regained if anubis had thousands of variations of PoW algorithms, each different enough that they must be solved independently. I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...
It seems clear to me we are moving towards a world where you will have to perform device attestation to access the internet. The spam/abuse is too great and accelerating.
TIL shallow clones are expensive. That's wild to me. It's supposed to be cheaper!