A year of fighting scrapers on my 1.5 million-page website

Post by econ » Fri, Aug 07, 2026, 4:14 PM UTC

inigyou wrote: post #47

drum55 wrote: post #31

Earlier quoted context omitted.

Which is trivially bypassed by an actual implementation of the proof of work in non-javascript, rendering it absolutely useless. The website is approximately 3800x times slower than native code, and hundreds of thousands of times slower than the CUDA kernel claude wrote. The "proof of work" is just non existent at that point, they're solved in milliseconds for what would take the browser version 10 minutes or more, i…

But the people you're defending against don't do that. They also don't load CSS but for some reason the security theater PoW won the mindshare.

Post by drum55 » Fri, Aug 07, 2026, 4:14 PM UTC

RobotToaster wrote: post #124

drum55 wrote: post #84

Earlier quoted context omitted.

People tried this in 2013 or so, there's no point to it. Doing proof of work in javascript in a browser is so crushingly, pointlessly slow that there's no value at all. Some browsers also intentionally detect attempts to do proof of work and attempt to block it entirely.

> Some browsers also intentionally detect attempts to do proof of work and attempt to block it entirely. Then they'd be blocking themselves from the website.

    To help you combat these unwelcome scripts, we’ve partnered with Disconnect and compiled lists of domains that serve cryptomining scripts so you can block them directly through Firefox. When you download Firefox,  tracking protection is enabled by default, blocking known cryptominers.

Post by aomix » Fri, Aug 07, 2026, 4:15 PM UTC

eigencoder wrote: post #111

jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. I think you've hit the nail on the head here. I think that's a big reason why people want to ban bots.

Post by whynotmaybe » Fri, Aug 07, 2026, 4:16 PM UTC

gmerc wrote: post #130

aeddZX_0 wrote: post #116

Earlier quoted context omitted.

How do you monetise bot traffic?

https://claw-guard.org/adnet

Post by kube-system » Fri, Aug 07, 2026, 4:18 PM UTC

throwaway219450 wrote: post #99

marginalia_nu wrote: post #60

Earlier quoted context omitted.

Static files on literally any hardware from the last 15 years on modern server software simply won't get cooked by bots. The network switch will bottleneck you before the server will. Your ephemeral port range will run out before the server will.

Bandwidth is still not free, if you cloud host. I had a client ask me to look into why their bill spiked 5x (admittedly not by much - from a few dollars to tens). They run a static site, and it turned out someone had inadvertently replaced an image with a massive uncompressed photo. Every unique visitor was caching O(25MB) when they loaded the homepage. Also funny because now everyone has 100Mb+ internet, nobody noti…