A year of fighting scrapers on my 1.5 million-page website

Post by everybodyknows » Fri, Aug 07, 2026, 4:20 PM UTC

qbane wrote: post #103

ihuman wrote: post #68

Earlier quoted context omitted.

There's a difference between someone running a scraping tool occasionally and bots constantly and rapidly re-scraping the same site over and over again

This is an important context: the site is more likely to be targeted by scrapers because it is a curated collection of scraped information.

Post by throw93003838 » Fri, Aug 07, 2026, 4:21 PM UTC

bob1029 wrote: post #121

jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

> the real problem is that the website is poorly built This is almost always the problem. This is currently the problem that GitHub is having too. If they had remained with their crusty old rails architecture, very little of this mess [0] would be occurring right now. They would have been able to focus all their engineering talent on scaling the product rather than inventing elaborate client side state synchronizatio…

Post by paul7986 » Fri, Aug 07, 2026, 4:22 PM UTC

eigencoder wrote: post #111

jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. I think you've hit the nail on the head here. I think that's a big reason why people want to ban bots.

Post by rokkamokka » Fri, Aug 07, 2026, 4:23 PM UTC

drum55 wrote: post #31

johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

Which is trivially bypassed by an actual implementation of the proof of work in non-javascript, rendering it absolutely useless. The website is approximately 3800x times slower than native code, and hundreds of thousands of times slower than the CUDA kernel claude wrote. The "proof of work" is just non existent at that point, they're solved in milliseconds for what would take the browser version 10 minutes or more, i…

Post by runjake » Fri, Aug 07, 2026, 4:23 PM UTC

throwaway219450 wrote: post #53

conartist6 wrote: post #29

Earlier quoted context omitted.

That's the thing: even if you don't use AI, if most of what you read is AI, soon this is what you'll sound like, just because it's so much of the training data your own brain's model has. Using it slowly sucks the uniqueness out of you.

Using AI to write blogs doesn’t bother me in principle, but I hate the prose that gets left in. Even if it’s not AI writing, being human doesn’t give you a pass for writing like a self-help guru. This sort of grammar is straight out of Claude: > A human on a VPN sees one CAPTCHA and passes, but a headless browser fleet sees a wall. I do wish we’d stop complaining about em dashes though, that’s lazy criticism. The sen…