A year of fighting scrapers on my 1.5 million-page website • Hacker News Board

Re: A year of fighting scrapers on my 1.5 million-page website

#231

Post by autoexec » Fri, Aug 07, 2026, 4:56 PM UTC

aeddZX_0 wrote: post #116

jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

How do you monetise bot traffic?

You take your site off the public internet and paywall it off. Websites on the open internet should be accessible and ideally the goal should be to share something cool with the world, not just to make yourself rich.

Re: A year of fighting scrapers on my 1.5 million-page website

#232

Post by veeti » Fri, Aug 07, 2026, 4:56 PM UTC

hk__2 wrote: post #188

elorant wrote: post #147

Earlier quoted context omitted.

What if I’m a web alerts company? I crawl your content but my clients are all end users who actually see your content at your site.

Fetch my RSS then.

Cloudflare and other WAF blocking is increasingly an issue with RSS feeds, especially if you use a centralized reader.

Re: A year of fighting scrapers on my 1.5 million-page website

#233

Post by AlienRobot » Fri, Aug 07, 2026, 4:57 PM UTC

hskalin wrote: post #112

ddxv wrote: post #10

I'm in a similar boat. Probably 99.999% is bots. I have nearly 1m unique "visitors" according to cloudflare and my real users are in the dozens a day. That being said, I love the open internet and am holding on to keeping as much open as I can.

But what do these bots gain from this?

Have you ever use an AI chatbot? It's basically a democratized scraper.

Type something stupid, it "visits" 200 websites and regurgitates some random gibberish.

Re: A year of fighting scrapers on my 1.5 million-page website

#234

Post by miladyincontrol » Fri, Aug 07, 2026, 4:57 PM UTC

RattlesnakeJake wrote: post #32

johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.

I'd argue its rather functioning exactly as intended with regards to obtaining paid users.

Also this view seems a bit elderly. For most towards the end of the millennial curve and younger, anime is no longer subculture, its just general culture at this point. Although I would agree it isnt necessarily what you'd want for every platform and web presence.

Re: A year of fighting scrapers on my 1.5 million-page website

#235

Post by dzonga » Fri, Aug 07, 2026, 4:59 PM UTC

yoursred wrote: post #14

dzonga wrote: post #6

blocking by geo yeah might work - but what happens when someone is traveling abroad ? they've to use a VPN to access your site ? my take with all the bots - the web is gonna be a bunch of private walled gardens. with most sites set to no index. you will only discover them via referral from someone real.

What happens when someone is from a shit country?

good question ? I worry about that too since I'm from a poor/shit country that I travel to constantly & live for half the time !!

Re: A year of fighting scrapers on my 1.5 million-page website

#236

Post by Bender » Fri, Aug 07, 2026, 4:59 PM UTC

IMSAI8080 wrote: post #224

brownieman1325 wrote: post #202

I recently got a surge from Singapore and heard a lot more peeps in my circle of friends saying the same thing...

It'll be the Chinese. The big Chinese cloud providers have data centres in Singapore for some reason. My Chinese mobile phone sends all it's telemetry back to Singapore, not mainland China.

for some reason

It's outside the great firewall. No requirements to hand over SSL keys to China. I worked for companies that ran into these challenges and solutions.

Re: A year of fighting scrapers on my 1.5 million-page website

#237

Post by aprdm » Fri, Aug 07, 2026, 5:03 PM UTC

sdellis wrote: post #219

aprdm wrote: post #192

Earlier quoted context omitted.

Why do you feel that ?

Because AI cannot be trusted as a reliable source of information. Information is more trustworthy when it comes from from primary sources on the open web.

Why not ? Can google be trusted ? Or facebook ? How do people have been accessing the internet for the last decade you reckon and how's this any different ?

Re: A year of fighting scrapers on my 1.5 million-page website

#238

Post by IMSAI8080 » Fri, Aug 07, 2026, 5:04 PM UTC

bediger4000 wrote: post #105

I think that asymmetry is why scraping keeps getting worse. The scrapers' costs fell faster than everyone's defenses improved. There it is. Just like the fckn spammers who ruined SMTP email, scrapers externalize the costs. Who finances the effort to use residential proxies? That takes a lot of effort, even if it's shoddy

There was a popular pirate streaming stick sold on Amazon that ran a residential proxy by day and did ad fraud clicks by night.

Re: A year of fighting scrapers on my 1.5 million-page website

#239

Post by brazukadev » Fri, Aug 07, 2026, 5:04 PM UTC

RattlesnakeJake wrote: post #32

johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.

that's really smart of them. Just pay to use the service you need.

Re: A year of fighting scrapers on my 1.5 million-page website

#240

Post by Bender » Fri, Aug 07, 2026, 5:06 PM UTC

spockz wrote: post #173

Bender wrote: post #97

Earlier quoted context omitted.

I thought about that but to your point CSS is cachable. In fact I made mine immutable. No I just visually spot patterns and use that to study other facets of the agent, other client headers or lack thereof, supported protocol, accepted encoding and so on.

Maybe it is enough to include some css/js which is served without cache and is loaded after all user visible css/js is loaded. Make it small enough to not cause too much bandwidth for the server and legitimate clients. Then anyone who doesn’t hit that CSS file gets banned.

Possibly. I've noticed however that if people default to reader view they don't always pull the CSS. Seems to vary by client software. I'm weird enough to just stick with visual inspection and add characteristics when botters get too clever. Playing with the botters is almost as fun as Megabonk.