Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

11–20 of 449 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#11
> There's a real conflict here. I want Google and Bing and DuckDuckGo to crawl my site and send me new readers. But I don't want everyone else strip-mining it.

Kinda sounds like we're missing a peer to peer network here.

Instead of downloading the same data over and over again we can just download it once and then share it.

Wouldn't that be better for everyone involved?

It would also function as a distributed WayBack Machine, in case anything ever happens to the Internet Archive. (Which I think is desperately needed, bot apocalypse aside.)

Re: A year of fighting scrapers on my 1.5 million-page website

#14
post #6

blocking by geo yeah might work - but what happens when someone is traveling abroad ? they've to use a VPN to access your site ? my take with all the bots - the web is gonna be a bunch of private walled gardens. with most sites set to no index. you will only discover them via referral from someone real.

What happens when someone is from a shit country?

Re: A year of fighting scrapers on my 1.5 million-page website

#17
post #9

Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.

All of my boxes are cheap VPS. Highly recommend people throw Cloudflare in front of their stuff. I switched all of my load balancers over to there and all of that bot crap went away. That combined with a proper UFW setup keeps the weather clear for me.

Re: A year of fighting scrapers on my 1.5 million-page website

#18
> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer. That's usually the easiest way I spot a bot.

> And they were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch.

> But those 4,000 bots were just the warm-up.

I just hate this style of writing like you're on Twitter. Why does the above need to be 3 different paragraphs? A paragraph break indicates a separate thought but the author is still talking about the same data and still making their point. The sentence "But those 4,000 bots were just the warm-up." is effective when still the last line of a paragraph and it signals respect for your readers. I stopped reading after this because it's just a terrible reading experience.

Here's a correct version that doesn't read like the author left for a week to think about what the next sentence would be or having some sort of anxiety-induced mental pause:

> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer, which is usually the easiest way I spot a bot. They were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch. But those 4,000 bots were just the warm-up.

Re: A year of fighting scrapers on my 1.5 million-page website

#19
post #11

> There's a real conflict here. I want Google and Bing and DuckDuckGo to crawl my site and send me new readers. But I don't want everyone else strip-mining it. Kinda sounds like we're missing a peer to peer network here. Instead of downloading the same data over and over again we can just download it once and then share it. Wouldn't that be better for everyone involved? It would also function as a distributed WayBack…

The problem there is trust

Re: A year of fighting scrapers on my 1.5 million-page website

#20
post #11

> There's a real conflict here. I want Google and Bing and DuckDuckGo to crawl my site and send me new readers. But I don't want everyone else strip-mining it. Kinda sounds like we're missing a peer to peer network here. Instead of downloading the same data over and over again we can just download it once and then share it. Wouldn't that be better for everyone involved? It would also function as a distributed WayBack…

Common Crawl?

https://commoncrawl.org/

Post reply on HN