Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

61–70 of 454 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#61

Post by inigyou »

Venn1 wrote: post #39

I'm blocking the Amazon search crawler, anything coming from Googleusercontent, and limiting AI crawlers to search rather than allowing AI assistants. The residential proxy waves are something to behold, but Cloudflare does an okay job catching those in the AI labyrinth. Still, it's all a bit silly, and I can't imagine what large sites deal with when I'm tangoing with this much nonsense on a small tech blog.

Don't. Just serve the page unless it's an unusually expensive page to serve (like search).

Re: A year of fighting scrapers on my 1.5 million-page website

#62

Post by sp1982 »

The annoying part is a large percentage of misbehaving bots (not obeying robots.txt for example) are via end user proxies across the world. However most of these aren't doing full-browser loop, so if you are behind cloudflare, you can do non-interactive challenge and that can help quite a bit.

Re: A year of fighting scrapers on my 1.5 million-page website

#64

Post by nickgray »

righthand wrote: post #18

> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer. That's usually the easiest way I spot a bot. > And they were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch. > But those 4,000 bots were just the warm-up. I just hate this style…

Hey! I'm the OP - thanks for feedback on my writing style. I went ahead and fixed this in the article. It should be updated by the time you read this:

https://patronview.com/news/99-percent-of-my-website-traffic...

And you're totally right: I mostly post on X (nee Twitter) and I probably have ADHD or just a low attention span, so I prefer to read things broken up into paragraphs. But for a smarter audience like this, and that reads long-form blog posts, I should tighten it up.

Thank you for the suggestion. LMK any other edits and I'll be happy to tighten it up.

Re: A year of fighting scrapers on my 1.5 million-page website

#65

Post by tsunamifury »

1bpp wrote: post #52
RattlesnakeJake wrote: post #32

Earlier quoted context omitted.

I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.

If you 'don't mesh' with that then you are not worth protecting anyway :)

I hope I’m missing the sarcasm here…

Re: A year of fighting scrapers on my 1.5 million-page website

#66

Post by inigyou »

tarr11 wrote: post #57

> My normal bill for running this whole site is around $90 a month. During one bad spike month, it jumped about 500%. This is D1 - which has very surprising costs. you may just want to drop D1 and move to a static site. There’s no reason your site should cost this much.

There's a lot of people who host their site at extremely expensive places and then do everything they can to minimise unneeded traffic - instead of just moving to a cheaper host. Vercel is another popular extremely expensive host.

Re: A year of fighting scrapers on my 1.5 million-page website

#67

Post by petu »

Galanwe wrote: post #49
drum55 wrote: post #31

Earlier quoted context omitted.

Which is trivially bypassed by an actual implementation of the proof of work in non-javascript, rendering it absolutely useless. The website is approximately 3800x times slower than native code, and hundreds of thousands of times slower than the CUDA kernel claude wrote. The "proof of work" is just non existent at that point, they're solved in milliseconds for what would take the browser version 10 minutes or more, i…

The point of PoW access is not that its hard to bypass, it's that you cannot bypass it at scale.

If algorithm used is static and GPU-friendly, then what stops bypass at scale?

Re: A year of fighting scrapers on my 1.5 million-page website

#68

Post by ihuman »

qbane wrote: post #4

> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.

There's a difference between someone running a scraping tool occasionally and bots constantly and rapidly re-scraping the same site over and over again

Re: A year of fighting scrapers on my 1.5 million-page website

#69

Post by FerretFred »

Sigh .. same here. I don't write blog posts often (enough) but the ones I do write are from personal experiences and I take a lot of care with them. I look at my logs snd see bots everywhere, but now I just let them get on with it. AI scrapers are different though; they get to read my content which, just for them contains a smsttering of finest digital toxin. A pox on your datasets!
Post reply on HN