Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

71–80 of 453 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#71

Post by marginalia_nu »

jerf wrote: post #55
storus wrote: post #9

Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.

"A $5/month VPS won't cut it anymore." Are you speaking from experience, or inferring from articles like this? I serve a static site on the lowest Linode $5/month VPS and it is grotesquely overprovisioned for that use case. It is not the case that every site is getting slammed every second by hundreds of requests per second. Now, if you have some sort of dynamically-computed website that is generated by a slow script…

It's almost always the DBMS that's the bottleneck when websites drop from traffic. As long as you don't do anything fancier than primary key look-ups you're probably fine though.

I survived handling the search search traffic generated by this thread[1] on PC hardware off a residential broadband connection without any sort of degradation. Only time I've gone offline from traffic was when Elon Musk tweeted a link to one of my blog posts, and that was just a short temporary blip.

[1] https://news.ycombinator.com/item?id=28550764

Re: A year of fighting scrapers on my 1.5 million-page website

#72

Post by speak_plainly »

storus wrote: post #9

Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.

Cloudflare offers a free plan that's fantastic. The free plan gives you effectively unlimited DNS/CDN traffic for a normal site, while the main practical cap is 100,000 Worker invocations per day, with 10 ms CPU per Worker request and a 100 MB request body limit. (If that means anything to you).

The next tier up from free is $25/month or $240 per year.

https://www.cloudflare.com/plans/ https://www.cloudflare.com/plans/free/

Re: A year of fighting scrapers on my 1.5 million-page website

#73

Post by nickgray »

tarr11 wrote: post #57

> My normal bill for running this whole site is around $90 a month. During one bad spike month, it jumped about 500%. This is D1 - which has very surprising costs. you may just want to drop D1 and move to a static site. There’s no reason your site should cost this much.

Thank you! I need to tighten up my KV compression, which is actually carrying a lot of D1's load otherwise. We also had some bad queries some months, as the pages and database grew, that were counting the wrong things (or extremely inefficiently) and those have since been fixed.

Re: A year of fighting scrapers on my 1.5 million-page website

#74

Post by drum55 »

petu wrote: post #67
Galanwe wrote: post #49

Earlier quoted context omitted.

The point of PoW access is not that its hard to bypass, it's that you cannot bypass it at scale.

If algorithm used is static and GPU-friendly, then what stops bypass at scale?

It's more or less designed for it, it's SHA256 with a break in the middle for midstate compression to be effective, and the difficulty system is based on a misunderstanding of how bitcoin PoW works ("number of zeros" is never, ever a consideration in bitcoin, it's a match to a floating point target).

sha256(challenge + ascii(nonce)) means that the first compression round of the function can be cached and the second compression round is just the nonce plus the cache. This is the same trick used in Bitcoin mining and would have been avoidable by putting the nonce first, so immediately any non-naive code has to do half the proof of work as the vanilla solver.

Re: A year of fighting scrapers on my 1.5 million-page website

#75

Post by [deleted] »

righthand wrote: post #18

> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer. That's usually the easiest way I spot a bot. > And they were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch. > But those 4,000 bots were just the warm-up. I just hate this style…

[deleted]

Re: A year of fighting scrapers on my 1.5 million-page website

#76

Post by smolder »

gfaster wrote: post #38
RattlesnakeJake wrote: post #32

Earlier quoted context omitted.

I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.

I think that's partly why they do it? If you care about that, you should pay?

Or just have an LLM write the same thing for next to nothing...

Re: A year of fighting scrapers on my 1.5 million-page website

#77

Post by youaremidwit »

gfaster wrote: post #38
RattlesnakeJake wrote: post #32

Earlier quoted context omitted.

I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.

I think that's partly why they do it? If you care about that, you should pay?

[dead]

Re: A year of fighting scrapers on my 1.5 million-page website

#79

Post by strenholme »

storus wrote: post #9

Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.

I have a low cost VPS (actually two in two different pre-AI datacenters) and it runs fine. I use nginx to serve the web pages, and the content is about 99% static content.

The vBulletin and PHPbb style forums have issues with slowdown (I haven’t had a forum since 2015; even back then those forums were overrun with spambots), but static content on a nginx site can be served lightning fast.

Re: A year of fighting scrapers on my 1.5 million-page website

#80

Post by drum55 »

gum_wobble wrote: post #41
drum55 wrote: post #31

Earlier quoted context omitted.

Which is trivially bypassed by an actual implementation of the proof of work in non-javascript, rendering it absolutely useless. The website is approximately 3800x times slower than native code, and hundreds of thousands of times slower than the CUDA kernel claude wrote. The "proof of work" is just non existent at that point, they're solved in milliseconds for what would take the browser version 10 minutes or more, i…

how so, can you link to any sources?

The prompt used for Opus 4.8 was:

    write a implementation of the anubis proof of work in native c code, optimized for speed above all else. use every trick available to make the proof of work as efficient and fast as possible, including modern processor tricks on the x86 platform. your code should avoid using external libraries where possible, include tests, and be readable and concise. a reference for what needs to be met is in this repository. https://github.com/TecharoHQ/anubis
Then

    let’s develop this more. turn this solver into a local HTTP server that can be given work in the request, and it returns solved work. make an end to end tester that sends test work to the solver and waits for a valid response. add support for solving with a GPU using cuda.
Then it was done more or less, it happily made a local server that supports solving the challenges given to it in bulk with priority based queue and can tolerate potentially tens of thousands of requests a second with no issue. The CPU time spent solving the challenges is less than the SSL setup for the connections. The GPU version does in excess of 20GH/s (but with high latency) though I didn't really test it, I'm not using this for anything but proving a point that the LLM itself can write the bypass tools and run them happily.
Post reply on HN