Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

51–60 of 453 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#51
post #31

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

Which is trivially bypassed by an actual implementation of the proof of work in non-javascript, rendering it absolutely useless. The website is approximately 3800x times slower than native code, and hundreds of thousands of times slower than the CUDA kernel claude wrote. The "proof of work" is just non existent at that point, they're solved in milliseconds for what would take the browser version 10 minutes or more, i…

It is not absolutely useless, empirically, which you'd discover if you had a website getting hammered by bots and experimented with anubis as a countermeasure.

While dedicated scrapers/attackers could work around it, and they could do so much more efficiently than the client-side js, almost none of them do. Unless you like paying additional hosting resource fees to serve bots, it's a worthwhile option, and is less annoying to typical human visitors than cloudflare's interactive captcha/challenge which is what most people use.

The main author is aware that the algorithm is far from ideal for this purpose. See https://news.ycombinator.com/item?id=48869064 . If more bots start to answer the primitive challenge anubis uses now, that'll hasten implementation of a different algorithm.

Don't let the perfect be the enemy of the good enough. For now, the algorithm or challenge scheme almost doesn't matter. Since it's much smaller-scale than cloudflare's challenges, that's probably why very few scrapers and botnets bother to solve anubis's trivial sha2 pow.

Targeted attacks may not be repelled at all. That's not the point.

Re: A year of fighting scrapers on my 1.5 million-page website

#52

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.

If you 'don't mesh' with that then you are not worth protecting anyway :)

Re: A year of fighting scrapers on my 1.5 million-page website

#53
post #24

Earlier quoted context omitted.

Not the OP, but because it appears structured just like AI output? (This isn't a condemnation. AI can often do a better job of representing thoughts than humans.) Examples: - The article organization - The general language flow - The bullet points with a bolded gist, a colon, and then elaboration (and bonus with details stats). - The images are almost certainly AI generated. They look AI generated.

That's the thing: even if you don't use AI, if most of what you read is AI, soon this is what you'll sound like, just because it's so much of the training data your own brain's model has. Using it slowly sucks the uniqueness out of you.

Using AI to write blogs doesn’t bother me in principle, but I hate the prose that gets left in. Even if it’s not AI writing, being human doesn’t give you a pass for writing like a self-help guru. This sort of grammar is straight out of Claude:

> A human on a VPN sees one CAPTCHA and passes, but a headless browser fleet sees a wall.

I do wish we’d stop complaining about em dashes though, that’s lazy criticism. The sentence structure is far worse, and a big tell is subheadings that are all variants of “The ”.

Re: A year of fighting scrapers on my 1.5 million-page website

#54
post #11

> There's a real conflict here. I want Google and Bing and DuckDuckGo to crawl my site and send me new readers. But I don't want everyone else strip-mining it. Kinda sounds like we're missing a peer to peer network here. Instead of downloading the same data over and over again we can just download it once and then share it. Wouldn't that be better for everyone involved? It would also function as a distributed WayBack…

Anyway, Google doesn't send traffic to your site any more. Only important sites and obvious scams seem to get indexed.

Re: A year of fighting scrapers on my 1.5 million-page website

#55
post #9

Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.

"A $5/month VPS won't cut it anymore."

Are you speaking from experience, or inferring from articles like this?

I serve a static site on the lowest Linode $5/month VPS and it is grotesquely overprovisioned for that use case. It is not the case that every site is getting slammed every second by hundreds of requests per second.

Now, if you have some sort of dynamically-computed website that is generated by a slow scripting language that is poorly optimized and hits the database too many times for a single page, yeah, it doesn't take many RPS to take you out. But that isn't the only option; it's the slowest of the slow options. Realistic, there are plenty of sites that match that description, but I concatenated that many clauses on purpose. Drop any one of them and your personal site will be fine.

Re: A year of fighting scrapers on my 1.5 million-page website

#56
post #9

Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.

A $5/month VPS should cut it completely fine unless you're doing something very complicated.

Re: A year of fighting scrapers on my 1.5 million-page website

#57
> My normal bill for running this whole site is around $90 a month. During one bad spike month, it jumped about 500%.

This is D1 - which has very surprising costs. you may just want to drop D1 and move to a static site. There’s no reason your site should cost this much.

Re: A year of fighting scrapers on my 1.5 million-page website

#58
post #9

Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.

Yeah, you just make sure your site is fast enough to handle more than 1 RPS.

But if you want them to actually stop, you can also just serve a little JavaScript page that sets a cookie and refreshes, to anyone who hasn't set the cookie. The DDOS attacker doesn't run JavaScript.

Re: A year of fighting scrapers on my 1.5 million-page website

#59
post #17
post #9

Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.

All of my boxes are cheap VPS. Highly recommend people throw Cloudflare in front of their stuff. I switched all of my load balancers over to there and all of that bot crap went away. That combined with a proper UFW setup keeps the weather clear for me.

Highly do not recommend centralising the internet.

Re: A year of fighting scrapers on my 1.5 million-page website

#60
post #9

Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.

Static files on literally any hardware from the last 15 years on modern server software simply won't get cooked by bots. The network switch will bottleneck you before the server will. Your ephemeral port range will run out before the server will.
Post reply on HN