Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

101–110 of 450 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#101

Post by jwr »

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see.

A second side effect of a knee-jerk reaction to bots crawling websites is that if you try to fight all bots, you also end up hurting real users that use "bots". If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. That might or might not be what you expected, but it's worth taking into account.

And finally, something worth noting is that there are so many websites whose owners complain about bots, but the real problem is that the website is poorly built and should be improved anyway. Bot traffic is not necessarily bad.

Re: A year of fighting scrapers on my 1.5 million-page website

#102

Post by allthetime »

storus wrote: post #9

Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.

A $5/month VPS is fine for mostly static content. I’ve got a site that has been pulling millions of viewers and sits at the top of google for its niche for a decade. It is now hammered with bot traffic constantly. But everything is cached so it’s fine.

Re: A year of fighting scrapers on my 1.5 million-page website

#103

Post by qbane »

ihuman wrote: post #68
qbane wrote: post #4

> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.

There's a difference between someone running a scraping tool occasionally and bots constantly and rapidly re-scraping the same site over and over again

This is an important context: the site is more likely to be targeted by scrapers because it is a curated collection of scraped information.

Re: A year of fighting scrapers on my 1.5 million-page website

#104

Post by vablings »

RattlesnakeJake wrote: post #32
johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.

It's actually a genius idea. If you are someone who the professional presentation of not having an anime girl on the loading page is required, then you can afford to fork over the cash to fund development.

Re: A year of fighting scrapers on my 1.5 million-page website

#105

Post by bediger4000 »

I think that asymmetry is why scraping keeps getting worse. The scrapers' costs fell faster than everyone's defenses improved.

There it is. Just like the fckn spammers who ruined SMTP email, scrapers externalize the costs.

Who finances the effort to use residential proxies? That takes a lot of effort, even if it's shoddy

Re: A year of fighting scrapers on my 1.5 million-page website

#106

Post by jjgreen »

pixl97 wrote: post #33
mysterydip wrote: post #27

Earlier quoted context omitted.

That’s exactly what an AI prompted to reply to AI accusations would say!

I'm not a bot, you're a bot! So this is how AI wins, humans kill each other off because we might be bots and the bots inherit the earth.

Load-bearing

Re: A year of fighting scrapers on my 1.5 million-page website

#107

Post by leros »

basilikum wrote: post #95
johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

It doesn't detect "real browsers". It simply adds some friction and is niche enough that AI scrapers have not bothered to bypass it yet.

Yeah this is one of those things that hurts some real users and AI scrapers have figured out how to bypass. Not worth it IMO.

Re: A year of fighting scrapers on my 1.5 million-page website

#108

Post by allthetime »

marginalia_nu wrote: post #71
jerf wrote: post #55

Earlier quoted context omitted.

"A $5/month VPS won't cut it anymore." Are you speaking from experience, or inferring from articles like this? I serve a static site on the lowest Linode $5/month VPS and it is grotesquely overprovisioned for that use case. It is not the case that every site is getting slammed every second by hundreds of requests per second. Now, if you have some sort of dynamically-computed website that is generated by a slow script…

It's almost always the DBMS that's the bottleneck when websites drop from traffic. As long as you don't do anything fancier than primary key look-ups you're probably fine though. I survived handling the search search traffic generated by this thread[1] on PC hardware off a residential broadband connection without any sort of degradation. Only time I've gone offline from traffic was when Elon Musk tweeted a link to on…

For a “personal website” or any relatively simple static content site, the DB should not be getting hit much at all. Cache is your friend. For my sites I have a map of all relevant queries and when we do updates we run the relevant queries one time and cache them to both memory and file system.

Re: A year of fighting scrapers on my 1.5 million-page website

#109

Post by creshal »

inigyou wrote: post #66
tarr11 wrote: post #57

> My normal bill for running this whole site is around $90 a month. During one bad spike month, it jumped about 500%. This is D1 - which has very surprising costs. you may just want to drop D1 and move to a static site. There’s no reason your site should cost this much.

There's a lot of people who host their site at extremely expensive places and then do everything they can to minimise unneeded traffic - instead of just moving to a cheaper host. Vercel is another popular extremely expensive host.

I'm always flabbergasted when I see what people pay and how much effort they need to invest to keep their cloud websites from eating them alive.

My allegedly more complicated VPS stack needs an afternoon of attention every two years when a new Debian major release is necessary, and costs have been predictable for 15 years, no matter what happened traffic wise.

Re: A year of fighting scrapers on my 1.5 million-page website

#110

Post by wandr »

I'm working on a web app right now, with the intention of it going to be 100% paywalled. It's 95% complete, but the remaining 5% is just implementing the paywall. In the meantime, the app is live and operational with a fully functional signup. I am deleting about 100 new bot signups per day right now, it is crazy out there.
Post reply on HN