Did he really say that the VPS would be in trouble if it received 5 requests per second of mostly static pages? Something that currently costs $90 per month to serve through CloudFlare?
A year of fighting scrapers on my 1.5 million-page website
381–390 of 451 posts
Re: A year of fighting scrapers on my 1.5 million-page website
#382> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer. That's usually the easiest way I spot a bot. > And they were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch. > But those 4,000 bots were just the warm-up. I just hate this style…
I write a lot of my comments and emails like this.
It’s a learned habit - I’m more apt to receive a thoughtful reply (or any) in this format.
Re: A year of fighting scrapers on my 1.5 million-page website
#383The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user You are mistaking yourself, well your bot, as his target audience. You might as well say “If I want to send you my commercial email, and you block it, you hurt me, the email user.”. While your point of being concerned about cloudflair becoming a global arbiter of who…
My understanding would be that, if an API is available, then bots are essentially welcome. If no API is available and the website has to be 'visited' then it seems that it's intended for human consumption 'the old fashioned way'.
Re: A year of fighting scrapers on my 1.5 million-page website
#384Earlier quoted context omitted.
Sometimes the cure is worse than the disease.
True, but for whom? The cure here isn't perfect, but much better than the disease of paying hundreds of dollars a month for scrapers which will never be beneficial. Worse for the scrapers really isn't anyone's problem but the scrapers'.
Some of us are labelled as bots, much like dolphins getting caught in fishing nets. I guess it's not material since it's not life or death (yet? if access to essential services is gated by bot detection we are all screwed).
Re: A year of fighting scrapers on my 1.5 million-page website
#385Earlier quoted context omitted.
I suppose it would depend, yeah. I find having paying customers being blocked to be completely unacceptable. Not only is a bad customer experience, but they tend to be very vocal about it. Posting online, leaving bad reviews, etc.
Having to serve 1000x more traffic from bots is completely unacceptable for most site owners as well. The fact that this trade-off isn't obvious indicates you haven't tried to run a site at scale and make a living off of it.
You are quite incorrect. I make a living off a website and 99.9% of my traffic is from bots. Literally hundreds of millions of requests from bots. The cost of bots is worth the cost of not harming legit users.
Re: A year of fighting scrapers on my 1.5 million-page website
#386The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.
Writing a script to fetch HTML is no different than writing a web browser. I think it's the scale of the operation that distinguishes bots vs human. The browser is the user's agent, but not the only one.
Re: A year of fighting scrapers on my 1.5 million-page website
#387Earlier quoted context omitted.
> I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. The point is that not all bots are bad. Assuming as much / implementing policies to that effect won't block all bots but it _will_ block the portion of bots that represent users considering giving you money. I have a series of bots that monitor various eCom sites to monitor prices over time for big-ticket items I am consid…
How should they know that your bot belongs to a potential customer and not to a competitor surveilling their prices? Not a rhetorical question, because most companies would welcome that interest from customers.
> How should they know that your bot belongs to a potential customer and not to a competitor surveilling their prices?
That's the thing; they can't, really.
Best we can do is compromise; allow bots and rate limit them. Maybe someday there will be a relatively universal standard for fetching pages akin to the `index.html/index.md` pattern that is emerging now. Put the bot filters on the .html path and some sort of micro transactions system on the .md endpoint.
In basically every cat/mouse game defense is forever one step behind...
Re: A year of fighting scrapers on my 1.5 million-page website
#388Seems about right. I rotated my logs this morning. Most humans go to access.log and most bots go to botpoop.log. This is the line count: 2 access.log [1] 40 botpoop.log [2] 2 is really 1 since a human will grab the CSS file. Most bots do not bother with the style-sheet so that's a 40:1 bots to humans. I could cut that down by blocking data-centers but then I inadvertently block a lot of VPN's which I really don't nee…
More realistic numbers now that HN and others have been visiting all day. Line count:
104 access.log
366 botpoop.log
54 access.log
181 botpoop.logRe: A year of fighting scrapers on my 1.5 million-page website
#389Earlier quoted context omitted.
Bandwidth is cheap at places that don't call themselves "cloud". You can even get unmetered. 10TB/month is usually enough, even with bots. If you're serving large files, put a simple bot test on the page before the files.
> 10TB/month is usually enough, even with bots. My single static webpage with no updates in 3 years is doing that, which is (one of the reasons) how I end up where that site (and many others in business and personally) is. You're chasing a dream for a world that doesn't exist anymore.
no sorry I don't actually believe your single static webpage is doing 10TB/month. That's about 500 RPS average.
Re: A year of fighting scrapers on my 1.5 million-page website
#390The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.
A web browser is an entity that acts on your behalf. That's why it's called a user agent.
They're just better at English now.