Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

171–180 of 450 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#171

Post by matsemann »

hk__2 wrote: post #117
jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

I don't disagree, but there is a sliding scale here.

For instance, I wanted to buy a piece of equipment the other day from a local company for a specific usecase. I wanted to find a specific price/weight/specs ratio, and asked an llm to loop through the 20 or so items, fetch their page and calculate and present some values for each. This then led to me going and buying the one I found. So the llm was mainly just an extension of me clicking into every page and making a spreadsheet myself.

However, if it were to continuously poll, or just scrape or something with no intention of buying, I would be no better than a bot.

Re: A year of fighting scrapers on my 1.5 million-page website

#172

Post by [deleted] »

elorant wrote: post #147
hk__2 wrote: post #117

Earlier quoted context omitted.

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

What if I’m a web alerts company? I crawl your content but my clients are all end users who actually see your content at your site.

[deleted]

Re: A year of fighting scrapers on my 1.5 million-page website

#173

Post by spockz »

Bender wrote: post #97
spockz wrote: post #91

Earlier quoted context omitted.

Is there some existing mechanism already that counts how often an ip only scrapes the page and not the css and then block those origin IPs if it occurs “too often”? Unfortunately, the best practice is to make css cacheable so you need to keep long histories.

I thought about that but to your point CSS is cachable. In fact I made mine immutable. No I just visually spot patterns and use that to study other facets of the agent, other client headers or lack thereof, supported protocol, accepted encoding and so on.

Maybe it is enough to include some css/js which is served without cache and is loaded after all user visible css/js is loaded. Make it small enough to not cause too much bandwidth for the server and legitimate clients. Then anyone who doesn’t hit that CSS file gets banned.

Re: A year of fighting scrapers on my 1.5 million-page website

#174

Post by doc_ick »

Terretta wrote: post #150
hk__2 wrote: post #117

Earlier quoted context omitted.

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

you know what a browser is called by the web site? check the header that tells the version. USER AGENT not user, an agent on behalf of the user. the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT. for a glimpse at one possible…

[dead]

Re: A year of fighting scrapers on my 1.5 million-page website

#175

Post by doc_ick »

ballooney wrote: post #169
ako wrote: post #138

Earlier quoted context omitted.

The best way to read the information on the internet today is via a LLM. Just like you don't process raw data in a data warehouse or datalake by looking at tables, you use a SQL or BI tool, to process all the information on the internet you need to tool to digest it for you. For many, today that tool is a LLM chat interface or agent.

This is such a grim thing to read.

100%

Re: A year of fighting scrapers on my 1.5 million-page website

#176

Post by matsemann »

speak_plainly wrote: post #72
storus wrote: post #9

Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.

Cloudflare offers a free plan that's fantastic. The free plan gives you effectively unlimited DNS/CDN traffic for a normal site, while the main practical cap is 100,000 Worker invocations per day, with 10 ms CPU per Worker request and a 100 MB request body limit. (If that means anything to you). The next tier up from free is $25/month or $240 per year. https://www.cloudflare.com/plans/ https://www.cloudflare.com/plan…

I use this. Buy a domain, set up a cloud flare pages that builds an Astro page on git push, served statically for free.

Re: A year of fighting scrapers on my 1.5 million-page website

#178

Post by oaw93j4oij »

I despise cloudflare. They've decided that my home IP address is bad, so I have to capchas for most websites. Sometimes on infinite loop and I never get to the website. I even reset my home IP address more than once, but it instantly continues. Especially if I use any VPN, even my work VPN.

Re: A year of fighting scrapers on my 1.5 million-page website

#179

Post by righthand »

nickgray wrote: post #64
righthand wrote: post #18

> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer. That's usually the easiest way I spot a bot. > And they were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch. > But those 4,000 bots were just the warm-up. I just hate this style…

Hey! I'm the OP - thanks for feedback on my writing style. I went ahead and fixed this in the article. It should be updated by the time you read this: https://patronview.com/news/99-percent-of-my-website-traffic... And you're totally right: I mostly post on X (nee Twitter) and I probably have ADHD or just a low attention span, so I prefer to read things broken up into paragraphs. But for a smarter audience like this,…

Glad to hear you’re willing to accept feedback. You maybe don’t have ADHD and sorry for continued advice but you shouldn’t assume you have undiagnosed conditions IMO as it allows you to defer your mistakes from the self. Even if you do have ADHD you can still correct and understand good article structure. I highlighted the last sentence of that paragraph because what is clear from the split sentence style is that you’re writing for impact, this lends well to 140 characters but falls apart in longer form writing but as I stated the sentence is still impactful as you’re saying “but wait…there’s more to this!” Which is very intriguing. I think if you’re comfortable writing that way and it helps you split your ideas and sentences up so each one is impactful, that’s a good thing. But consider that style as a draft and then you can go back and group up your impactful ideas into paragraphs very easily.

For other edits I think my qualm applies to other parts of the article but I found that specific paragraph the most impactful way to illustrate what I was talking about. I leave the rest to you as a challenge. Don’t lose sleep over it, there will be more writing in the future to apply it to.

As for the ADHD stuff and the urge to self diagnose consider something less severe but similar symptoms. Have you considered VAST? Here is a good HN comment briefly detailing it and mentioning a book (titled ADHD 2.0 I believe) that may be more in line. I am not a doctor of course and VAST is rather new.

https://news.ycombinator.com/item?id=49035436

Anyways I will finish reading your article now since you’re so wonderful to take a bit of feedback and be proactive.

Re: A year of fighting scrapers on my 1.5 million-page website

#180

Post by [deleted] »

skinfaxi wrote: post #144
hk__2 wrote: post #117

Earlier quoted context omitted.

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

I assume you are unfamiliar with the concept of user agents? Otherwise your browser would count as a bot user no? And if not, what if it was a custom browser and not Chrome/Firefox/Edge?

[deleted]
Post reply on HN