Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

331–340 of 453 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#331

Post by mostlysimilar »

basilikum wrote: post #323
mostlysimilar wrote: post #291

Earlier quoted context omitted.

How do you figure it hurts real users? The amount of compute/energy used on the proof of work is pretty minimal. You're using more when you watch a YouTube video or browse a JS-heavy web app. Of course a sophisticated scraper can "figure out" how to bypass. It isn't trying to be foolproof, it's adding an extra cost to deter massive amounts of bot traffic. I put it in front of my hobby project because I can't afford t…

1. It makes the web more annoying and time wasting. 2. I'm a real user and I have to enable JavaScript and run arbitrary programs from untrusted websites to access them. One browser that I use doesn't even support JavaScript (w3m). Fortunately it is unaffected because it doesn't have the string "Mozilla" in its UA and Anubis lets every user without that pass to not hurt legitimate bots like search engine crawlers. As…

I empathize, I hate the bloated JS-heavy web and avoid it when I can. I just don't see a better solution. The options for me are: sacrifice some legit users by using Anubis, or sacrifice all users by shutting down my service because I can't afford to serve the bot traffic.

Re: A year of fighting scrapers on my 1.5 million-page website

#332

Post by godwinson__4-8 »

hk__2 wrote: post #117
jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

"Bot" access on behalf of users should be fine, even preferred. The missing piece is some micro transaction layer and some sort of attestation somewhere in the layer that the person driving the bot is not a bad actor.

Equating bot with bad actor in 2026 is Luddite behavior. Driving people to your site so you can serve them adspam or just make whatever operation they want to do 10x more difficult is the same.

MCP style APIs should eat the web. This doesn't mean the "open" web goes away. > 99% of the time I don't care for a domains particular FE at all. It's a complete waste not only of time, but resources and bloat. Pushing the contract into the agent should become good UX. Making things harder for good faith users should never be the goal.

Re: A year of fighting scrapers on my 1.5 million-page website

#333

Post by lefra »

pwillia7 wrote: post #212
hk__2 wrote: post #117

Earlier quoted context omitted.

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

There is bot traffic initiated by a human and bot traffic not initiated by a human. I would want to serve the first but not the second if it impacted my cost/performance at all.

Isn't all bot traffic ultimately initiated by a human? Someone plugged the computer in and gave it instructions. It may result in one http request or billions of them, but the human is still the initiator.

Re: A year of fighting scrapers on my 1.5 million-page website

#334

Post by baby_souffle »

wredcoll wrote: post #298
baby_souffle wrote: post #270

Earlier quoted context omitted.

> I would love some pointers to read up on this, what are some examples of this? Genuinely, not being a troll or ironic There's a "look what I built!" thread every few days on the /r/homeassistant sub and the home assistant forums, too. The data sources vary but it's usually a mix of devices that the user has locally and some remote data sources like weather/public transit schedules APIs or RSS and in some cases also…

It's an interesting example. People did very similar things during the "desktop shell modification" scene in the early 2000s, by far the most common thing people modified their desktop shell to do was display a bunch of charts and graphs of semi-random information, usually about their local computer. It was useless in 99.9% of the cases. You don't need or want to have constant access to how much free ram your compute…

> Stuff like calendar appointments would count, knowing you have things scheduled at specific moments or tasks to do, go grocery shopping, etc, todo lists in otherword

So ... most people.

> That's basically all self-generated stuff by you about you, you don't really need any outside data sources.

Other than the ones you just enumerated that most people are likely to use, I need to know - using external sources:

- When to expect the garbage truck (do I put the recycle out tonight, too?)

- Is it a school holiday?

- When it's cheapest for me to charge the car up; if there are any "grid stress, please turn non-essentials off or we're getting rolling blackouts" alerts.

- Same goes for nearby fires or other issues that are going to change the "when do you open/close the windows and switch to AC" math.

- If it's likely to rain in the next few days so I change what plants get watered / when

- Packages that I am likely to get in the post today

- Weather (for display, not just for use in the automations mentioned above)

- Where my power provider is currently getting its power from (so I can calculate estimated CO2 emissions, if any)

And that's about 1/2, probably. Can't be bothered to go spend 20 min pouring through my HA config to build the full list.

I also stream the npr newscast as an MP3 as part of the "good morning, while you wait for coffee..." automations, if that counts for anything.

Re: A year of fighting scrapers on my 1.5 million-page website

#335

Post by [deleted] »

hluska wrote: post #206
matsemann wrote: post #171

Earlier quoted context omitted.

I don't disagree, but there is a sliding scale here. For instance, I wanted to buy a piece of equipment the other day from a local company for a specific usecase. I wanted to find a specific price/weight/specs ratio, and asked an llm to loop through the 20 or so items, fetch their page and calculate and present some values for each. This then led to me going and buying the one I found. So the llm was mainly just an e…

I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. They clearly don’t mind so what’s the issue? You still used a bot.

[deleted]

Re: A year of fighting scrapers on my 1.5 million-page website

#336

Post by duskwuff »

vablings wrote: post #104
RattlesnakeJake wrote: post #32

Earlier quoted context omitted.

I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.

It's actually a genius idea. If you are someone who the professional presentation of not having an anime girl on the loading page is required, then you can afford to fork over the cash to fund development.

Per personal communication, this is precisely why the developer did that.

Re: A year of fighting scrapers on my 1.5 million-page website

#337

Post by snailmailman »

zuzululu wrote: post #302
vablings wrote: post #104

Earlier quoted context omitted.

It's actually a genius idea. If you are someone who the professional presentation of not having an anime girl on the loading page is required, then you can afford to fork over the cash to fund development.

I guess the disconnect here is a bunch of HN'ers believing professional companies and websites want to attach their branding to a sexualized anime character and that they are willing to pay to remove it. Which one then wonders why they would install it in the first place.

Sexualized? It’s just a cartoon/anime character holding a magnifying glass

Re: A year of fighting scrapers on my 1.5 million-page website

#338

Post by pwillia7 »

lefra wrote: post #333
pwillia7 wrote: post #212

Earlier quoted context omitted.

There is bot traffic initiated by a human and bot traffic not initiated by a human. I would want to serve the first but not the second if it impacted my cost/performance at all.

Isn't all bot traffic ultimately initiated by a human? Someone plugged the computer in and gave it instructions. It may result in one http request or billions of them, but the human is still the initiator.

Yeah but you know what I mean -- A person specifically interacting with my brand vs anthropic hitting all sites 100000 times a day

Re: A year of fighting scrapers on my 1.5 million-page website

#339

Post by duskwuff »

drum55 wrote: post #153
RobotToaster wrote: post #124

Earlier quoted context omitted.

> Some browsers also intentionally detect attempts to do proof of work and attempt to block it entirely. Then they'd be blocking themselves from the website.

https://blog.mozilla.org/en/firefox/block-cryptominers-with-... To help you combat these unwelcome scripts, we’ve partnered with Disconnect and compiled lists of domains that serve cryptomining scripts so you can block them directly through Firefox. When you download Firefox, tracking protection is enabled by default, blocking known cryptominers.

Technically true, but the list is mostly nonfunctional - cryptominers are basically extinct in the wild, and most of the domains on the list have expired or no longer serve mining scripts.

Re: A year of fighting scrapers on my 1.5 million-page website

#340

Post by rsolva »

I made a small booking site for a local dutch canal boat, which has a calender function. A simple PHP app. I checked the Apache logs recently, and it had THOUSANDS of claudebot and other AI UserAgents flooding the logs every day, apparently because the scrapers keep hitting the 'next month' button on the calendar in a an infinite loop, all day, everyday!

This is a small booking app without any useful information at all, it surprises me that the AI boots have no discernment about what the are scraping, just wasting their own and other peoples resources. And their own reputation! You would thing they could spare a few tokens on a classifier model to do a quick evaluation of their scraping efforts, but apparently they do not.

Anyway, I have done my best to block these UAs and so far it seems to have improved the situation.

Post reply on HN