Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

311–320 of 451 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#311

Post by weego »

matsemann wrote: post #171
hk__2 wrote: post #117

Earlier quoted context omitted.

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

I don't disagree, but there is a sliding scale here. For instance, I wanted to buy a piece of equipment the other day from a local company for a specific usecase. I wanted to find a specific price/weight/specs ratio, and asked an llm to loop through the 20 or so items, fetch their page and calculate and present some values for each. This then led to me going and buying the one I found. So the llm was mainly just an e…

No, there isn't, it's a bot. Justifying it with a strawman case study doesn't change it.

This is giving "why is this plane full? What in doing is important but what is everyone else doing here?"

Re: A year of fighting scrapers on my 1.5 million-page website

#312

Post by inigyou »

esseph wrote: post #266
inigyou wrote: post #213

Earlier quoted context omitted.

Until proven otherwise in your specific case, the better technology is just hosting directly on a VPS without Cloudflare. And if your 5$ VPS is maxed out, while serving any less than 10 requests per second, then you need to optimize your software before considering an upgrade.

Okay but the bots are loose and you have a bandwidth cap on most VPSs. Because you have no CDN, all the bots are coming directly to you. Because that's no CDN or distributed WAF, they're hammering your box. Because your IP is directly exposed, you're also showing up on Shodan. Your server may have enough CPU and RAM to handle the load, but does your monthly bandwidth allocation with your provider? Many are finding pr…

Bandwidth is cheap at places that don't call themselves "cloud". You can even get unmetered. 10TB/month is usually enough, even with bots. If you're serving large files, put a simple bot test on the page before the files.

Re: A year of fighting scrapers on my 1.5 million-page website

#313

Post by volkl48 »

baby_souffle wrote: post #261
hluska wrote: post #206

Earlier quoted context omitted.

I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. They clearly don’t mind so what’s the issue? You still used a bot.

> I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. The point is that not all bots are bad. Assuming as much / implementing policies to that effect won't block all bots but it _will_ block the portion of bots that represent users considering giving you money. I have a series of bots that monitor various eCom sites to monitor prices over time for big-ticket items I am consid…

If this is the case, they probably outright do not want you as a customer.

Significant sales (beyond for excess stock clearance/discontinued items) exist largely to generate some excitement from potential customers and to lure them in to taking a look at the site/store and to hopefully get them to buy some extra stuff that's less discounted while they're there.

The actual on-sale item is potentially losing money for the business or at break-even.

------

A person who's running a series of bots like this to track the exact lowest price on the internet for items over time and to always only buy at the lowest price it's ever been sold for....is probably the least likely kind of customer to buy anything extra that's actually profitable for the business.

Re: A year of fighting scrapers on my 1.5 million-page website

#314

Post by qingcharles »

m463 wrote: post #305
jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

I am denied by cloudflare CONSTANTLY on one system. I have an old os (macos 10.11), running the highest firefox esr I can run, and I get denied by cloudflare. But not always immediately - I get to enable javascript/cookies sometimes just to be denied. they are not the folks we want gatekeeping the internet, they are opportunists increasing their OKRs

And here's the rub -- the bots are "running" the latest "MacOS" and have no problem accessing the site.

Re: A year of fighting scrapers on my 1.5 million-page website

#315

Post by PeterHolzwarth »

travisgriggs wrote: post #255

I wonder what the web would be like if we priced bandwidth at the requester point and as you go.

Your comment kind of reminds me of the old slashdot spam-email-solution copypasta - the purpose of it is to highlight how so many sensible sounding ideas just can't work in practice.

Re: A year of fighting scrapers on my 1.5 million-page website

#316

Post by pessimizer »

Terretta wrote: post #150
hk__2 wrote: post #117

Earlier quoted context omitted.

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

you know what a browser is called by the web site? check the header that tells the version. USER AGENT not user, an agent on behalf of the user. the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT. for a glimpse at one possible…

> you know what a browser is called by the web site? check the header that tells the version.

You mean that key whose only legitimate values also call every browser Mozilla? Arguing from the dictionary is so tedious. I'm going to scrape your site, but the idea that you have some obligation to like it because of the definition of the word "agent" is profoundly stupid.

A lot of people live off ads. I block ads. People who live off ads aren't obligated to like it - and are free to take any means to block it without moral hazard.

Re: A year of fighting scrapers on my 1.5 million-page website

#317

Post by drnick1 »

autoexec wrote: post #231
aeddZX_0 wrote: post #116

Earlier quoted context omitted.

How do you monetise bot traffic?

You take your site off the public internet and paywall it off. Websites on the open internet should be accessible and ideally the goal should be to share something cool with the world, not just to make yourself rich.

This. The Internet was a better place when people hosted things they wanted to share at home or on their ISP's servers, without the expectation of making money.

Re: A year of fighting scrapers on my 1.5 million-page website

#318

Post by leros »

mostlysimilar wrote: post #291
leros wrote: post #107

Earlier quoted context omitted.

Yeah this is one of those things that hurts some real users and AI scrapers have figured out how to bypass. Not worth it IMO.

How do you figure it hurts real users? The amount of compute/energy used on the proof of work is pretty minimal. You're using more when you watch a YouTube video or browse a JS-heavy web app. Of course a sophisticated scraper can "figure out" how to bypass. It isn't trying to be foolproof, it's adding an extra cost to deter massive amounts of bot traffic. I put it in front of my hobby project because I can't afford t…

There are false positives. Every time I've used a tool like this, I get reports from my users about it blocking their access. It's especially bad for users who use privacy oriented browsers like Brave or use VPNs.

For a personal site where you don't care about 0.1% of your users getting blocked, it's probably fine. For a business, blocking some of your paying customers isn't acceptable.

Re: A year of fighting scrapers on my 1.5 million-page website

#319

Post by hk__2 »

mmh0000 wrote: post #254
ballooney wrote: post #169

Earlier quoted context omitted.

This is such a grim thing to read.

I disagree. I personally consider these my biggest problems with the Web: - Bias, specifically commercial bias - Webpage formatting: every website looks different, hides the information I want in different places. Sure, it "looks pretty", but I don't want pretty; I want info - Scams/SEO/etc... The LLMs are very good at reading from multiple sources, parsing, and presenting only the data in a consistent format. There…

I personally prefer to read a couple of recipes to understand the context, what people have tried, what they thought of it. Only when I have a good general idea of the recipe I choose the best one I found based on difficulty and the ingredients I have. I then do the plate, often not following entirely the recipe I chose, and after that and eating it I write down the name of the plate, the recipe link, the date, and a comment about what I did and how it went. Cooking is a lot more than executing a list of steps like a robot.

Re: A year of fighting scrapers on my 1.5 million-page website

#320

Post by mostlysimilar »

leros wrote: post #318
mostlysimilar wrote: post #291

Earlier quoted context omitted.

How do you figure it hurts real users? The amount of compute/energy used on the proof of work is pretty minimal. You're using more when you watch a YouTube video or browse a JS-heavy web app. Of course a sophisticated scraper can "figure out" how to bypass. It isn't trying to be foolproof, it's adding an extra cost to deter massive amounts of bot traffic. I put it in front of my hobby project because I can't afford t…

There are false positives. Every time I've used a tool like this, I get reports from my users about it blocking their access. It's especially bad for users who use privacy oriented browsers like Brave or use VPNs. For a personal site where you don't care about 0.1% of your users getting blocked, it's probably fine. For a business, blocking some of your paying customers isn't acceptable.

> For a business, blocking some of your paying customers isn't acceptable.

That depends on your business and the costs you're incurring from the bots. It might be justifiable to lose 0.1% of your customers if it means saving server costs from serving useless bot traffic.

Post reply on HN