Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

431–440 of 450 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#431

Post by timbit42 »

basilikum wrote: post #323
mostlysimilar wrote: post #291

Earlier quoted context omitted.

How do you figure it hurts real users? The amount of compute/energy used on the proof of work is pretty minimal. You're using more when you watch a YouTube video or browse a JS-heavy web app. Of course a sophisticated scraper can "figure out" how to bypass. It isn't trying to be foolproof, it's adding an extra cost to deter massive amounts of bot traffic. I put it in front of my hobby project because I can't afford t…

1. It makes the web more annoying and time wasting. 2. I'm a real user and I have to enable JavaScript and run arbitrary programs from untrusted websites to access them. One browser that I use doesn't even support JavaScript (w3m). Fortunately it is unaffected because it doesn't have the string "Mozilla" in its UA and Anubis lets every user without that pass to not hurt legitimate bots like search engine crawlers. As…

> 1.

It's less annoying than having to click a checkbox.

Re: A year of fighting scrapers on my 1.5 million-page website

#434

Post by Bender »

Bender wrote: post #388
Bender wrote: post #34

Seems about right. I rotated my logs this morning. Most humans go to access.log and most bots go to botpoop.log. This is the line count: 2 access.log [1] 40 botpoop.log [2] 2 is really 1 since a human will grab the CSS file. Most bots do not bother with the style-sheet so that's a 40:1 bots to humans. I could cut that down by blocking data-centers but then I inadvertently block a lot of VPN's which I really don't nee…

More realistic numbers now that HN and others have been visiting all day. Line count: 104 access.log 366 botpoop.log Unique IP's: 54 access.log 181 botpoop.log

One of the botters Lily is showing off their mobile/residential bots to me. It is refreshing to see at least one person can follow instructions.

Re: A year of fighting scrapers on my 1.5 million-page website

#435

Post by matsemann »

vector_spaces wrote: post #372
matsemann wrote: post #357

Earlier quoted context omitted.

No, the first case I used their website in a way that gave them a sale, which is why they have it. Why is it a straw man case when it's reality? Don't argue in bad faith, please. Express your reasoning, like I did, don't just dismiss me because you disagree..

You are assuming that because you are giving the operator money that you are entitled to use their website however you please -- that's not how it works. If you enter a brick & mortar establishment and break their rules, even as a paying customer, you might risk e.g. being kicked out and banned, depending on the behavior. This is not unique to e-commerce To be fair, I do not generally support wholesale banning of scr…

That's an incredibly dishonest characterization of what I wrote. Please re-read the guidelines of the site, they say exactly not to do that.

> saying that you should be entitled to behave as you please because you are a paying customer

No, I'm saying my usage of the site was exactly as if I used my browser agent instead of the llm agent. It visited the category for the product I was interested it, then it visited the pages for the various items to read their specs. Exactly what I would do in my browser as well. In the end I bought one of the products. Point is that I used the site as desired. No extra calls / mass scraping, no stealing of content without them getting anything in return, no breaking of rules.

Re: A year of fighting scrapers on my 1.5 million-page website

#436

Post by SahAssar »

GodelNumbering wrote: post #187

I just checked Cloudflare for SignalBloom ( https://www.signalbloom.ai , which I own and operate). Over the last 72 hours, Claude-searchbot [1] alone fetched ~205,000 pages. Sent exactly 1 referral. There is a lot of free financial data on the site, hoping for real users to benefit from it. It is hard to not feel a little cheated out that Claude gets to claim "Found it!" to its users without me getting no credits or…

Isn't your whole product scraping other sites and summarizing? I also don't see clear sources for your info where it is displayed, so it seems like you do the exact same thing.

Re: A year of fighting scrapers on my 1.5 million-page website

#437

Post by 1718627440 »

hk__2 wrote: post #117
jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

Very few user write a HTTP request themself. By that measure everyone that uses an UA, is a bot user.

Re: A year of fighting scrapers on my 1.5 million-page website

#439

Post by threecheese »

matsemann wrote: post #435
vector_spaces wrote: post #372

Earlier quoted context omitted.

You are assuming that because you are giving the operator money that you are entitled to use their website however you please -- that's not how it works. If you enter a brick & mortar establishment and break their rules, even as a paying customer, you might risk e.g. being kicked out and banned, depending on the behavior. This is not unique to e-commerce To be fair, I do not generally support wholesale banning of scr…

That's an incredibly dishonest characterization of what I wrote. Please re-read the guidelines of the site, they say exactly not to do that. > saying that you should be entitled to behave as you please because you are a paying customer No, I'm saying my usage of the site was exactly as if I used my browser agent instead of the llm agent. It visited the category for the product I was interested it, then it visited the…

If we want to be pedantic, we could say that your automation is like a particularly aggressive DoorDash driver; runs into the store, grabs all items as fast as possible, and is perhaps rude to the checkout girl on the way out. And if you scale this to checking N vendors - and only buying from one - you fall right into the bad behavior.

I’m not sure if your bot is like this, but my understanding is that many of them are - and they interfere with the other shoppers and generally change the economics of it.

I think you are on the right side of this, but you are almost indistinguishable from the negative case.

Post reply on HN