[flagged]
A year of fighting scrapers on my 1.5 million-page website
371–380 of 450 posts
Re: A year of fighting scrapers on my 1.5 million-page website
#372Earlier quoted context omitted.
No, there isn't, it's a bot. Justifying it with a strawman case study doesn't change it. This is giving "why is this plane full? What in doing is important but what is everyone else doing here?"
No, the first case I used their website in a way that gave them a sale, which is why they have it. Why is it a straw man case when it's reality? Don't argue in bad faith, please. Express your reasoning, like I did, don't just dismiss me because you disagree..
You are assuming that because you are giving the operator money that you are entitled to use their website however you please -- that's not how it works. If you enter a brick & mortar establishment and break their rules, even as a paying customer, you might risk e.g. being kicked out and banned, depending on the behavior. This is not unique to e-commerce
To be fair, I do not generally support wholesale banning of scrapers and non-browser clients, since this tends to have lots of ugly side effects.
But regardless, I'm interpreting your last comment as saying that you should be entitled to behave as you please because you are a paying customer. If that's the case, then you are in for a rude awakening one day
Re: A year of fighting scrapers on my 1.5 million-page website
#373[dead]
[deleted]
Re: A year of fighting scrapers on my 1.5 million-page website
#374Earlier quoted context omitted.
No, the first case I used their website in a way that gave them a sale, which is why they have it. Why is it a straw man case when it's reality? Don't argue in bad faith, please. Express your reasoning, like I did, don't just dismiss me because you disagree..
You are assuming that because you are giving the operator money that you are entitled to use their website however you please -- that's not how it works. If you enter a brick & mortar establishment and break their rules, even as a paying customer, you might risk e.g. being kicked out and banned, depending on the behavior. This is not unique to e-commerce To be fair, I do not generally support wholesale banning of scr…
I think the implicit assumption is that if you explained to the owner of the website what the scraper was doing, they would be more than happy to allow it.
The bot was also being generally courteous, only scraping the 2-3 pages needed to get the information for the human who spun up the LLM.
I don't think GP was at all suggesting he should be able to act however he wants, just because he's a paying customer.
I run an ecommerce site and I'm actually more worried about bots being rejected than accepted because of stuff like GP. We get a decent amount of business through ChatGPT.
Re: A year of fighting scrapers on my 1.5 million-page website
#375Earlier quoted context omitted.
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.
Are you gonna police the means through which I'm browsing the internet? If you want to, you are free to put your stuff behind the paywall and give the key only to people who agree to obey your conditions. If you put it in the open you can't make conditions. That's what publishing means. Author can't make demands in what manner their book should be read.
The entire blog was about how the site owner doesn't want to put it in the open.
Your pontificating can't change the fact that he can do whatever he wants with his site.
Re: A year of fighting scrapers on my 1.5 million-page website
#376> My normal bill for running this whole site is around $90 a month. During one bad spike month, it jumped about 500%. This is D1 - which has very surprising costs. you may just want to drop D1 and move to a static site. There’s no reason your site should cost this much.
Thank you! I need to tighten up my KV compression, which is actually carrying a lot of D1's load otherwise. We also had some bad queries some months, as the pages and database grew, that were counting the wrong things (or extremely inefficiently) and those have since been fixed.
How often does your data change?
If infrequent you could cache the generated page and not query D1 for subsequent requests at all.
Re: A year of fighting scrapers on my 1.5 million-page website
#377Earlier quoted context omitted.
> For a business, blocking some of your paying customers isn't acceptable. That depends on your business and the costs you're incurring from the bots. It might be justifiable to lose 0.1% of your customers if it means saving server costs from serving useless bot traffic.
I suppose it would depend, yeah. I find having paying customers being blocked to be completely unacceptable. Not only is a bad customer experience, but they tend to be very vocal about it. Posting online, leaving bad reviews, etc.
Having to serve 1000x more traffic from bots is completely unacceptable for most site owners as well.
The fact that this trade-off isn't obvious indicates you haven't tried to run a site at scale and make a living off of it.
Re: A year of fighting scrapers on my 1.5 million-page website
#378The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…
> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.
And this is a complaint? I find nothing in this statement to sympathize with whatsoever. Isn't the contract of, essentially everything, that effort is required to obtain something worthwhile?
Please let me know where I'm getting this long-standing, fairly fundamental understanding of the world, wrong.
Re: A year of fighting scrapers on my 1.5 million-page website
#379Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.
I host a static site on S3 with cloudfront on top and my bill is usually a few dollars. It was $2.88 last month. The site could also probably handle many orders of magnitude more traffic than it currently does because cloudfront scales to that level without work needed on my side.
Re: A year of fighting scrapers on my 1.5 million-page website
#380The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…
This is a whole lot of things you want and feel entitled to. Nobody has to cater to you - we can block whatever we want to block. And if our poorly designed sites bother you, that’s too bad for you. But your wants are not my problem. If you want someone to cater to you, pay them. You aren’t entitled to anything.
The attitude you're describing seems startlingly common, as if it's not the actual site owner that has made the active choice to block something that's causing them trouble.
They all sound as if they've been logic-twisted by some product idea they think is going to make them rich, and these blocks on bots are costing them access to the raw materials for their magnificently worthwhile project.