Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

201–210 of 454 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#201

Post by IMSAI8080 »

mgbmtl wrote: post #44

I run scripts on my servers on an hourly basis to check which are the top 25 IPs visiting the server (aggregated by /24). If anyone in those top 25 IPs are from China, Vietnam, etc, or from Alibaba/Amazon/etc, the /24 gets blocked by iptables. It's far from perfect, but it was a quick way to get rid of bots, while not completely blocking people from countries such as Vietnam. However, on a Gitlab instance I manage (5…

My solution was similar. Anything coming from the ASN of a major cloud provider gets a CAPTCHA with a little nuance to allow Google and Bing to index. That seems to do a pretty good job. Also, anything coming out of China or Singapore also gets a CAPTCHA as my site is not popular in those regions and many Chinese bots seem to show up as a Chinese mobile provider. So far, the bots have never attempted to solve the CAPTCHA.

Re: A year of fighting scrapers on my 1.5 million-page website

#203

Post by thorsson12 »

The experience of browsing the web has really suffered lately. The mandatory 3-4 second "verifying that you're a human" block from Cloudflare seem to show up on more and more websites. Seems like a questionable choice from Cloudflare to teach everyone to associate Cloudflare's logo with high latency...

Re: A year of fighting scrapers on my 1.5 million-page website

#204

Post by asgraham »

Terretta wrote: post #150
hk__2 wrote: post #117

Earlier quoted context omitted.

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

you know what a browser is called by the web site? check the header that tells the version. USER AGENT not user, an agent on behalf of the user. the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT. for a glimpse at one possible…

Can you recommend a specific home assistant community to check out?

Re: A year of fighting scrapers on my 1.5 million-page website

#205

Post by Flashtoo »

selckin wrote: post #149
criley2 wrote: post #141

Earlier quoted context omitted.

Scenario one: you use software to connect to their server and download a webpage. You are a user. Scenario two: you use software to connect to their server and download a webpage. You are a "bot". Make it make sense

because they maintain the websites for social status, if the user never sees the specific website or knows they used it, you can't gain any social status

> they maintain the websites for social status

Are you implying that's a bad thing? Doing things for social status has been an integral part of society for millennia. It's a legitimate motivation that can benefit both the recipient of the status and the rest of society. In this specific case, if you assume that the author makes content that is useful to you only because of the social status reward, taking away that reward means the author will stop making content that is useful to you.

Re: A year of fighting scrapers on my 1.5 million-page website

#206

Post by hluska »

matsemann wrote: post #171
hk__2 wrote: post #117

Earlier quoted context omitted.

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

I don't disagree, but there is a sliding scale here. For instance, I wanted to buy a piece of equipment the other day from a local company for a specific usecase. I wanted to find a specific price/weight/specs ratio, and asked an llm to loop through the 20 or so items, fetch their page and calculate and present some values for each. This then led to me going and buying the one I found. So the llm was mainly just an e…

I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. They clearly don’t mind so what’s the issue? You still used a bot.

Re: A year of fighting scrapers on my 1.5 million-page website

#207

Post by pessimizer »

elorant wrote: post #147
hk__2 wrote: post #117

Earlier quoted context omitted.

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

What if I’m a web alerts company? I crawl your content but my clients are all end users who actually see your content at your site.

Then you are a uninvited bot that crawls my content in order to send alarms to people to tell them to visit my site?

This is a "what if it's for your own good?" argument. What if I break into your house to clean your toilets? Although amusingly, if you're a "web alerts company" that I didn't contract and sends out alerts in batches on your own schedule, you will probably send a ton of your customers to my site at the same time and slashdot me off of the web entirely.

Re: A year of fighting scrapers on my 1.5 million-page website

#208

Post by kijin »

harshreality wrote: post #100
drum55 wrote: post #80

Earlier quoted context omitted.

The prompt used for Opus 4.8 was: write a implementation of the anubis proof of work in native c code, optimized for speed above all else. use every trick available to make the proof of work as efficient and fast as possible, including modern processor tricks on the x86 platform. your code should avoid using external libraries where possible, include tests, and be readable and concise. a reference for what needs to b…

In a thread last month about scrapers, the author mentioned working on a switch to hashx.[1] In addition, nothing prevents anubis from sending a wasm solver instead of js, reducing the gap between a custom native solver and a js solver. [1] https://news.ycombinator.com/item?id=48869064

They should make it mine actual coins for the site owner.

The more bots try to access the site, the more profitable it will be!

Re: A year of fighting scrapers on my 1.5 million-page website

#210

Post by inigyou »

creshal wrote: post #109
inigyou wrote: post #66

Earlier quoted context omitted.

There's a lot of people who host their site at extremely expensive places and then do everything they can to minimise unneeded traffic - instead of just moving to a cheaper host. Vercel is another popular extremely expensive host.

I'm always flabbergasted when I see what people pay and how much effort they need to invest to keep their cloud websites from eating them alive. My allegedly more complicated VPS stack needs an afternoon of attention every two years when a new Debian major release is necessary, and costs have been predictable for 15 years, no matter what happened traffic wise.

The predictability of costs is underrated too. You don't want your hosting solution to automatically scale up to $20,000. You want that if it's overloaded it's simply overloaded.
Post reply on HN