Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

471–479 of 479 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#471

Post by zImPatrick »

aitchnyu wrote: post #466
zImPatrick wrote: post #432

Regarding the residental botnets: check their TLS fingerprint, they usually (at least from my experience) have the same few JA4 hashes that don‘t match with modern browsers

Are they all LG TVs or similar?

Not sure, I couldn‘t find those fingerprints anywhere and I couldn‘t replicate them in any browser I had access to (Safari iOS, Windows & Linux Firefox & Chromium), so I decided to block them. I haven‘t had a single person complain yet. Their UA are usually the most random ones you can find (who uses a PPC mac nowadays?)

Re: A year of fighting scrapers on my 1.5 million-page website

#472

Post by hk__2 »

illiac786 wrote: post #418
hk__2 wrote: post #411

Earlier quoted context omitted.

I’m not forgetting anything here. Bots and humans behind browsers are different things.

How is the difference relevant here?

I care about people reading my content with their browser. I don’t care about bots.

Re: A year of fighting scrapers on my 1.5 million-page website

#473

Post by illiac786 »

hk__2 wrote: post #472
illiac786 wrote: post #418

Earlier quoted context omitted.

How is the difference relevant here?

I care about people reading my content with their browser. I don’t care about bots.

That wasn’t the question. Do you care about humans using a bot to grab your webpage and extract information about it?

If not, that’s fine, you just don’t care about a specific category of human users.

Re: A year of fighting scrapers on my 1.5 million-page website

#474

Post by TaLiTr »

account42 wrote: post #468
brazukadev wrote: post #239

Earlier quoted context omitted.

that's really smart of them. Just pay to use the service you need.

It's really smart of them to manufacture a negative sentiment about the project because the sexual fetish icon gets injected into all kinds of places where it is absolutely inappropriate?

Ah yes, all of anime is a "fetish thing."

People like you are the reason every single communication, logo, and document is overly formal corpo-slop that reads like a LinkedIn post.

Re: A year of fighting scrapers on my 1.5 million-page website

#475

Post by throwawaypath »

nektro wrote: post #453
basilikum wrote: post #95

Earlier quoted context omitted.

It doesn't detect "real browsers". It simply adds some friction and is niche enough that AI scrapers have not bothered to bypass it yet.

> and is niche enough that AI scrapers have not bothered to bypass it yet. it is not niche. it is state of the art and widely deployed on major websites.

>it is state of the art and widely deployed on major websites.

No it's not.

Re: A year of fighting scrapers on my 1.5 million-page website

#476

Post by krapp »

account42 wrote: post #467
krapp wrote: post #356

Earlier quoted context omitted.

There is absolutely nothing sexualized about the Anubis mascot.

You are very naive if you believe there is nothing sexualized about furries.

I'm not talking about furries I'm talking about the mascot of the Anubis software program, which is not in any way sexualized.

Do you believe any anthropomorphic cartoon animal must be implicitly sexual because "furries exist and have sex?" I hate to break it to you but just about everyone has sex.

Also, Looney Tunes must horrify you. Wile E. Coyote isn't even wearing any clothes at all, and they show that to children!

Re: A year of fighting scrapers on my 1.5 million-page website

#477

Post by infinitezest »

lostmsu wrote: post #443
infinitezest wrote: post #228

Earlier quoted context omitted.

Because the economic, social, and climate impacts are at best uncertain and at worst devastating to the majority of human beings on the planet. It's always surprising to me that people don't intuitively separate the usefulness of AI from its risks. It feels like everyone has to be a doomer or a booster.

> at worst devastating This is such a low bar literally anything would clear it. Even tomato farming.

I think you're going to have to flesh this out a little more, friend. I'm not sure what you're trying to say.

Re: A year of fighting scrapers on my 1.5 million-page website

#478

Post by knuckleheads »

ccgreg wrote: post #415
knuckleheads wrote: post #93

Previously, I had done a fair amount of research into how Google's monopoly on web crawling further entrenches their monopoly in the search engine market. You can read more about this here, https://knuckleheads.club , there is a long report from ~2020 or so that explains how it worked at the time. The club is mothballed, I am doing other things with my life, and I'm happy to say that we played a very small role in th…

> It's a shame that CCBot is caught in the cross fire, but that's life. We're used to it. Sadly.

RIP to a real one. Have you considered using one billion Chinese residential ips to address your problem? I’ve heard one billion Chinese residential ips really does the trick here.

Re: A year of fighting scrapers on my 1.5 million-page website

#479

Post by allthetime »

marginalia_nu wrote: post #128
allthetime wrote: post #108

Earlier quoted context omitted.

For a “personal website” or any relatively simple static content site, the DB should not be getting hit much at all. Cache is your friend. For my sites I have a map of all relevant queries and when we do updates we run the relevant queries one time and cache them to both memory and file system.

Should being the operative word. In practice it's both easy and common to have a database do unnecessary work when generating a page, a problem that often only rears its head when under unusual load.

It depends. If you don't have user accounts / user generated data I don't see why you can't completely determine all necessary content GETs. You could even go as far as to not even allow web requests to trigger database calls and have them only ever pull from cache.
Post reply on HN