Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

471–477 of 477 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#471

Post by zImPatrick »

aitchnyu wrote: post #466
zImPatrick wrote: post #432

Regarding the residental botnets: check their TLS fingerprint, they usually (at least from my experience) have the same few JA4 hashes that don‘t match with modern browsers

Are they all LG TVs or similar?

Not sure, I couldn‘t find those fingerprints anywhere and I couldn‘t replicate them in any browser I had access to (Safari iOS, Windows & Linux Firefox & Chromium), so I decided to block them. I haven‘t had a single person complain yet. Their UA are usually the most random ones you can find (who uses a PPC mac nowadays?)

Re: A year of fighting scrapers on my 1.5 million-page website

#472

Post by hk__2 »

illiac786 wrote: post #418
hk__2 wrote: post #411

Earlier quoted context omitted.

I’m not forgetting anything here. Bots and humans behind browsers are different things.

How is the difference relevant here?

I care about people reading my content with their browser. I don’t care about bots.

Re: A year of fighting scrapers on my 1.5 million-page website

#473

Post by illiac786 »

hk__2 wrote: post #472
illiac786 wrote: post #418

Earlier quoted context omitted.

How is the difference relevant here?

I care about people reading my content with their browser. I don’t care about bots.

That wasn’t the question. Do you care about humans using a bot to grab your webpage and extract information about it?

If not, that’s fine, you just don’t care about a specific category of human users.

Re: A year of fighting scrapers on my 1.5 million-page website

#474

Post by TaLiTr »

account42 wrote: post #468
brazukadev wrote: post #239

Earlier quoted context omitted.

that's really smart of them. Just pay to use the service you need.

It's really smart of them to manufacture a negative sentiment about the project because the sexual fetish icon gets injected into all kinds of places where it is absolutely inappropriate?

Ah yes, all of anime is a "fetish thing."

People like you are the reason every single communication, logo, and document is overly formal corpo-slop that reads like a LinkedIn post.

Re: A year of fighting scrapers on my 1.5 million-page website

#475

Post by throwawaypath »

nektro wrote: post #453
basilikum wrote: post #95

Earlier quoted context omitted.

It doesn't detect "real browsers". It simply adds some friction and is niche enough that AI scrapers have not bothered to bypass it yet.

> and is niche enough that AI scrapers have not bothered to bypass it yet. it is not niche. it is state of the art and widely deployed on major websites.

>it is state of the art and widely deployed on major websites.

No it's not.

Re: A year of fighting scrapers on my 1.5 million-page website

#476

Post by krapp »

account42 wrote: post #467
krapp wrote: post #356

Earlier quoted context omitted.

There is absolutely nothing sexualized about the Anubis mascot.

You are very naive if you believe there is nothing sexualized about furries.

I'm not talking about furries I'm talking about the mascot of the Anubis software program, which is not in any way sexualized.

Do you believe any anthropomorphic cartoon animal must be implicitly sexual because "furries exist and have sex?" I hate to break it to you but just about everyone has sex.

Also, Looney Tunes must horrify you. Wile E. Coyote isn't even wearing any clothes at all, and they show that to children!

Re: A year of fighting scrapers on my 1.5 million-page website

#477

Post by infinitezest »

lostmsu wrote: post #443
infinitezest wrote: post #228

Earlier quoted context omitted.

Because the economic, social, and climate impacts are at best uncertain and at worst devastating to the majority of human beings on the planet. It's always surprising to me that people don't intuitively separate the usefulness of AI from its risks. It feels like everyone has to be a doomer or a booster.

> at worst devastating This is such a low bar literally anything would clear it. Even tomato farming.

I think you're going to have to flesh this out a little more, friend. I'm not sure what you're trying to say.
Post reply on HN