Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

21–30 of 449 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#21
post #3

Earlier quoted context omitted.

what gave you this idea?

Take a look at the dashes. It's all there. (i skimmed the whole post, there are none)

I knew it... Wuthering Heights was written by AI and Lucy Maud Montgomery, Edgar Allan Poe, et. al. were AI bots churning out content!

Or maybe -- just maybe -- using dashes isn't a sign of content being AI written, just a style it picked up from the training data.

Re: A year of fighting scrapers on my 1.5 million-page website

#22
Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software.

[1] https://anubis.techaro.lol/

Re: A year of fighting scrapers on my 1.5 million-page website

#23
post #6

blocking by geo yeah might work - but what happens when someone is traveling abroad ? they've to use a VPN to access your site ? my take with all the bots - the web is gonna be a bunch of private walled gardens. with most sites set to no index. you will only discover them via referral from someone real.

What happens when someone is from a shit country?

AFAIK you already have to use VPN if you are e.g. westerner visiting China or Belarus.

Re: A year of fighting scrapers on my 1.5 million-page website

#24
post #3
post #2

[flagged]

what gave you this idea?

Not the OP, but because it appears structured just like AI output?

(This isn't a condemnation. AI can often do a better job of representing thoughts than humans.)

Examples:

- The article organization

- The general language flow

- The bullet points with a bolded gist, a colon, and then elaboration (and bonus with details stats).

- The images are almost certainly AI generated. They look AI generated.

Re: A year of fighting scrapers on my 1.5 million-page website

#25
post #13
post #2

[flagged]

HN's blanket AI allegations are almost as annoying and tired as the thing they are rallying against. I think you are AI.

>HN's blanket AI allegations are almost as annoying and tired as the thing they are rallying against.

Best thing I've read on HN so far this year.

Re: A year of fighting scrapers on my 1.5 million-page website

#26
post #5
post #3

Earlier quoted context omitted.

what gave you this idea?

Because the ubermensch on HN can detect AI text from 10 miles away with 100% accuracy... like that time they called documents from 2015 AI written.

I'm just beyond thrilled that it's actually starting to be a public embarrassment to cough up a glob of AI text.

Re: A year of fighting scrapers on my 1.5 million-page website

#28
post #9

Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.

I run my personal website (and a bunch of other websites and services) off a somewhat more expensive but still reasonable VPS (I think 20€ at TransIP - it's so little that I forgot). Load is basically nil most of the time anyway. I think the bot problem is not so bad for personal websites.

Re: A year of fighting scrapers on my 1.5 million-page website

#29
post #24
post #3

Earlier quoted context omitted.

what gave you this idea?

Not the OP, but because it appears structured just like AI output? (This isn't a condemnation. AI can often do a better job of representing thoughts than humans.) Examples: - The article organization - The general language flow - The bullet points with a bolded gist, a colon, and then elaboration (and bonus with details stats). - The images are almost certainly AI generated. They look AI generated.

That's the thing: even if you don't use AI, if most of what you read is AI, soon this is what you'll sound like, just because it's so much of the training data your own brain's model has.

Using it slowly sucks the uniqueness out of you.

Post reply on HN