Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

121–130 of 451 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#121

Post by bob1029 »

jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

> the real problem is that the website is poorly built

This is almost always the problem.

This is currently the problem that GitHub is having too. If they had remained with their crusty old rails architecture, very little of this mess [0] would be occurring right now. They would have been able to focus all their engineering talent on scaling the product rather than inventing elaborate client side state synchronization mechanisms.

[0]: https://www.githubstatus.com/incidents/qcvjkzcs7j74

Re: A year of fighting scrapers on my 1.5 million-page website

#122

Post by wbl »

johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

Anubis sucks because CPU is cheap for scrapers and hard for humans.

Re: A year of fighting scrapers on my 1.5 million-page website

#123

Post by ryan_n »

RattlesnakeJake wrote: post #32
johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.

You just discovered their business model congrats.

Re: A year of fighting scrapers on my 1.5 million-page website

#124

Post by RobotToaster »

drum55 wrote: post #84
marklar423 wrote: post #48

Earlier quoted context omitted.

I'm assuming a bot running a headless browser instance can still get past it? It's still valuable to raise the cost of scraping of course. I don't think anything can really stop a determined scraper from impersonating a human. I wonder though if a system similar to Anubis but mining some crypto would make bots _welcome_ - since they're paying for their traffic.

People tried this in 2013 or so, there's no point to it. Doing proof of work in javascript in a browser is so crushingly, pointlessly slow that there's no value at all. Some browsers also intentionally detect attempts to do proof of work and attempt to block it entirely.

> Some browsers also intentionally detect attempts to do proof of work and attempt to block it entirely.

Then they'd be blocking themselves from the website.

Re: A year of fighting scrapers on my 1.5 million-page website

#125

Post by Lalabadie »

ihuman wrote: post #68
qbane wrote: post #4

> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.

There's a difference between someone running a scraping tool occasionally and bots constantly and rapidly re-scraping the same site over and over again

The author's website is responsible for storing its own data. AI services currently treat the entire web as their storage and cache layer.

Re: A year of fighting scrapers on my 1.5 million-page website

#126

Post by splatter9859 »

rhdunn wrote: post #21
chrisandchris wrote: post #7

Earlier quoted context omitted.

Take a look at the dashes. It's all there. (i skimmed the whole post, there are none)

I knew it... Wuthering Heights was written by AI and Lucy Maud Montgomery, Edgar Allan Poe, et. al. were AI bots churning out content! Or maybe -- just maybe -- using dashes isn't a sign of content being AI written, just a style it picked up from the training data.

As odd as this sounds, and as odd as the world gets, I still find a small comfort in the fact that there is enough stability in this world that I can count on one thing: someone always accusing a blog post on HN as being AI generated.

Never fails.

The OCD part of me can now go about my day.

Re: A year of fighting scrapers on my 1.5 million-page website

#127

Post by [deleted] »

runjake wrote: post #24
Yiin wrote: post #3

Earlier quoted context omitted.

what gave you this idea?

Not the OP, but because it appears structured just like AI output? (This isn't a condemnation. AI can often do a better job of representing thoughts than humans.) Examples: - The article organization - The general language flow - The bullet points with a bolded gist, a colon, and then elaboration (and bonus with details stats). - The images are almost certainly AI generated. They look AI generated.

[deleted]

Re: A year of fighting scrapers on my 1.5 million-page website

#128

Post by marginalia_nu »

allthetime wrote: post #108
marginalia_nu wrote: post #71

Earlier quoted context omitted.

It's almost always the DBMS that's the bottleneck when websites drop from traffic. As long as you don't do anything fancier than primary key look-ups you're probably fine though. I survived handling the search search traffic generated by this thread[1] on PC hardware off a residential broadband connection without any sort of degradation. Only time I've gone offline from traffic was when Elon Musk tweeted a link to on…

For a “personal website” or any relatively simple static content site, the DB should not be getting hit much at all. Cache is your friend. For my sites I have a map of all relevant queries and when we do updates we run the relevant queries one time and cache them to both memory and file system.

Should being the operative word. In practice it's both easy and common to have a database do unnecessary work when generating a page, a problem that often only rears its head when under unusual load.

Re: A year of fighting scrapers on my 1.5 million-page website

#129

Post by hk__2 »

storus wrote: post #9

Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.

All my personal websites are on a single $5/mo OVH VPS and it works perfectly fine.

Re: A year of fighting scrapers on my 1.5 million-page website

#130

Post by gmerc »

aeddZX_0 wrote: post #116
jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

How do you monetise bot traffic?

https://claw-guard.org/adnet
Post reply on HN