Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

161–170 of 450 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#161

Post by peter_stokes »

RattlesnakeJake wrote: post #32
johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.

you really revved the weebs with this one

Re: A year of fighting scrapers on my 1.5 million-page website

#162

Post by everybodyknows »

qbane wrote: post #103
ihuman wrote: post #68

Earlier quoted context omitted.

There's a difference between someone running a scraping tool occasionally and bots constantly and rapidly re-scraping the same site over and over again

This is an important context: the site is more likely to be targeted by scrapers because it is a curated collection of scraped information.

A curated collection of people who give away money. Was ever sweeter honey ever found in a pot?

Re: A year of fighting scrapers on my 1.5 million-page website

#163

Post by coldpie »

storus wrote: post #9

Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.

I host a couple WordPress blogs on Lithium Hosting. It's like $5/mo, fixed cost, and I've never had any issues. If I somehow hit a transfer usage cap, they would just shut my site down until the next month, but I've never had that happen.

Re: A year of fighting scrapers on my 1.5 million-page website

#164

Post by throw93003838 »

bob1029 wrote: post #121
jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

> the real problem is that the website is poorly built This is almost always the problem. This is currently the problem that GitHub is having too. If they had remained with their crusty old rails architecture, very little of this mess [0] would be occurring right now. They would have been able to focus all their engineering talent on scaling the product rather than inventing elaborate client side state synchronizatio…

We tried to deploy private cloud github enterprise node back in 2018. It was pure garbage without CI integration. I am happy microsoft took harder long term decision!

Re: A year of fighting scrapers on my 1.5 million-page website

#165

Post by nullsanity »

jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

[dead]

Re: A year of fighting scrapers on my 1.5 million-page website

#166

Post by paul7986 »

eigencoder wrote: post #111
jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. I think you've hit the nail on the head here. I think that's a big reason why people want to ban bots.

Exactly and Cloudflare (there should be others out there too - open source options) has proposed that AI bots need to pay for access to our websites. If they and or others could pull off blocking AI access until it pays creators then AI is forced to pay as it should and always should've!

Overall systems and different business models need to be created that forces AI to pay it's fair share! Elon says there will be an abundance thanks to AI and we wont have to work -- ok how's that gonna work without different systems in place paying us?

I feel strongly about this topic and proposed some systems in this Substack post https://ryanspahn.substack.com/p/ai-to-pay-for-all-americans...

Re: A year of fighting scrapers on my 1.5 million-page website

#167

Post by [deleted] »

ako wrote: post #138
hk__2 wrote: post #117

Earlier quoted context omitted.

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

The best way to read the information on the internet today is via a LLM. Just like you don't process raw data in a data warehouse or datalake by looking at tables, you use a SQL or BI tool, to process all the information on the internet you need to tool to digest it for you. For many, today that tool is a LLM chat interface or agent.

[deleted]

Re: A year of fighting scrapers on my 1.5 million-page website

#168

Post by rokkamokka »

drum55 wrote: post #31
johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

Which is trivially bypassed by an actual implementation of the proof of work in non-javascript, rendering it absolutely useless. The website is approximately 3800x times slower than native code, and hundreds of thousands of times slower than the CUDA kernel claude wrote. The "proof of work" is just non existent at that point, they're solved in milliseconds for what would take the browser version 10 minutes or more, i…

Like any lock, it's mainly to deter less determined adversaries (which account for the vast majority)

Re: A year of fighting scrapers on my 1.5 million-page website

#169

Post by ballooney »

ako wrote: post #138
hk__2 wrote: post #117

Earlier quoted context omitted.

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

The best way to read the information on the internet today is via a LLM. Just like you don't process raw data in a data warehouse or datalake by looking at tables, you use a SQL or BI tool, to process all the information on the internet you need to tool to digest it for you. For many, today that tool is a LLM chat interface or agent.

This is such a grim thing to read.

Re: A year of fighting scrapers on my 1.5 million-page website

#170

Post by runjake »

throwaway219450 wrote: post #53
conartist6 wrote: post #29

Earlier quoted context omitted.

That's the thing: even if you don't use AI, if most of what you read is AI, soon this is what you'll sound like, just because it's so much of the training data your own brain's model has. Using it slowly sucks the uniqueness out of you.

Using AI to write blogs doesn’t bother me in principle, but I hate the prose that gets left in. Even if it’s not AI writing, being human doesn’t give you a pass for writing like a self-help guru. This sort of grammar is straight out of Claude: > A human on a VPN sees one CAPTCHA and passes, but a headless browser fleet sees a wall. I do wish we’d stop complaining about em dashes though, that’s lazy criticism. The sen…

Using a global agent rule with something like the following makes agent output much more tolerable for me:

"Use only ASD-STE100 Simplified Technical English when communicating with me or writing documents, comments, and other communications intended for humans."

As a result, I've never seen Claude or Codex use it's weird terms like "load-bearing".

Post reply on HN