A year of fighting scrapers on my 1.5 million-page website • Hacker News Board

Re: A year of fighting scrapers on my 1.5 million-page website

#361

Post by carlosjobim » Fri, Aug 07, 2026, 11:10 PM UTC

baby_souffle wrote: post #261

hluska wrote: post #206

Earlier quoted context omitted.

I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. They clearly don’t mind so what’s the issue? You still used a bot.

> I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. The point is that not all bots are bad. Assuming as much / implementing policies to that effect won't block all bots but it _will_ block the portion of bots that represent users considering giving you money. I have a series of bots that monitor various eCom sites to monitor prices over time for big-ticket items I am consid…

How should they know that your bot belongs to a potential customer and not to a competitor surveilling their prices?

Not a rhetorical question, because most companies would welcome that interest from customers.

Re: A year of fighting scrapers on my 1.5 million-page website

#362

Post by pritambaral » Fri, Aug 07, 2026, 11:14 PM UTC

RattlesnakeJake wrote: post #341

peter_stokes wrote: post #161

Earlier quoted context omitted.

you really revved the weebs with this one

It seems to come up every time this complaint arises. "I want to use an open source project without being associated with furries" isn't that unreasonable of a request.

> open source project

So, one can put in the minimal effort of removing one div.

> being associated

There's no requirement of association of any kind. See above and below.

> request

Cool. Pay for your own branding. Request fulfilled. No? Then it's a demand, not a request.

Re: A year of fighting scrapers on my 1.5 million-page website

#363

Post by quectophoton » Fri, Aug 07, 2026, 11:18 PM UTC

wasmperson wrote: post #274

jt2190 wrote: post #222

Earlier quoted context omitted.

Yeah it's sort of weird that people are so used to using their computer to interact with other computers across the internet that the whole thing has become invisible to them: They completely forget that they're using a computer at all. Along come "bots" and suddenly everyone remembers that the whole thing involves computers. Driving the computer by hand for hours was never efficient, but is marginally entertaining I…

This reminds me of all the confused comments that pop up whenever someone calls plain HTML with forms and links a "REST API." "But APIs are for programs! People visiting websites aren't programs." https://news.ycombinator.com/item?id=32141027

Tangential but still related to that story, but the battle for the meaning of "REST" is already lost.

Saying an API follows REST but doesn't implement HATEOAS (part of its definition), is like saying a database follows ACID but doesn't implement Atomicity. Or like saying a software is Open Source but doesn't allow use for commercial purposes.

Yet somehow for one of those 3 cases it became accepted to misuse the term that way ("[original meaning] but without this one constraint"), but if someone misuses the term for the other two cases then everyone suddenly cares and insist that people should use a different term to describe their thing.

Re: A year of fighting scrapers on my 1.5 million-page website

#364

Post by throwaway219450 » Fri, Aug 07, 2026, 11:19 PM UTC

duskdozer wrote: post #265

throwaway219450 wrote: post #53

Earlier quoted context omitted.

Using AI to write blogs doesn’t bother me in principle, but I hate the prose that gets left in. Even if it’s not AI writing, being human doesn’t give you a pass for writing like a self-help guru. This sort of grammar is straight out of Claude: > A human on a VPN sees one CAPTCHA and passes, but a headless browser fleet sees a wall. I do wish we’d stop complaining about em dashes though, that’s lazy criticism. The sen…

Well, I definitely never saw nearly as many emdashes a few years ago. I didn't even realize it wasn't just a typeface quirk as I'd used hyphens a lot. I don't even know how to type an emdash on my devices and would have to copy/paste it from somewhere. But yes your other points are absolutely right

Oh yeah, emdashes are absolutely a tell, but I argue that they're nowhere near as abhorrent to read than the awkward phrasing that the frontier models like to use.