A year of fighting scrapers on my 1.5 million-page website • Hacker News Board

Re: A year of fighting scrapers on my 1.5 million-page website

#391

Post by andai » Sat, Aug 08, 2026, 1:44 AM UTC

eigencoder wrote: post #111

jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. I think you've hit the nail on the head here. I think that's a big reason why people want to ban bots.

...because they want to make it harder for people to get the information they need?

I think I must be missing something here.

Re: A year of fighting scrapers on my 1.5 million-page website

#392

Post by joshspankit » Sat, Aug 08, 2026, 1:49 AM UTC

How long until Cloudflare is the data broker for websites like this?

“For a low $/GB, we’ll give you everything from this site and 1000 others as (structured data/a database)!”

(yes there are lots of good counter arguments to this, but before you reply think ahead a couple extra steps)

Re: A year of fighting scrapers on my 1.5 million-page website

#393

Post by roboror » Sat, Aug 08, 2026, 1:57 AM UTC

ako wrote: post #138

hk__2 wrote: post #117

Earlier quoted context omitted.

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

The best way to read the information on the internet today is via a LLM. Just like you don't process raw data in a data warehouse or datalake by looking at tables, you use a SQL or BI tool, to process all the information on the internet you need to tool to digest it for you. For many, today that tool is a LLM chat interface or agent.

This feels like "the best way to read a book is via CliffsNotes"

Re: A year of fighting scrapers on my 1.5 million-page website

#394

Post by [deleted] » Sat, Aug 08, 2026, 2:03 AM UTC

hk__2 wrote: post #117

jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

[deleted]

Re: A year of fighting scrapers on my 1.5 million-page website

#395

Post by TheRealPomax » Sat, Aug 08, 2026, 2:06 AM UTC

jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

No one was hurt, you were at most inconvenienced. And that's perfectly fine.

Also, using jargon that doesn't apply: a knee-jerk reaction is one that does way too much to address a small problem. Objectively, this is the literal opposite: targeted reactions to different aspects of a huge problem.

Re: A year of fighting scrapers on my 1.5 million-page website

#396

Post by BLKNSLVR » Sat, Aug 08, 2026, 2:10 AM UTC

varenc wrote: post #118

Can someone help me understand the underlying motivation behind this? It makes sense that some crawlers, in the style of Google, would want to index the entire internet. But what is the point of the same crawler re-fetching a page they already fetched an hour ago? Or possibly all this traffic is just independent entities, each trying to cache the internet? The scale of bot traffic makes this seem unlikely. What's the…

My laymans understanding is that poorly written scripts get executed and then owner comes back occasionally to check that there is 'content' in their database. They're not sitting there troubleshooting their thing beyond 'it's getting data' because, for similar reasons to their scripts being poorly written, their strategy is 'get data'.

Nuance, complexity, and an awareness of 'other people' do not exist in their worlds.

Re: A year of fighting scrapers on my 1.5 million-page website

#397

Post by shakna » Sat, Aug 08, 2026, 2:16 AM UTC

Terretta wrote: post #150

hk__2 wrote: post #117

Earlier quoted context omitted.

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

you know what a browser is called by the web site? check the header that tells the version. USER AGENT not user, an agent on behalf of the user. the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT. for a glimpse at one possible…

Most pre-LLM systems also blocked badly behaved user agents... And right now, most LLM-suppliers, are badly behaved agents.

Re: A year of fighting scrapers on my 1.5 million-page website

#398

Post by peter_d_sherman » Sat, Aug 08, 2026, 2:18 AM UTC

An interesting article, and an interesting problem to have!

The problem, to recap the title, is that "99% of My Website Traffic Is Bots".

How to fight all of those bots, all of those web-scrapers...

If the sole underlying issue is bandwidth for one's website (someone doesn't have enough of it and/or they pay too much for it), then one possible solution to this might be for a single tech company to create an online cache of web pages, accessible to all bots as an alternate route to fetch the desired web pages.

If a tech company (or consortium thereof) decided to do this, they could certainly charge money to each AI / bot utilizing the service...

Which means that it might be an investable idea as a for-profit service...

Now, if the underlying issue is the privacy of individual web pages (or relative privacy, as the case may be!) from the public internet, then perhaps the solution is to simply put those behind user logins and/or a gauntlet of tests (which could also be tied to login!) designed to exclude AI/bot web traffic from human web traffic.

Consider what some BBS'es and the online services of yesteryear used to do (Compuserve, The Source, Prodigy, QuantumLink, AOL, etc.)... basically they'd get a user's mailing address, and then physically snail mail them their login and password to their physical home address...

Oh sure, user registration wasn't as fast back then as it is today... it might take several days to receive your username and password via postal mail -- but as a system operator you were 99.999999999% guaranteed that when you saw such a login on your BBS, that it was an actual live human user...

There may be a market for a service like that, too...

That is, validate that actual users are actual users, and give them some kind of credential that can be checked by an actual web site... find a way to do this at scale...

Anyway, just thinking aloud... a very interesting article, and a very interesting problem to have!

Re: A year of fighting scrapers on my 1.5 million-page website

#399

Post by luciana1u » Sat, Aug 08, 2026, 2:28 AM UTC

dabbz wrote: post #309

luciana1u wrote: post #245

at this point the web is mostly machines politely asking other machines for permission to read each other's content, and humans are the ones triggering the captchas.

Machines don't pay my bills though. If the content was free to share widely, then yea, who cares how it's accessed.

the captchas aren't for the machines though, they're for us — the machines are the ones deciding we're not robots.

Re: A year of fighting scrapers on my 1.5 million-page website

#400

Post by andrekandre » Sat, Aug 08, 2026, 2:49 AM UTC

Terretta wrote: post #295

zamadatix wrote: post #221

Earlier quoted context omitted.

I fancy the approach myself... but, much like the grand ideas of OSI in networking, Tim Burners Lee's idea of the Web was never what was going to be used and a lot of the original terms/concepts just plain don't make sense for the Web that exists. It was, originally, a true document system, nothing more. It wasn't until many changes from the original idea that it started to be what people wanted & actually use today.…

I was on the internet before HTML and long before Mosaic... I agree with one aspect of the narrative you're saying, his idea was held as being for «documents». To be sure, document was more abstract than document on a Mac. Either way, I'm very sad it mostly turned into animated brochures instead. But the reason styles were separate, the reason everyone talked about semantics, was always about the universal knowledge…

  > it mostly turned into animated brochures instead

just my crazy opinion but it was inevitable unless there was some law to prevent commercialization (in the ad sense) of the internet (keep it a public good for individuals only)