Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

281–290 of 450 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#281

Post by coreyp_1 »

bakugo wrote: post #252
autoexec wrote: post #227

Earlier quoted context omitted.

The social contract for putting a website on the public internet involves following standards and making your website accessible. You SHOULD make sure that your website is readable to people no matter what user agent they are using as long as it's standards compliant. For example, making a website that only works in Edge and refuses to load for anything else would be a shitty practice. You are free to do it. There's…

The social contract for putting a website on the public internet was built on the assumption that browsers would be used to access those websites, and that most browsers would generally behave in a similar way (click on page, display page, pretty simple). LLMs are not browsers, nor are they people, so I don't see why they should be part of this equation at all. They do not display the page for a real person to read,…

Don't tell this person about translation plugins. They change everything! The end user never reads what the website actually served!

Re: A year of fighting scrapers on my 1.5 million-page website

#282

Post by fooey »

ddxv wrote: post #10

I'm in a similar boat. Probably 99.999% is bots. I have nearly 1m unique "visitors" according to cloudflare and my real users are in the dozens a day. That being said, I love the open internet and am holding on to keeping as much open as I can.

I shutdown all my little informational hobby projects this year that I've tinkered with for decades

They all shifted from mostly paying for themselves (or being so cheap it didn't matter), to essentially producing zero income while resource usage leapt up in magnitudes

I couldn't justify the stress and hassle of making sites, that a few dozen people a day might find useful, into some complex hyperscalable obligations just to feed the bots

Re: A year of fighting scrapers on my 1.5 million-page website

#283

Post by autoexec »

coreyp_1 wrote: post #281
bakugo wrote: post #252

Earlier quoted context omitted.

The social contract for putting a website on the public internet was built on the assumption that browsers would be used to access those websites, and that most browsers would generally behave in a similar way (click on page, display page, pretty simple). LLMs are not browsers, nor are they people, so I don't see why they should be part of this equation at all. They do not display the page for a real person to read,…

Don't tell this person about translation plugins. They change everything! The end user never reads what the website actually served!

It's probably a pretty common misconception that websites should have total control over how their content is viewed, but it's strange to see that on a website like this. Part of the reason apps are so popular with companies is that it gives them control over how content is presented which is something webpages were never intended to give them.

Re: A year of fighting scrapers on my 1.5 million-page website

#284

Post by fooey »

thomashabets2 wrote: post #256

Something suspiciously absent from this article is addressing whether some of these are in fact human visitors, but humans issuing chatgpt or similar queries instead of going directly. Is it really a bot if it's in response to a human asking for some aggregate information about charities, triggering a web search and then following the result links to get details for the human? Well, clearly yes it is, but it's a very…

cloudflares lack of spending caps and and usage alerts is a seriously unnerving approach

Re: A year of fighting scrapers on my 1.5 million-page website

#285

Post by jader201 »

nickgray wrote: post #87
aeturnum wrote: post #82

Earlier quoted context omitted.

Similar to the dose making the poison - the thing that jumped out at me in this blog was the ratio of scraping to visits. Unless OP is scraping thousands of times a day I don't really think they're in the same class as the bots they are blocking.

OP here: I'm not scraping thousands of times per day! Usually just a few times per year.

But you could just be one of thousands of bots targeting the same sources you’re scraping.

Re: A year of fighting scrapers on my 1.5 million-page website

#287

Post by randomblock1 »

basilikum wrote: post #95
johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

It doesn't detect "real browsers". It simply adds some friction and is niche enough that AI scrapers have not bothered to bypass it yet.

Most of the friction is just JS overhead for the computations, a compiled solver is like 1000x faster. If Anubis ever gets popular enough that scrapers care, it would be trivial to defeat. And last I checked you could bypass it by just modifying the user agent

Re: A year of fighting scrapers on my 1.5 million-page website

#289

Post by antiterra »

aprdm wrote: post #237
sdellis wrote: post #219

Earlier quoted context omitted.

Because AI cannot be trusted as a reliable source of information. Information is more trustworthy when it comes from from primary sources on the open web.

Why not ? Can google be trusted ? Or facebook ? How do people have been accessing the internet for the last decade you reckon and how's this any different ?

I routinely run into Claude Opus and Fable making basic mistakes like misunderstanding a simple negation, which would be on top of whatever reliability issues there are with a source. I think that means it is functionally very different.

Re: A year of fighting scrapers on my 1.5 million-page website

#290

Post by johneth »

mmh0000 wrote: post #254
ballooney wrote: post #169

Earlier quoted context omitted.

This is such a grim thing to read.

I disagree. I personally consider these my biggest problems with the Web: - Bias, specifically commercial bias - Webpage formatting: every website looks different, hides the information I want in different places. Sure, it "looks pretty", but I don't want pretty; I want info - Scams/SEO/etc... The LLMs are very good at reading from multiple sources, parsing, and presenting only the data in a consistent format. There…

LLMs are not unbiased. Nothing is unbiased.
Post reply on HN