Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

461–470 of 476 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#461

Earlier quoted context omitted.

Isn't your whole product scraping other sites and summarizing? I also don't see clear sources for your info where it is displayed, so it seems like you do the exact same thing.

If that was it, wouldn't Claude go directly to the canonical source?

Why would it if it finds a source that has summarized them somewhat correctly?

Re: A year of fighting scrapers on my 1.5 million-page website

#462

Earlier quoted context omitted.

This is such a grim thing to read.

How is that grim? It's the dream of the Semantic Web coming true, just by different means than planed.

You know that the reason the semantic web didn't work out is that it only benefits a small number of participants (aggregators), right?

Re: A year of fighting scrapers on my 1.5 million-page website

#463
post #155

Earlier quoted context omitted.

The social contract of accepting scraping for visibility was always tenuous and is now fully dead. But it’s shocking the number of people who are essentially victim blaming here. You should spend YOUR time to optimize your free website so MY use case is not impacted. How is that not incredibly selfish on its face?

The social contract for putting a website on the public internet involves following standards and making your website accessible. You SHOULD make sure that your website is readable to people no matter what user agent they are using as long as it's standards compliant. For example, making a website that only works in Edge and refuses to load for anything else would be a shitty practice. You are free to do it. There's…

Funny how your idea of a social contract only cares about obligations for one side.

Re: A year of fighting scrapers on my 1.5 million-page website

#464

Earlier quoted context omitted.

> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. And this is a complaint? I find nothing in this statement to sympathize with whatsoever. Isn't the contract of, essentially everything, that effort is required to obtain something worthwhile? Please let me know where I'm getting this long-standing, fairly fundamental understanding of the wo…

Because the effort you are talking about is completely unnecessary. Imagine every grocery store would require you do a little dance when you buy a carton of milk. Your vision needs some narrowing (which I believe will justify the parent point), because as-is it has obvious counterexamples. I believe your problem is that "effort" is unspecified. "some effort" would make the statement correct, but some effort does not…

Grocery stores do in fact make you walk through a maze to get to the items you want and absolutely would block any technology that would let you get only what you had in mind without seeing what else is on offer.

Re: A year of fighting scrapers on my 1.5 million-page website

#465

> Sorry, you have been blocked You are unable to access patronview.com Why have I been blocked? > This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data. Thank you, from Vietnam.

I see popular blog posts, click it and see its blocked from India.

Re: A year of fighting scrapers on my 1.5 million-page website

#468

Earlier quoted context omitted.

I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.

that's really smart of them. Just pay to use the service you need.

It's really smart of them to manufacture a negative sentiment about the project because the sexual fetish icon gets injected into all kinds of places where it is absolutely inappropriate?

Re: A year of fighting scrapers on my 1.5 million-page website

#469

Earlier quoted context omitted.

you really revved the weebs with this one

It seems to come up every time this complaint arises. "I want to use an open source project without being associated with furries" isn't that unreasonable of a request.

My problem is more that I don't want open source infrastructure I don't control to be infected by this. For my own website I can just make it efficient enough not to need hacks like this.

Re: A year of fighting scrapers on my 1.5 million-page website

#470

Earlier quoted context omitted.

It seems to come up every time this complaint arises. "I want to use an open source project without being associated with furries" isn't that unreasonable of a request.

My understanding is that it's nigh-impossible to run a website without having a dependency on furry-maintained infra (even just basic stuff like BGP)

There is a difference between some maintainer somewhere being involved in a subculture and graphics from it being the first thing you see when you visit an unrelated website.
Post reply on HN