Earlier quoted context omitted.
Isn't your whole product scraping other sites and summarizing? I also don't see clear sources for your info where it is displayed, so it seems like you do the exact same thing.
If that was it, wouldn't Claude go directly to the canonical source?
A year of fighting scrapers on my 1.5 million-page website
461–470 of 476 posts
Re: A year of fighting scrapers on my 1.5 million-page website
#462Earlier quoted context omitted.
This is such a grim thing to read.
How is that grim? It's the dream of the Semantic Web coming true, just by different means than planed.
Re: A year of fighting scrapers on my 1.5 million-page website
#463Earlier quoted context omitted.
The social contract of accepting scraping for visibility was always tenuous and is now fully dead. But it’s shocking the number of people who are essentially victim blaming here. You should spend YOUR time to optimize your free website so MY use case is not impacted. How is that not incredibly selfish on its face?
The social contract for putting a website on the public internet involves following standards and making your website accessible. You SHOULD make sure that your website is readable to people no matter what user agent they are using as long as it's standards compliant. For example, making a website that only works in Edge and refuses to load for anything else would be a shitty practice. You are free to do it. There's…
Re: A year of fighting scrapers on my 1.5 million-page website
#464Earlier quoted context omitted.
> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. And this is a complaint? I find nothing in this statement to sympathize with whatsoever. Isn't the contract of, essentially everything, that effort is required to obtain something worthwhile? Please let me know where I'm getting this long-standing, fairly fundamental understanding of the wo…
Because the effort you are talking about is completely unnecessary. Imagine every grocery store would require you do a little dance when you buy a carton of milk. Your vision needs some narrowing (which I believe will justify the parent point), because as-is it has obvious counterexamples. I believe your problem is that "effort" is unspecified. "some effort" would make the statement correct, but some effort does not…
Re: A year of fighting scrapers on my 1.5 million-page website
#465> Sorry, you have been blocked You are unable to access patronview.com Why have I been blocked? > This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data. Thank you, from Vietnam.
Re: A year of fighting scrapers on my 1.5 million-page website
#466Regarding the residental botnets: check their TLS fingerprint, they usually (at least from my experience) have the same few JA4 hashes that don‘t match with modern browsers
Re: A year of fighting scrapers on my 1.5 million-page website
#467Re: A year of fighting scrapers on my 1.5 million-page website
#468Earlier quoted context omitted.
I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.
that's really smart of them. Just pay to use the service you need.
Re: A year of fighting scrapers on my 1.5 million-page website
#469Earlier quoted context omitted.
you really revved the weebs with this one
It seems to come up every time this complaint arises. "I want to use an open source project without being associated with furries" isn't that unreasonable of a request.
Re: A year of fighting scrapers on my 1.5 million-page website
#470Earlier quoted context omitted.
It seems to come up every time this complaint arises. "I want to use an open source project without being associated with furries" isn't that unreasonable of a request.
My understanding is that it's nigh-impossible to run a website without having a dependency on furry-maintained infra (even just basic stuff like BGP)