LLM are the best way to read the web. right to the point, no ADS
A year of fighting scrapers on my 1.5 million-page website
351–360 of 450 posts
Re: A year of fighting scrapers on my 1.5 million-page website
#352[dead]
Re: A year of fighting scrapers on my 1.5 million-page website
#353The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.
Are you gonna police the means through which I'm browsing the internet? If you want to, you are free to put your stuff behind the paywall and give the key only to people who agree to obey your conditions. If you put it in the open you can't make conditions. That's what publishing means. Author can't make demands in what manner their book should be read.
Re: A year of fighting scrapers on my 1.5 million-page website
#354> My visitor stats got so polluted I couldn't trust my own numbers. This was the one that hurt. I'm trying to run a business here. I want to know what real people read on my site so I know what to build next. I couldn't see them through the bots.
Are bots the solution to nosy websites that want to know things about their visitors?
Re: A year of fighting scrapers on my 1.5 million-page website
#355Earlier quoted context omitted.
In any case, if you run a professional website, this immediately comes off as juvenile and/or amateurish. And am y people just assume it's part of your website.
Money can be used to purchase goods or services, such as an unbranded version.
[flagged]
Re: A year of fighting scrapers on my 1.5 million-page website
#356Re: A year of fighting scrapers on my 1.5 million-page website
#357Earlier quoted context omitted.
I don't disagree, but there is a sliding scale here. For instance, I wanted to buy a piece of equipment the other day from a local company for a specific usecase. I wanted to find a specific price/weight/specs ratio, and asked an llm to loop through the 20 or so items, fetch their page and calculate and present some values for each. This then led to me going and buying the one I found. So the llm was mainly just an e…
No, there isn't, it's a bot. Justifying it with a strawman case study doesn't change it. This is giving "why is this plane full? What in doing is important but what is everyone else doing here?"
No, the first case I used their website in a way that gave them a sale, which is why they have it. Why is it a straw man case when it's reality? Don't argue in bad faith, please. Express your reasoning, like I did, don't just dismiss me because you disagree..
Re: A year of fighting scrapers on my 1.5 million-page website
#358Earlier quoted context omitted.
It's actually a genius idea. If you are someone who the professional presentation of not having an anime girl on the loading page is required, then you can afford to fork over the cash to fund development.
I guess the disconnect here is a bunch of HN'ers believing professional companies and websites want to attach their branding to a sexualized anime character and that they are willing to pay to remove it. Which one then wonders why they would install it in the first place.
What specific aspects of the image do you feel are sexualized?
Re: A year of fighting scrapers on my 1.5 million-page website
#359Earlier quoted context omitted.
Money can be used to purchase goods or services, such as an unbranded version.
[flagged]
> ... sexualized ...
Perhaps you should check if anybody else sees what you see. After browsing through every image I could find in the history of this project, I, at least, am convinced: any "sexualization" _you_ see is entirely in your head.
> ... in judeo christian societies
I really hope you're not trying to whitewash your own bias by alluding to those religions. From everything I've read of their texts, and interacted with practioners of those religions, I'm unaware of any bias that classifies _cartoon depictions of children_ as "sexual".
Perhaps you should try whitelabeling instead. Don't like the packaging? Package it yourself!
Re: A year of fighting scrapers on my 1.5 million-page website
#360Earlier quoted context omitted.
you really revved the weebs with this one
It seems to come up every time this complaint arises. "I want to use an open source project without being associated with furries" isn't that unreasonable of a request.
My understanding is that it's nigh-impossible to run a website without having a dependency on furry-maintained infra (even just basic stuff like BGP)