281–288 of 288 posts
Earlier quoted context omitted.
As someone on the browsing end, I love Anubis. I've only seen it a couple of times, but it sparks joy. It's rather refreshing compared to Cloudfare, which will usually make me immediately close the page and not bother with whatever content was behind it.
It really reminds me of old Internet, when things were allowed to be fun. Not this tepid corporate-approved landscape we have now.
I wouldn't recommend to show different versions of the site to search robots, as they probably have mechanisms that track differences, which could potentially lead to a lower ranking or a ban.
How can they track differences if they have access to only one version?
If you take a look at any website, even an unpopular one, you will see that there are hundreds of bots every day, and it's impossible to recognize what any of them is doing and why.
I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…
>I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! An even more insane idea -- minding the idea here is porn is radioactive to AI data training scrapers -- is there is something the powers that be view as far more disruptive and against community guidelineish than porn. And that would be wrongthink. The narratives. The historic narratives. The wok…
Earlier quoted context omitted.
I have a site with a complete and accurate sitemap.xml describing when its ~6k pages are last updated (on average, maybe weekly or monthly). What do the bots do? They scrape every page continuously 24/7, because of course they do. The amount of waste going into this AI craze is just obscene. It's not even good content.
It would be interesting if someone made a map that depicts the locations of the ip addresses that are sending so many requests, over the course of a day maybe.
This is very hacker-like thinking, using tech's biases against it! I can't help but feel like we're all doing it wrong against scraping. Cloudflare is not the answer, in fact, I think that they lost their geek cred when they added their "verify you are human" challenge screen to become the new gatekeeper of the internet. That must remain a permanent stain on their reputation until they make amends. Are there any open…
Earlier quoted context omitted.
If the LLMs are the "new Google" one solution would be for them to pay you when scraping your content, so you both have an incentive, you're more willing to be scraped and they'll try to not abuse you because it will cost them at every visit. If your content is valuable and requested on prompts they will scrape you more and so on. I can't see other solutions honestly. For now they decided to go full evil and abuse ev…
No disrespect to op, but I'm baffled as to how people keep coming up with ideas like this as if they are viable. Google is never ever ever ever going to "pay to scrape." I'm genuinely baffled as to how people think it would possibly come to this.
Earlier quoted context omitted.
If the LLMs are the "new Google" one solution would be for them to pay you when scraping your content, so you both have an incentive, you're more willing to be scraped and they'll try to not abuse you because it will cost them at every visit. If your content is valuable and requested on prompts they will scrape you more and so on. I can't see other solutions honestly. For now they decided to go full evil and abuse ev…
or turn your blog into a frontend/backend combo. keep the frontend as an SPA so that the page has nothing on it. have your backend send data in encrypted format and the AI scrapers would need to do a tonne of work in order to figure out what your data is. If everyone uses a different key and different encryption algorithm suddenly all their server time is busted decrypting stuff
Earlier quoted context omitted.
Maybe he wanted to verify whether what I was saying was true and asked ChatGPT, then tried to be helpful by pasting the response here?
Maybe I'm getting too jaded but I'm struggling to be quite that charitable. The entireity of the human-written text in that comment was "From ChatGPT:" and it was formatted as though it was a slam-dunk "you're wrong, the computer says so" (imagine it was "From Wikipedia" followed by a quote disagreeing with you instead). I'm sure some people do what you describe but then I would expect at least a little bit more expl…
My thought process was that the original comment was based on their personal experiences and since ChatGPT is trained on a large dataset, it may offer a different perspective derived from experiences of a lot more people.
> "you're wrong, the computer says so"
My thought: you're knowledge may be limited, this is what a computer trained on a lot more data says: