Singing copyrighted Billy Joel to make your footage unusable for reality television; thanks 30 Rock for an early view into this dystopian strategy
Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
221–230 of 288 posts
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#222I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…
[flagged]
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#223Earlier quoted context omitted.
I'm on the free tier, but I also watch my logs. The vast majority of the traffic I'm getting are scrapers and vulnerability scanners, a lot of them coming through residential proxies and other "laundered" egress points. I honestly don't think that Cloudflare is on top of the problem at all. They claim to be blocking abuse, but in my experience, most of the badness gets through.
when you combine a residential proxy with a tool like curl-impersonate (there are libraries in Go for this type of fingerprint spoofing now) they dont even show up as scrapers anymore, just users. especially when they adjust timings to mimic humans. clouflare only blocks the most dumb of bots, there are still a lot of them. this is why cloudflare will issue javascript challenges to you even when you are using google…
Is "residential proxy" another name for an hacked/owned computer that the bots have access to? Or are there legitimate services that sell access to residential IPs?
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#224Hidden links to porn sites? Lightweights.
If that's what it takes to fight back against AI crawlers, users will have to accept a fair amount of actually visible porn in blogs, maybe also on Wikipedia.
This is not enshittification, it's progress.
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#225I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…
[flagged]
Which cartoon are you referring to? The version of Anubis I installed only has the G-rated default images.
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#226This is very hacker-like thinking, using tech's biases against it!
I can't help but feel like we're all doing it wrong against scraping. Cloudflare is not the answer, in fact, I think that they lost their geek cred when they added their "verify you are human" challenge screen to become the new gatekeeper of the internet. That must remain a permanent stain on their reputation until they make amends.
Are there any open source tools we could install that detect a high number of requests and send those IP addresses to a common pool somewhere? So that individuals wouldn't get tracked, but bots would? Then we could query the pool for the current request's IP address and throttle it down based on volume (not block it completely). Possibly at the server level with nginx or at whatever edge caching layer we use.
I know there may be scaling and privacy issues with this. Maybe it could use hashing or zero knowledge proofs somehow? I realize this is hopelessly naive. And no, I haven't looked up whether someone has done this. I just feel like there must be a bulletproof solution to this problem, with a very simple explanation as to how it works, or else we've missed something fundamental. Why all the hand waving?
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#227Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#228Why is it so bad that AIs scrape your self-hosted blog?
because serving requires resources
What specific resources are we referring to here?
Are AI vendors re-crawling the whole blog repeatedly, or do they rely on caching primitives like ETag/If-Modified-Since (or hashes) to avoid fetching unchanged posts?
Also: is the scraping volume high enough to cause outages for smaller sites?
Separately, I see a bigger issue: blog content gets paraphrased and reproduced by AIs without clearly mentioning the author or linking back to the original post. It feels like you often have to explicitly ask the model for sources before it will surface the exact citations.
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#229Ah, I wonder if corporate proxies will end up flagging your blog as porn, if you protect it this way?
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#230Earlier quoted context omitted.
That’s so many scrapers. There must be a ton of companies with very large document collections at this point, and it really sucks that they don’t at least do us the courtesy of indexing them and making them available for keyword search, but instead only do AI. It’s kind of crazy how much scraping goes on and how little search engine development goes on. I guess search engines aren’t fashionable. Reminds me of this ar…
> There must be a ton of companies with very large document collections at this point See, I don't think there is, I don't think they want that expense. It's basically the Linus Torvalds philosophy of data storage, if it's on the Internet, I don't need a backup. While I have absolutely no proof of this, I'd guess that many AI companies just crawl the Internet constantly, never saving any of the data. We're seeing som…
Could be. Can you train a model without saving things though?