Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

github.com

Post reply on HN
Print view View original

251–260 of 288 posts

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#251

Post by lelanthran »

kelvinjps10 wrote: post #243
mrweasel wrote: post #176

Earlier quoted context omitted.

> There must be a ton of companies with very large document collections at this point See, I don't think there is, I don't think they want that expense. It's basically the Linus Torvalds philosophy of data storage, if it's on the Internet, I don't need a backup. While I have absolutely no proof of this, I'd guess that many AI companies just crawl the Internet constantly, never saving any of the data. We're seeing som…

Where did Linus Torvalds expressed this philosophy I have never seen it

> Where did Linus Torvalds expressed this philosophy I have never seen it

https://www.goodreads.com/quotes/574706-only-wimps-use-tape-...

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#252

Post by lelanthran »

amypetrik8 wrote: post #244
kstrauser wrote: post #11

I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…

>I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! An even more insane idea -- minding the idea here is porn is radioactive to AI data training scrapers -- is there is something the powers that be view as far more disruptive and against community guidelineish than porn. And that would be wrongthink. The narratives. The historic narratives. The wok…

I like this better than of NSFW links; just include a (possible LLM generated) paragraph about not supporting transitions in minor children. Or perhaps that libraries that remove instructional booklets for how to have same-sex intercourse aren't actually banning the books.

That sort of thing; nothing that 80% of people object to (so there's no problem if someone actually sees it), but something that definitely triggers the filters.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#253

Post by zhengyi13 »

throw10920 wrote: post #213
PeterStuer wrote: post #156

Earlier quoted context omitted.

Or some anti-ddos/bot companies using ultra cheap scraping services to annoy you enough to get you into their "free" anti bot protection, so they can charge the few real ai scrapers for access to your site.

Is there any evidence that this has actually happened?

Even if there isn't (yet?), there's probably someone who's honestly thinking this is potentially a viable business model and at least napkin-mathing it out.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#254

Post by lelanthran »

mook wrote: post #237
vivzkestrel wrote: post #118

Earlier quoted context omitted.

or turn your blog into a frontend/backend combo. keep the frontend as an SPA so that the page has nothing on it. have your backend send data in encrypted format and the AI scrapers would need to do a tonne of work in order to figure out what your data is. If everyone uses a different key and different encryption algorithm suddenly all their server time is busted decrypting stuff

That makes the mistake of thinking they'll care. Most likely they'll just keep downloading the encrypted garbage and never notice.

> That makes the mistake of thinking they'll care. Most likely they'll just keep downloading the encrypted garbage and never notice.

Not going to happen. They aren't going to add encrypted garbage to their dataset.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#255

Post by dvfjsdhgfv »

zackmorris wrote: post #226

This is very hacker-like thinking, using tech's biases against it! I can't help but feel like we're all doing it wrong against scraping. Cloudflare is not the answer, in fact, I think that they lost their geek cred when they added their "verify you are human" challenge screen to become the new gatekeeper of the internet. That must remain a permanent stain on their reputation until they make amends. Are there any open…

Your approach to GenAI scrapers is similar to our fight with email spam. The reason email spam got solved was because the industry was interested in solving it. But this issue got the industry split: without scraping, GenAI tools are less functional. And there is some serious money involved, so they will use whatever means necessary, technical and legal, to fight such initiatives.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#256

Post by smegger001 »

zackmorris wrote: post #226

This is very hacker-like thinking, using tech's biases against it! I can't help but feel like we're all doing it wrong against scraping. Cloudflare is not the answer, in fact, I think that they lost their geek cred when they added their "verify you are human" challenge screen to become the new gatekeeper of the internet. That must remain a permanent stain on their reputation until they make amends. Are there any open…

maybe some proof of work scheme to load page content with increasing difficulty based on ip address behavior profiling.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#257

Post by kstrauser »

zhengyi13 wrote: post #253
throw10920 wrote: post #213

Earlier quoted context omitted.

Is there any evidence that this has actually happened?

Even if there isn't (yet?), there's probably someone who's honestly thinking this is potentially a viable business model and at least napkin-mathing it out.

My napkin mathing is that their ROI would be negative. That's a lot of compute and bandwidth they'd have to pay for even if they were just throwing away the results.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#258

Post by n1xis10t »

socialcommenter wrote: post #242
n1xis10t wrote: post #235

Earlier quoted context omitted.

I think "scraper vs siteowners" is a false dichotomy. Scrapers will always need to exist as long as we want search engines and archival services. We will need small versions of these services to keep popping up every now and then to keep the big guys on their toes, and the smaller guys need advice for scraping politely.

That's fair - though are we in an isolated bout of "every now and then" or has AI created a new normal of abuse (e.g. of robots.txt)? Hopefully we're at a local maximum and some of the scrapers perpetrating harmful behaviours will soon pull their heads in.

Hopefully. It would also be nice to see more activity in the actual search engine and archiving market; there really isn’t much right now.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#259

Post by prmoustache »

stefanka wrote: post #205
prmoustache wrote: post #191

Earlier quoted context omitted.

Anyone is free to replace the cat girl with an actual cat or a vintage computer logo or whatnot anyway. My issue is that it blocks away people using browsers without javascript.

How can one do this? Did not find it in the docs

The 3 images are in the repo, you can replace them and rebuild or point to other ones in the templates.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#260

Post by ATechGuy »

zackmorris wrote: post #226

This is very hacker-like thinking, using tech's biases against it! I can't help but feel like we're all doing it wrong against scraping. Cloudflare is not the answer, in fact, I think that they lost their geek cred when they added their "verify you are human" challenge screen to become the new gatekeeper of the internet. That must remain a permanent stain on their reputation until they make amends. Are there any open…

Scrapers use residential IP proxies, so blocking based on IP addresses is not a solution.
Post reply on HN