Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

github.com

Post reply on HN
Print view View original

271–280 of 288 posts

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#271

Post by conrs »

zackmorris wrote: post #226

This is very hacker-like thinking, using tech's biases against it! I can't help but feel like we're all doing it wrong against scraping. Cloudflare is not the answer, in fact, I think that they lost their geek cred when they added their "verify you are human" challenge screen to become the new gatekeeper of the internet. That must remain a permanent stain on their reputation until they make amends. Are there any open…

I've been exploring decentralized trust algorithms lately, and so reading this was nice. I've a similar intuition - for every advance in scraping detection, scrapers will learn too, and so it's an ongoing war of mutations, but no real victor.

The internet has seen success with social media content moderation and so it seems natural enough that an application could exist for web traffic itself. Hosts being able to "downvote" malicious traffic, and some sort of decay mechanism given IP's recycling. This exists in a basic sense with known TOR exit nodes and known AWS, GCP IP's, etc.

That said, we probably don't have the right building blocks yet, IP's are too ephemeral, yet anything more identity-bound is a little too authoritarian IMO. Further, querying something for every request is probably too heavy.

Fun to think about, though.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#272

Post by rglynn »

ProllyInfamous wrote: post #123
ddtaylor wrote: post #101

Earlier quoted context omitted.

Cloudflare has created a bit of grief with regular users getting spammed with "prove your human" requests.

Yes, e.g: I'll immediately close any attempt at Cloudfare's verification.

Out of interest, why that extreme? Just out of principle or some other reason?

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#273

Post by throw10920 »

zhengyi13 wrote: post #253
throw10920 wrote: post #213

Earlier quoted context omitted.

Is there any evidence that this has actually happened?

Even if there isn't (yet?), there's probably someone who's honestly thinking this is potentially a viable business model and at least napkin-mathing it out.

So, it hasn't happened, and you're just making stuff up.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#274

Post by rl3 »

>The problem with that approach is that it will absolutely nuke your website's SEO. So fuzzycanary also checks user agents and won't show the links to legitimate search engines, so Google and Bing won't see them.

Those legitimate search engines will then totally feed much of what they scrape into AI. Granted, last I checked they're at least well-behaved crawlers.

I kind of like this idea sans SEO carve-out for the scenario where one just wants to link their blog around to friends without having to worry about it getting popular, and it reduces the chances identity thieves or other malicious actors would target it.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#275

Post by Rastonbury »

rglynn wrote: post #272
ProllyInfamous wrote: post #123

Earlier quoted context omitted.

Yes, e.g: I'll immediately close any attempt at Cloudfare's verification.

Out of interest, why that extreme? Just out of principle or some other reason?

Not OP but it isn't super extreme if you are just surfing, it's like if the site is slow to load sometimes I wasn't that invested to use your site anyway

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#276

Post by 654wak654 »

Looking through all the methods people are developing and proposing in this thread, there is a story developing where the "clean" machines are pushing humans to devolve into toxic porn-crazed racists with stolen material.

Makes me wish I was a good enough writer to develop this into something. Maybe I can use an LLM to write it...

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#277

Post by 654wak654 »

654wak654 wrote: post #276

Looking through all the methods people are developing and proposing in this thread, there is a story developing where the "clean" machines are pushing humans to devolve into toxic porn-crazed racists with stolen material. Makes me wish I was a good enough writer to develop this into something. Maybe I can use an LLM to write it...

Ah wait this is literally in the Matrix where humanity darkened the sky.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#279

Post by rurban »

n1xis10t wrote: post #15
kstrauser wrote: post #11

I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…

That’s so many scrapers. There must be a ton of companies with very large document collections at this point, and it really sucks that they don’t at least do us the courtesy of indexing them and making them available for keyword search, but instead only do AI. It’s kind of crazy how much scraping goes on and how little search engine development goes on. I guess search engines aren’t fashionable. Reminds me of this ar…

Just block all the big hosters IP ranges, when they ignore robots.txt.

For fun add long timeouts and huge content sizes. No private individual will browse from there, and all scrapers will do.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#280

Post by ProllyInfamous »

rglynn wrote: post #272
ProllyInfamous wrote: post #123

Earlier quoted context omitted.

Yes, e.g: I'll immediately close any attempt at Cloudfare's verification.

Out of interest, why that extreme? Just out of principle or some other reason?

My main terminal uses a PiHole with 120,000+ blacklist rules (not Cloudfare specifically — I allow most CDN's). This includes an entire blackout of Google/Facebook products, as well as most tracking/analytics services.

For example, I do not allow reCAPTCHA.

As a similar commentor noted, when just casually browsing I don't really have any desire to try hard to read random content. Should I absolutely need to access some information garden-walled behind Cloudfare: I have another computer that uses much less restrictive black-listing.

Post reply on HN