Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

github.com

Post reply on HN
Print view View original

261–270 of 288 posts

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#261

Post by 63stack »

vivzkestrel wrote: post #173
hollowturtle wrote: post #151

Earlier quoted context omitted.

Ehm what would stop ai scrapers from using a browser like a normal user would? Google bot already does, it can execute js and can read spa client side generated content, so it proves can be done at scale, and I'm pretty sure some ai scrapers already do

rate limit per ip that progressively keeps decreasing req/mins every few mins?

Just skip the whole encryption/decryption shebang then.

But no, this does not work, scrapers are using residential ips, and they have enough that they can rotate between them if they get blocked.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#263

Post by opem »

anonymous908213 wrote: post #67
kstrauser wrote: post #11

I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…

As someone on the browsing end, I love Anubis. I've only seen it a couple of times, but it sparks joy. It's rather refreshing compared to Cloudfare, which will usually make me immediately close the page and not bother with whatever content was behind it.

yes, very true! Anubis is a hell lot better than cloudflare turnstile or its older cousin sister google recaptcha.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#264

Post by rogerwong »

Terrible idea, but I do have a question. Like many, I have a self-hosted website and have seen a spike in traffic, particularly from Singapore.

Seems like the consensus is that these are AI scrapers. But could they also be from answer engines like Perplexity, or searches from APIs like Tavily?

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#265

Post by mannanj »

Is a suitable solution to require visitors to fill out intent for why they came, and align that with your approved lists of supported intents, AND quiz them on some personal insider knowledge that only reasonable past visitors or new visitors who heard of you would have?

Like the credibility social proof of an introduction of a person into a social group. "Here's John, he likes Cats. I know him from School."

The filtering algorithm asks "Who who are you?" -> "What is your intent?" -> "How did you hear about me?" and stops visitors from proceeding until answered. The additional validation steps might kick away visitors but it also might protect you from spammers if you throw a minimally frictional challenge. Use cookies to not require this on every visit. Most LLMs would have the knowledge required to pass & for scrapers it's more costly to acquire this for a site than pay 128mb of ram to pass the Anubis approach.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#266

Post by stevage »

thethingundone wrote: post #19

I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.

I'd love to know the answer to this question. AI scrapers wanting everything on the internet makes sense to me. But I don't understand how that leads to every site being hit hundreds of thousands of times per day.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#267

Post by n1xis10t »

tonymet wrote: post #262
kstrauser wrote: post #225

Earlier quoted context omitted.

Which cartoon are you referring to? The version of Anubis I installed only has the G-rated default images.

[flagged]

If you keep referring to non-explicit material as pornography, you will continue to confuse people.

If you have an objection to the image other than it’s pornographic status, please word it clearly.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#268

Post by kstrauser »

tonymet wrote: post #262
kstrauser wrote: post #225

Earlier quoted context omitted.

Which cartoon are you referring to? The version of Anubis I installed only has the G-rated default images.

[flagged]

I'm being sincere here: I genuinely don't know what you're talking about.

I'm referring to these default images: https://github.com/TecharoHQ/anubis/tree/main/docs/static/im.... Do you mean something different?

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#269

Post by tonymet »

kstrauser wrote: post #268
tonymet wrote: post #262

Earlier quoted context omitted.

[flagged]

I'm being sincere here: I genuinely don't know what you're talking about. I'm referring to these default images: https://github.com/TecharoHQ/anubis/tree/main/docs/static/im... . Do you mean something different?

Similar but yeah. Whatever prompts during the challenge . It’s creepy , out of context and inappropriate .

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#270

Post by tonymet »

n1xis10t wrote: post #267
tonymet wrote: post #262

Earlier quoted context omitted.

[flagged]

If you keep referring to non-explicit material as pornography, you will continue to confuse people. If you have an objection to the image other than it’s pornographic status, please word it clearly.

I was clear on the issue
Post reply on HN