Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

github.com

Post reply on HN
Print view View original

171–180 of 288 posts

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#171

Post by Imustaskforhelp »

kstrauser wrote: post #80
n1xis10t wrote: post #76

Earlier quoted context omitted.

So maybe there are more people who like the “anime catgirl” than there are who think it’s weird

*anime jackalgirl ;-) Quite possibly. Or, in my case, I think it's more quirky and fun than weird. It's non-zero amounts of weird, sure, but far below my threshold of troublesome. I probably wouldn't put my business behind it. I'm A-OK with using it on personal and hobby projects. Frankly, anyone so delicate that they freak out at the utterly anodyne imagery is someone I don't want to deal with in my personal time. I…

For what its worth, I think that a UN/(Unicef?) website (not sure which one) did use anubis so maybe you can put it behind businesses too :)

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#172

Post by jeroenhd »

wazoox wrote: post #9

Isn't there a risk to get your blog blocked in corporate environment though? If it's a technical blog that would be unfortunate.

That depends on how terrible the middleboxes those corporate environments use are. If they only block actual malicious pages, it shouldn't be a problem unless the user un-hides the links and clicks on them.

There's a good chance corporate firewalls will end up blocking your domain if you do this but that sounds like a problem for the customers of those corporate firewalls to me.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#173

Post by vivzkestrel »

hollowturtle wrote: post #151
vivzkestrel wrote: post #142

Earlier quoted context omitted.

- the frontend has a decryption module that ll show users what they want to see, - the backend has an encryption module. - The bots and crawlers will see the encrypted text - Can someone who peeks deeply inside the client side code decrypt it? YES - Will 99% of the scrapers bother doing this? NO - The key can be anything, it could be a per session key agreed upon between the client and the server, a csrf token, or ev…

Ehm what would stop ai scrapers from using a browser like a normal user would? Google bot already does, it can execute js and can read spa client side generated content, so it proves can be done at scale, and I'm pretty sure some ai scrapers already do

rate limit per ip that progressively keeps decreasing req/mins every few mins?

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#175

Post by _lvbh »

anonymous908213 wrote: post #67
kstrauser wrote: post #11

I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…

As someone on the browsing end, I love Anubis. I've only seen it a couple of times, but it sparks joy. It's rather refreshing compared to Cloudfare, which will usually make me immediately close the page and not bother with whatever content was behind it.

As someone on the hosting end, Anubis has unfortunately been overused and thus scrapers, especially Huawei ones, bypass it. I've gone for go-away instead which is similar but more configurable in challenges

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#176

Post by mrweasel »

n1xis10t wrote: post #15
kstrauser wrote: post #11

I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…

That’s so many scrapers. There must be a ton of companies with very large document collections at this point, and it really sucks that they don’t at least do us the courtesy of indexing them and making them available for keyword search, but instead only do AI. It’s kind of crazy how much scraping goes on and how little search engine development goes on. I guess search engines aren’t fashionable. Reminds me of this ar…

> There must be a ton of companies with very large document collections at this point

See, I don't think there is, I don't think they want that expense. It's basically the Linus Torvalds philosophy of data storage, if it's on the Internet, I don't need a backup. While I have absolutely no proof of this, I'd guess that many AI companies just crawl the Internet constantly, never saving any of the data. We're seeing some of these scrapers go to great length attempting to circumvent any and all forms of caching, they aren't interested in having a two week old copy of anything.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#177

Post by mrweasel »

xena wrote: post #99
n1xis10t wrote: post #13

Earlier quoted context omitted.

*anime jackalgirl Also you mentioned Anubis, so it’s creator will probably read this. Hi Xena!

Ohai! I'm working on dataset poisoning. The early prototype generates vapid LinkedIn posts but future versions will be fully pluggable with WebAssembly.

Now I'm picturing an AI trained exclusively on LinkedIn posts. One could probably sell that model to an online ad agency for a pretty penny.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#179

Post by mrweasel »

thethingundone wrote: post #19

I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.

Why pay for storage when you do it for them?
Post reply on HN