Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

github.com

Post reply on HN
Print view View original

191–200 of 288 posts

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#191

Post by prmoustache »

anonymous908213 wrote: post #67
kstrauser wrote: post #11

I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…

As someone on the browsing end, I love Anubis. I've only seen it a couple of times, but it sparks joy. It's rather refreshing compared to Cloudfare, which will usually make me immediately close the page and not bother with whatever content was behind it.

Anyone is free to replace the cat girl with an actual cat or a vintage computer logo or whatnot anyway.

My issue is that it blocks away people using browsers without javascript.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#192

Post by prmoustache »

vivzkestrel wrote: post #173
hollowturtle wrote: post #151

Earlier quoted context omitted.

Ehm what would stop ai scrapers from using a browser like a normal user would? Google bot already does, it can execute js and can read spa client side generated content, so it proves can be done at scale, and I'm pretty sure some ai scrapers already do

rate limit per ip that progressively keeps decreasing req/mins every few mins?

What if scrapers ips are millions of smartphones? If I was as evil as an AI scraper company that is not obeying robots.txt I would totally build/buy thousands of small games/apps for mobiles to use them as jumphosts to scape the web. This is probably happening already.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#193

Post by phyzome »

efilife wrote: post #144
cyphar wrote: post #139

Earlier quoted context omitted.

Yeah, I really have to wonder what the thought process is behind leaving such a comment. When people first started doing it I wondered if it was some kind of guerrilla outrage marketing campaign.

Maybe he wanted to verify whether what I was saying was true and asked ChatGPT, then tried to be helpful by pasting the response here?

Yeah, I agree that that's likely the thought process. It just happens to be the opposite of helpful.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#194

Post by timpera »

PunchyHamster wrote: post #159
anonymous908213 wrote: post #67

Earlier quoted context omitted.

As someone on the browsing end, I love Anubis. I've only seen it a couple of times, but it sparks joy. It's rather refreshing compared to Cloudfare, which will usually make me immediately close the page and not bother with whatever content was behind it.

My experience with it is that it somehow took 20 seconds to load (site might've been hn-hugged at the time), only to "protect" some fucking static page instead of just serving that shit in the first place rather than wasting CPU on... whatever it was doing to cause delay

Same experience for me. I tried it on a low-end smartphone and the Anubis challenge took about 45 seconds to complete.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#195

Post by santiagobasulto »

Offtopic: when did js/ts apps get so complicated? I tried to browse the repo and there are so many configuration files and directories for such a simple functionality that should be 1 or 2 modules. It reminds me of the old Java days.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#196

Post by prmoustache »

reconnecting wrote: post #12

I wouldn't recommend to show different versions of the site to search robots, as they probably have mechanisms that track differences, which could potentially lead to a lower ranking or a ban.

How can they track differences if they have access to only one version?

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#197

Post by socialcommenter »

asphero wrote: post #89

Interesting approach. The scraper-vs-site-owner arms race is real. On the flip side of this discussion - if you're building a scraper yourself, there are ways to be less annoying: 1. Run locally instead of from cloud servers. Most aggressive blocking targets VPS IPs. A desktop app using the user's home IP looks like normal browsing. 2. Respect rate limits and add delays. Obvious but often ignored. 3. Use RSS feeds wh…

Without wanting to upset anyone - what makes you interested in sharing tips for team scraper?

(Overgeneralising a bit) site owners are mostly cting for public benefit whereas scrapers act for their own benefit/for private interests.

I imagine most people would land on team site-owner, if they were asked. I certainly would.

P.S. is the best way to scrape fairly just to respect robots.txt?

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#198

Post by prmoustache »

montroser wrote: post #63

Reminds me of poisoning bot responses with zip bombs of sorts: https://idiallo.com/blog/zipbomb-protection

I was thinking of adding links to zip bombs that would not be shown to the users unless they clicks in a one pixel area on the screen in the down/left corner but then I realized some people have browsers/extensions that preload links to show thumnails and I would totally zip bomb them.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#199

Post by nephihaha »

n1xis10t wrote: post #95
tokioyoyo wrote: post #91

Earlier quoted context omitted.

Large scale scraping tech is not as sophisticated as you'd think. A significant chunk of it is "get as much as possible, categorize and clean up later". Man, I really want the real web of the 2000s back, when things felt "real" more or less... how can we even get there.

If people start making search engines again and there is more competition for Google, I think things would be pretty sweet.

There are other search engines, they've just been marginalised. Even something as mainstream as Bing has been pushed to the side.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#200

Post by xena »

63stack wrote: post #183
xena wrote: post #108

Earlier quoted context omitted.

So I can plan it, how much detail do you want? Here's what I have about the prototype: https://anubis.techaro.lol/docs/admin/honeypot/overview

This is amazing, I was just wondering about if it's possible to tie anubis together with iocaine, but it seems you already thought of that.

It's slightly different in subtle ways. If I recall iocaine makes you configure a subprocess that it executes to generate garbage. One rule I have for Anubis in the code is that fork()/exec() are banned. So the pluggable garbage generator is gonna be powered by CGI handlers compiled to WebAssembly. It should be fun!
Post reply on HN