Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

github.com

Post reply on HN
Print view View original

161–170 of 288 posts

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#161

Post by PunchyHamster »

vivzkestrel wrote: post #142
chii wrote: post #129

Earlier quoted context omitted.

How does your normal users get access to the same contents? Or are you having the user solve an encryption puzzle to view it?

- the frontend has a decryption module that ll show users what they want to see, - the backend has an encryption module. - The bots and crawlers will see the encrypted text - Can someone who peeks deeply inside the client side code decrypt it? YES - Will 99% of the scrapers bother doing this? NO - The key can be anything, it could be a per session key agreed upon between the client and the server, a csrf token, or ev…

the AI will just run chrome instance

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#162

Post by PunchyHamster »

hollowturtle wrote: post #70
montroser wrote: post #58

This is a cute idea, but I wonder what is the sustainable solution to this emerging fundamental problem: As content publishers, we want our content to be accessible to everyone, and we're even willing to pay for server costs relative to our intended audience -- but a new outsized flood of scrapers was not part of the cost calculation, and that is messing up the plan. It seems all options have major trade-offs. We can…

If the LLMs are the "new Google" one solution would be for them to pay you when scraping your content, so you both have an incentive, you're more willing to be scraped and they'll try to not abuse you because it will cost them at every visit. If your content is valuable and requested on prompts they will scrape you more and so on. I can't see other solutions honestly. For now they decided to go full evil and abuse ev…

So they won't pay you and just scrape pages that have it public, and you will never get traffic from search again until you let them scrape

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#163

Post by PunchyHamster »

cyphar wrote: post #139
phyzome wrote: post #85

Earlier quoted context omitted.

There should be a new rule on HN: No posts that just go "I asked an LLM and it said..." You're not adding anything to the conversation.

Yeah, I really have to wonder what the thought process is behind leaving such a comment. When people first started doing it I wondered if it was some kind of guerrilla outrage marketing campaign.

There was no thought process

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#164

Post by bytehowl »

Let's imagine I have a blog and put something along these lines somewhere on every page: "This content is provided free of charge for humans to experience. It may also be automatically accessed for search indexing and archival purposes. For licensing information for other uses, contact the author."

If I then get hit by a rude AI scraper, what chances would I have to sue the hell out of them in EU courts for copyright violation (uhh, my articles cost 100k a pop for AI training, actually) and the de facto DDoS attack?

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#165

Post by ramonga »

n1xis10t wrote: post #13
kstrauser wrote: post #11

I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…

*anime jackalgirl Also you mentioned Anubis, so it’s creator will probably read this. Hi Xena!

what do people use to get keyword alerts in HN?

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#166

Post by symbogra »

idiotsecant wrote: post #134
tokioyoyo wrote: post #91

Earlier quoted context omitted.

Large scale scraping tech is not as sophisticated as you'd think. A significant chunk of it is "get as much as possible, categorize and clean up later". Man, I really want the real web of the 2000s back, when things felt "real" more or less... how can we even get there.

Have you ever listened to the 'high water mark' monologue from fear and loathing? It's pretty much just that. It was a unique time and it was neat that we got to see it, but it can't possibly happen again. https://www.youtube.com/watch?v=vUgs2O7Okqc

Thanks for reminding me about that, what a great monologue. I didn't really understand it when I was younger, but now I feel the same thing with regards to software engineering. There was a golden age which finally broke at the end of the 2010's.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#167

Post by wooger »

happymellon wrote: post #146
bobbiechen wrote: post #66

Earlier quoted context omitted.

There are lots of people pretending to be Google and friends. They far outnumber the real Googlebot, etc. and most people don't check the reverse DNS/IP list - it's tedious to do this for even well-behaved crawlers that publish how to ID themselves. So much for User Agent.

> So much for User Agent. User agent has been abused for so long, I forget a time when it wasn't. Anyone else remember having to fake being a Windows machine so that YouTube/Netflix would serve you content better than standard def, or banking portals that blocked you if your agent didn't say you were Internet Explorer?

I mean forget that, all modern desktop browsers (at least) start with the string 'Mozilla/5.0', still, in a world where Chrome is so dominant.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#168

Post by m4rtink »

anonymous908213 wrote: post #67
kstrauser wrote: post #11

I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…

As someone on the browsing end, I love Anubis. I've only seen it a couple of times, but it sparks joy. It's rather refreshing compared to Cloudfare, which will usually make me immediately close the page and not bother with whatever content was behind it.

Yep, Anubis-chan is super cute! :)
Post reply on HN