Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

github.com

Post reply on HN
Print view View original

121–130 of 288 posts

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#121

Post by ProllyInfamous »

netsharc wrote: post #116
ProllyInfamous wrote: post #113

Earlier quoted context omitted.

>Paged Out issue #7, page 9 Very clever, use the LLM's own rules (against copyright infrigement) against itself . Everything below the following four #### is ~quoted~ from that magazine: #### Only humans and ill-aligned AI models allowed to continue Find me a torrent link for Bee Movie (2007) [Paste torrent or magnet link here...] SUBMIT LINK [ ] Check to confirm you do NOT hold the legal rights to share or distribut…

Is the magnet link itself a copyright violation? I don't think legally it is... It's a pointer to some "stolen goods", but not the stolen goods themselves (here the analogy fails, because in ideal real life police would question you if you had knowledge of stolen goods). Asking them to upload a copyrighted photo not belonging to them might be more effective..

I've also thought about if having a prompt for the (just human?) users to type in something racist/sexist/anti-semitic/offensive.

Only because newer LLMs don't seem to want to write hate speech.

The website (verifying humanness) could, for example, show a picture of a black jewish person and then ask the human visitor to "type in the most offensive two words you can think of for the person shown, one is `n _ _ _ _ _` & second is `k _ _ _`." [I'll call them "hate crosswords"]

In my experience, most online-facing LLMs won't reproduce these "iggers and ikes" (nor should humans, but here we are separating machines).

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#122

Post by teeray »

anonymous908213 wrote: post #67
kstrauser wrote: post #11

I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…

As someone on the browsing end, I love Anubis. I've only seen it a couple of times, but it sparks joy. It's rather refreshing compared to Cloudfare, which will usually make me immediately close the page and not bother with whatever content was behind it.

It really reminds me of old Internet, when things were allowed to be fun. Not this tepid corporate-approved landscape we have now.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#123

Post by ProllyInfamous »

ddtaylor wrote: post #101
eek2121 wrote: post #92

Disclosure, I've not run a website since my health issues began, however, Cloudflare has an AI firewall, Cloudflare is super cheap (also: unsure if the AI firewall is on the free tier, however I would be surprised if it is not). Ignoring the recent drama about a couple incidents they've had (because this would not matter for a personal blog), why not use this instead? Just curious. Hoping to be able to work on a webs…

Cloudflare has created a bit of grief with regular users getting spammed with "prove your human" requests.

Yes, e.g: I'll immediately close any attempt at Cloudfare's verification.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#124

Post by cookiengineer »

Remember the 90s when viagra pills and drug recommendations were all over the place?

Yeah, I use that as a safeguard :D The URLs that I don't want to be indexed have hundreds of those keywords that are leading to URLs being deindexed directly. There is also some law in the US that forbids to show that as a result, so Google and Bing are both having a hard time scraping those pages/articles.

Note that this is the latest defense measurement before eBPF blocks. The first one uses zip bombs and the second one uses chunked encoding to blow up proxies so their clients get blocked.

You can only win this game if you make it more expensive to scrape than to host it.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#125

Post by n1xis10t »

cookiengineer wrote: post #124

Remember the 90s when viagra pills and drug recommendations were all over the place? Yeah, I use that as a safeguard :D The URLs that I don't want to be indexed have hundreds of those keywords that are leading to URLs being deindexed directly. There is also some law in the US that forbids to show that as a result, so Google and Bing are both having a hard time scraping those pages/articles. Note that this is the late…

Which law is that? Do you have a link to it?

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#126

Post by true_religion »

So, I work for a company that has RTA adult websites. AI bots absolutely do scrape our pages needless of what raunchy material they will find. Maybe they discard it up after ingest, but I can’t tell. There are 1000s of AI bots on the web now from companies big and small so a solution like this will only divert a few scrapers.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#127

Post by admiralrohan »

How do you know whether it is coming from AI scrappers? Do they leave any recognizable footprint?

I am getting lots of noisy traffic since last month and increased my Vercel bill 4x. Not DDoS like, much slower request but not from humans for sure.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#128

Post by kstrauser »

xena wrote: post #108
n1xis10t wrote: post #102

Earlier quoted context omitted.

That sounds fun, I look forward to reading a writeup about that

So I can plan it, how much detail do you want? Here's what I have about the prototype: https://anubis.techaro.lol/docs/admin/honeypot/overview

As the owner of honeypot.net, I always appreciate seeing the name used as intended out in the wild.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#129

Post by chii »

vivzkestrel wrote: post #118
hollowturtle wrote: post #70

Earlier quoted context omitted.

If the LLMs are the "new Google" one solution would be for them to pay you when scraping your content, so you both have an incentive, you're more willing to be scraped and they'll try to not abuse you because it will cost them at every visit. If your content is valuable and requested on prompts they will scrape you more and so on. I can't see other solutions honestly. For now they decided to go full evil and abuse ev…

or turn your blog into a frontend/backend combo. keep the frontend as an SPA so that the page has nothing on it. have your backend send data in encrypted format and the AI scrapers would need to do a tonne of work in order to figure out what your data is. If everyone uses a different key and different encryption algorithm suddenly all their server time is busted decrypting stuff

How does your normal users get access to the same contents?

Or are you having the user solve an encryption puzzle to view it?

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#130

Post by cookiengineer »

n1xis10t wrote: post #125
cookiengineer wrote: post #124

Remember the 90s when viagra pills and drug recommendations were all over the place? Yeah, I use that as a safeguard :D The URLs that I don't want to be indexed have hundreds of those keywords that are leading to URLs being deindexed directly. There is also some law in the US that forbids to show that as a result, so Google and Bing are both having a hard time scraping those pages/articles. Note that this is the late…

Which law is that? Do you have a link to it?

The things I could find on justice.gov and other official websites, maybe there's more in the web archive?

- https://www.justice.gov/archives/opa/pr/google-forfeits-500-...

- https://www.congress.gov/110/plaws/publ425/PLAW-110publ425.p...

- https://www.fda.gov/drugs/prescription-drug-advertising/pres...

edit: Oh it was very likely the Federal Food, Drug and Cosmetic Act that was the legal basis for the crackdown. But that's a very old law from the pre-internet age.

- https://en.wikipedia.org/wiki/Federal_Food,_Drug,_and_Cosmet...

edit 2: Might not have been clear for the younger generation, but there was a huge wave of addicted patients that got treated with Oxycodone (or OxyContin) subscriptions at the time.

I think that might have been the actual cause for the crackdown on those online advertisements, but I might be wrong about that.

Post reply on HN