Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

github.com

Post reply on HN
Print view View original

71–80 of 288 posts

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#71

Post by danpalmer »

thethingundone wrote: post #46
danpalmer wrote: post #21

Earlier quoted context omitted.

How do you define a user, and how do you define online? If the forum considers unique cookies to be a user and creates a new cookie for any new cookie-less request, and if it considers a user to be online for 1 hour after their last request, then actually this may be one scraper making ~6 requests per second. That may be a pain in its own way, but it's far from 23k online bots.

AFAIK it keeps a user counted as online for 5 or 15 minutes (I think 5). It’s a Woltlab Burning Board. Edit: it’s 15 minutes.

And what is a "user"?

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#72

Post by reallyhuh »

n1xis10t wrote: post #52
thethingundone wrote: post #49

Earlier quoted context omitted.

The bots are exposing themselves as Google, Bing and Yandex. I can’t verify whether it’s being attributed by IP address or whether the forum trusts their user agent. It could basically be anyone.

Interesting. When it was just normal search engines I didn’t hear of people having this problem, so this either means that there are a bunch of people pretending to be bing google and yandex, or those companies have gotten a lot more aggressive.

What are the proportions for the attributions? Is it equally distributed or lopsided towards one of the three?

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#73

Post by whynotmaybe »

username223 wrote: post #7

The more ways people mess with scrapers, the better -- let a thousand flowers bloom! You as an individual can't compete with VC-funded looters, but there aren't enough of them to defeat a thousand people resisting in different ways.

Should we subtlety poison every forum we encounter with simple yet false statements?

Like put "Water is green, supergreen" in every signature so that when we ask "is water blue" to an llm it might answer "not it's supergreen"?

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#75

Post by n1xis10t »

hollowturtle wrote: post #70
montroser wrote: post #58

This is a cute idea, but I wonder what is the sustainable solution to this emerging fundamental problem: As content publishers, we want our content to be accessible to everyone, and we're even willing to pay for server costs relative to our intended audience -- but a new outsized flood of scrapers was not part of the cost calculation, and that is messing up the plan. It seems all options have major trade-offs. We can…

If the LLMs are the "new Google" one solution would be for them to pay you when scraping your content, so you both have an incentive, you're more willing to be scraped and they'll try to not abuse you because it will cost them at every visit. If your content is valuable and requested on prompts they will scrape you more and so on. I can't see other solutions honestly. For now they decided to go full evil and abuse ev…

This would require new laws though, wouldn’t it?

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#76

Post by n1xis10t »

kstrauser wrote: post #69
anonymous908213 wrote: post #67

Earlier quoted context omitted.

As someone on the browsing end, I love Anubis. I've only seen it a couple of times, but it sparks joy. It's rather refreshing compared to Cloudfare, which will usually make me immediately close the page and not bother with whatever content was behind it.

Same here, really. That's why I started using it. I'd seen it pop up for a moment on a few sites I'd visited, and it was so quirky and completely not disruptive that I didn't mind routing my legit users through it.

So maybe there are more people who like the “anime catgirl” than there are who think it’s weird

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#78

Post by n1xis10t »

efilife wrote: post #68

> Alright so if you run a self-hosted blog, you've probably noticed AI companies scraping it for training data. ... There isn't much you can do about it without cloudflare I'm sorry, what? I can't believe I am reading this on HackerNews. All you have to do is code your own, BASIC captcha-like system. You can just create a page that sets a cookie using JS and check on the server whether it exists. 99.9999% of these sc…

Oops you just leaked your own intellectual property

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#79

Post by ATechGuy »

efilife wrote: post #68

> Alright so if you run a self-hosted blog, you've probably noticed AI companies scraping it for training data. ... There isn't much you can do about it without cloudflare I'm sorry, what? I can't believe I am reading this on HackerNews. All you have to do is code your own, BASIC captcha-like system. You can just create a page that sets a cookie using JS and check on the server whether it exists. 99.9999% of these sc…

From ChatGPT:

This approach can stop very basic scripts, but the claim that “99.9999% of scrapers can’t execute JS or handle cookies” isn’t accurate anymore. Modern scraping tools commonly use headless browsers (Playwright, Puppeteer, Selenium), execute JavaScript, support cookies, and spoof realistic user agents. Any scraper beyond the most trivial will pass a JS-set cookie check without effort. That said, using a lightweight JS challenge can be reasonable as one signal among many, especially for low-value content and when minimizing user friction is a priority. It’s just not a reliable standalone defense. If it’s working for you, that likely means your site isn’t a high-value scraping target — not that the technique is fundamentally robust.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#80

Post by kstrauser »

n1xis10t wrote: post #76
kstrauser wrote: post #69

Earlier quoted context omitted.

Same here, really. That's why I started using it. I'd seen it pop up for a moment on a few sites I'd visited, and it was so quirky and completely not disruptive that I didn't mind routing my legit users through it.

So maybe there are more people who like the “anime catgirl” than there are who think it’s weird

*anime jackalgirl ;-)

Quite possibly. Or, in my case, I think it's more quirky and fun than weird. It's non-zero amounts of weird, sure, but far below my threshold of troublesome. I probably wouldn't put my business behind it. I'm A-OK with using it on personal and hobby projects.

Frankly, anyone so delicate that they freak out at the utterly anodyne imagery is someone I don't want to deal with in my personal time. I can only abide so much pearl clutching when I'm not getting paid for it.

Post reply on HN