Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

github.com

Post reply on HN
Print view View original

241–250 of 288 posts

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#241

Post by tmnvix »

tokioyoyo wrote: post #91
thethingundone wrote: post #19

I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.

Large scale scraping tech is not as sophisticated as you'd think. A significant chunk of it is "get as much as possible, categorize and clean up later". Man, I really want the real web of the 2000s back, when things felt "real" more or less... how can we even get there.

A curated web directory. Kind of like Yahoo had. The internet according to the dewey system with pages somehow rated for quality by actual humans (maybe something to learn from Wikipedia's approach here?)

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#242

Post by socialcommenter »

n1xis10t wrote: post #235
socialcommenter wrote: post #197

Earlier quoted context omitted.

Without wanting to upset anyone - what makes you interested in sharing tips for team scraper? (Overgeneralising a bit) site owners are mostly cting for public benefit whereas scrapers act for their own benefit/for private interests. I imagine most people would land on team site-owner, if they were asked. I certainly would. P.S. is the best way to scrape fairly just to respect robots.txt?

I think "scraper vs siteowners" is a false dichotomy. Scrapers will always need to exist as long as we want search engines and archival services. We will need small versions of these services to keep popping up every now and then to keep the big guys on their toes, and the smaller guys need advice for scraping politely.

That's fair - though are we in an isolated bout of "every now and then" or has AI created a new normal of abuse (e.g. of robots.txt)? Hopefully we're at a local maximum and some of the scrapers perpetrating harmful behaviours will soon pull their heads in.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#243

Post by kelvinjps10 »

mrweasel wrote: post #176
n1xis10t wrote: post #15

Earlier quoted context omitted.

That’s so many scrapers. There must be a ton of companies with very large document collections at this point, and it really sucks that they don’t at least do us the courtesy of indexing them and making them available for keyword search, but instead only do AI. It’s kind of crazy how much scraping goes on and how little search engine development goes on. I guess search engines aren’t fashionable. Reminds me of this ar…

> There must be a ton of companies with very large document collections at this point See, I don't think there is, I don't think they want that expense. It's basically the Linus Torvalds philosophy of data storage, if it's on the Internet, I don't need a backup. While I have absolutely no proof of this, I'd guess that many AI companies just crawl the Internet constantly, never saving any of the data. We're seeing som…

Where did Linus Torvalds expressed this philosophy I have never seen it

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#244

Post by amypetrik8 »

kstrauser wrote: post #11

I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…

>I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that!

An even more insane idea -- minding the idea here is porn is radioactive to AI data training scrapers -- is there is something the powers that be view as far more disruptive and against community guidelineish than porn. And that would be wrongthink. The narratives. The historic narratives. The woke ideology. Anything related to an academic department whose field is studies. Alls you need to do is plop in a little diatribe staunchly opposing any such enforced views and that AI bot will shoot away from your website and lightspeed

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#245

Post by nospice »

oidar wrote: post #223
cakealert wrote: post #157

Earlier quoted context omitted.

when you combine a residential proxy with a tool like curl-impersonate (there are libraries in Go for this type of fingerprint spoofing now) they dont even show up as scrapers anymore, just users. especially when they adjust timings to mimic humans. clouflare only blocks the most dumb of bots, there are still a lot of them. this is why cloudflare will issue javascript challenges to you even when you are using google…

Is "residential proxy" another name for an hacked/owned computer that the bots have access to? Or are there legitimate services that sell access to residential IPs?

People legitimately sell egress. It's "free" money. But of course, if you have a botnet, you can sell that through the same channels, no one is looking too closely.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#246

Post by 6031769 »

ATechGuy wrote: post #79
efilife wrote: post #68

> Alright so if you run a self-hosted blog, you've probably noticed AI companies scraping it for training data. ... There isn't much you can do about it without cloudflare I'm sorry, what? I can't believe I am reading this on HackerNews. All you have to do is code your own, BASIC captcha-like system. You can just create a page that sets a cookie using JS and check on the server whether it exists. 99.9999% of these sc…

From ChatGPT: This approach can stop very basic scripts, but the claim that “99.9999% of scrapers can’t execute JS or handle cookies” isn’t accurate anymore. Modern scraping tools commonly use headless browsers (Playwright, Puppeteer, Selenium), execute JavaScript, support cookies, and spoof realistic user agents. Any scraper beyond the most trivial will pass a JS-set cookie check without effort. That said, using a l…

So an LLM says that a technique used to foil LLM scrapers is ineffective against LLM scrapers.

It's almost as if it might have an ulterior motive in saying so.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#247

Post by [deleted] »

icepush wrote: post #215
bytehowl wrote: post #164

Let's imagine I have a blog and put something along these lines somewhere on every page: "This content is provided free of charge for humans to experience. It may also be automatically accessed for search indexing and archival purposes. For licensing information for other uses, contact the author." If I then get hit by a rude AI scraper, what chances would I have to sue the hell out of them in EU courts for copyright…

If the scraper is based (Or has meaningful assets) in the EU, then your chances are good. If they do not, then the lawsuit would be meaningless.

[deleted]

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#248

Post by andersmurphy »

efilife wrote: post #83
ATechGuy wrote: post #79

Earlier quoted context omitted.

From ChatGPT: This approach can stop very basic scripts, but the claim that “99.9999% of scrapers can’t execute JS or handle cookies” isn’t accurate anymore. Modern scraping tools commonly use headless browsers (Playwright, Puppeteer, Selenium), execute JavaScript, support cookies, and spoof realistic user agents. Any scraper beyond the most trivial will pass a JS-set cookie check without effort. That said, using a l…

From someone who actually does this stuff: The claim is very accurate. Maybe not for the biggest websites, but very accurate for a self-hosted blog. You are not that important to waste compute power to set up a whole ass headless browser to scrape your page. Why am I even arguing with ChatGPT?

Yup another trick is to only serve br compressed resources and serve nothing to clients that don't support brotli. A lot of http clients don't support brotli out of the box.

I take it further and only stream content to clients that have a cookie, support js and br. Otherwise all you get is a minimal static pre br compressed shim. Seems to work well enough.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#249

Post by integralid »

dirkc wrote: post #219
hollowturtle wrote: post #151

Earlier quoted context omitted.

Ehm what would stop ai scrapers from using a browser like a normal user would? Google bot already does, it can execute js and can read spa client side generated content, so it proves can be done at scale, and I'm pretty sure some ai scrapers already do

If you decrypt the content on the client side using an expensive decryption algorithm the scraper needs to spend the computing resource to decrypt.

Every visitor too. Mobile users are going to love this.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#250

Post by integralid »

vivzkestrel wrote: post #143
chii wrote: post #129

Earlier quoted context omitted.

How does your normal users get access to the same contents? Or are you having the user solve an encryption puzzle to view it?

for example this is what my backend renders on the static html page {"z":"gxCit6xEQf0N9IIoG909xfSxypRX7j0BLlXnd5IgWrrzEWzBUxDiS4o4AlIkNYOyuzkY8w4IVoEgUmW02jj84BxhMrNPetK8n6nIn2ORLKQPfIVTS48nGQ1PldtdlpiUNYUm04N+WrMBGGceKYnQoORQO3XbFOVzboFYOWbdMhLdMS2N26YtCUYHwy7jw1AwlXS0Nm1SClb0U1qk2KnDB6s9bcMmpstaOY2RkmGbQ4KMuKHaGzByVzeIPHrtXtNjLnj68cgyLALyO3E5ncqspyjbnuZtfusn2Y49Nu3LVZDDk/JojC6x6GlLZKFEDoiunyfhnqd0SRsDvynKNpFObi3uu+…

Well, sure, and the AI scrappers and Google are using your frontend to render your website.
Post reply on HN