Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

github.com

Post reply on HN
Print view View original

131–140 of 288 posts

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#131

Post by thisislife2 »

sethops1 wrote: post #51
thethingundone wrote: post #19

I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.

I have a site with a complete and accurate sitemap.xml describing when its ~6k pages are last updated (on average, maybe weekly or monthly). What do the bots do? They scrape every page continuously 24/7, because of course they do. The amount of waste going into this AI craze is just obscene. It's not even good content.

If you are in the US, have you considered suing them for robot.txt / copyright violation? AI companies are currently flush with cash from VCs and there may be a few big law firms willing to fight a law suit against them on your behalf. AI companies have already lost some copyright cases.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#132

Post by nospice »

JohnMakin wrote: post #16

Cloudflare offers bot mitigation for free, and pretty generous WAF rules that makes mitigations like this seem a little overblown to me

I'm on the free tier, but I also watch my logs. The vast majority of the traffic I'm getting are scrapers and vulnerability scanners, a lot of them coming through residential proxies and other "laundered" egress points.

I honestly don't think that Cloudflare is on top of the problem at all. They claim to be blocking abuse, but in my experience, most of the badness gets through.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#133

Post by tommica »

xena wrote: post #99
n1xis10t wrote: post #13

Earlier quoted context omitted.

*anime jackalgirl Also you mentioned Anubis, so it’s creator will probably read this. Hi Xena!

Ohai! I'm working on dataset poisoning. The early prototype generates vapid LinkedIn posts but future versions will be fully pluggable with WebAssembly.

Hi Xena! Your blog is amazing! Didn't realize you're working on Anubis - it's a really nice tool for the internet! Reminds me a bit of the ye' olde internet for some reason.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#134

Post by idiotsecant »

tokioyoyo wrote: post #91
thethingundone wrote: post #19

I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.

Large scale scraping tech is not as sophisticated as you'd think. A significant chunk of it is "get as much as possible, categorize and clean up later". Man, I really want the real web of the 2000s back, when things felt "real" more or less... how can we even get there.

Have you ever listened to the 'high water mark' monologue from fear and loathing? It's pretty much just that. It was a unique time and it was neat that we got to see it, but it can't possibly happen again.

https://www.youtube.com/watch?v=vUgs2O7Okqc

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#135

Post by onion2k »

So fuzzycanary also checks user agents and won't show the links to legitimate search engines, so Google and Bing won't see them.

Unscrupulous AI scrapers will not be using a genuine UA string. They'll be using Google. You'll need to do reverse DNS check instead - https://developers.google.com/crawling/docs/crawlers-fetcher...

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#136

Post by bakugo »

onion2k wrote: post #135

So fuzzycanary also checks user agents and won't show the links to legitimate search engines, so Google and Bing won't see them. Unscrupulous AI scrapers will not be using a genuine UA string. They'll be using Google. You'll need to do reverse DNS check instead - https://developers.google.com/crawling/docs/crawlers-fetcher...

Most AI scrapers use normal browser user agents (usually random outdated Chrome versions, from my experience). They generally don't fake the UAs of legitimate bots like Googlebot, because Googlebot requests coming from non-Google IP ranges would be way too easy to block.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#137

Post by gettingoverit »

xena wrote: post #99
n1xis10t wrote: post #13

Earlier quoted context omitted.

*anime jackalgirl Also you mentioned Anubis, so it’s creator will probably read this. Hi Xena!

Ohai! I'm working on dataset poisoning. The early prototype generates vapid LinkedIn posts but future versions will be fully pluggable with WebAssembly.

You've made one of the best solutions, that matched what I thought of implementing myself, and at the time it was most needed. I think a couple of "thank you" are sorely missing in this comment section.

Thank you!

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#138

Post by yupyupyups »

username223 wrote: post #104
yupyupyups wrote: post #47

Earlier quoted context omitted.

We need to find more ways to poison their data.

> Wee knead two fine-d Moore Waze too Poisson there date... uh. Yes. Revel in your creativity mocking and blocking the slop machines. The "remote refactor" command, "rm -rf", is the best way to reduce the cyclomatic complexity of a local codebase.

Excellent advice! I tried it out and it helped. Thank you

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#139

Post by cyphar »

phyzome wrote: post #85
ATechGuy wrote: post #79

Earlier quoted context omitted.

From ChatGPT: This approach can stop very basic scripts, but the claim that “99.9999% of scrapers can’t execute JS or handle cookies” isn’t accurate anymore. Modern scraping tools commonly use headless browsers (Playwright, Puppeteer, Selenium), execute JavaScript, support cookies, and spoof realistic user agents. Any scraper beyond the most trivial will pass a JS-set cookie check without effort. That said, using a l…

There should be a new rule on HN: No posts that just go "I asked an LLM and it said..." You're not adding anything to the conversation.

Yeah, I really have to wonder what the thought process is behind leaving such a comment. When people first started doing it I wondered if it was some kind of guerrilla outrage marketing campaign.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#140

Post by yupyupyups »

n1xis10t wrote: post #106
username223 wrote: post #104

Earlier quoted context omitted.

> Wee knead two fine-d Moore Waze too Poisson there date... uh. Yes. Revel in your creativity mocking and blocking the slop machines. The "remote refactor" command, "rm -rf", is the best way to reduce the cyclomatic complexity of a local codebase.

Indeed, complexity (both cyclomatic and post-frontal) must be reduced such that the two spurving bearings make a direct line with the panametric fan. For more details consult this instructional video: https://youtu.be/RXJKdh1KZ0w

Very educational
Post reply on HN