Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

github.com

Post reply on HN
Print view View original

91–100 of 288 posts

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#91

Post by tokioyoyo »

thethingundone wrote: post #19

I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.

Large scale scraping tech is not as sophisticated as you'd think. A significant chunk of it is "get as much as possible, categorize and clean up later". Man, I really want the real web of the 2000s back, when things felt "real" more or less... how can we even get there.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#92

Post by eek2121 »

Disclosure, I've not run a website since my health issues began, however, Cloudflare has an AI firewall, Cloudflare is super cheap (also: unsure if the AI firewall is on the free tier, however I would be surprised if it is not). Ignoring the recent drama about a couple incidents they've had (because this would not matter for a personal blog), why not use this instead?

Just curious. Hoping to be able to work on a website again someday, if I ever regain my health/stamina/etc back.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#93

Post by brigandish »

eek2121 wrote: post #92

Disclosure, I've not run a website since my health issues began, however, Cloudflare has an AI firewall, Cloudflare is super cheap (also: unsure if the AI firewall is on the free tier, however I would be surprised if it is not). Ignoring the recent drama about a couple incidents they've had (because this would not matter for a personal blog), why not use this instead? Just curious. Hoping to be able to work on a webs…

All the best with getting back on your feet.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#94

Post by thethingundone »

tokioyoyo wrote: post #91
thethingundone wrote: post #19

I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.

Large scale scraping tech is not as sophisticated as you'd think. A significant chunk of it is "get as much as possible, categorize and clean up later". Man, I really want the real web of the 2000s back, when things felt "real" more or less... how can we even get there.

I would understand that, but it seems they don’t store the stuff but recollect the same content every hour.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#95

Post by n1xis10t »

tokioyoyo wrote: post #91
thethingundone wrote: post #19

I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.

Large scale scraping tech is not as sophisticated as you'd think. A significant chunk of it is "get as much as possible, categorize and clean up later". Man, I really want the real web of the 2000s back, when things felt "real" more or less... how can we even get there.

If people start making search engines again and there is more competition for Google, I think things would be pretty sweet.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#96

Post by thethingundone »

danpalmer wrote: post #71
thethingundone wrote: post #46

Earlier quoted context omitted.

AFAIK it keeps a user counted as online for 5 or 15 minutes (I think 5). It’s a Woltlab Burning Board. Edit: it’s 15 minutes.

And what is a "user"?

Whatever the forum software Woltlab Burning Board considers a user. If I recall correctly, it tries to build an identifier based on PHP session ids, so most likely simply cookies.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#97

Post by tokioyoyo »

thethingundone wrote: post #94
tokioyoyo wrote: post #91

Earlier quoted context omitted.

Large scale scraping tech is not as sophisticated as you'd think. A significant chunk of it is "get as much as possible, categorize and clean up later". Man, I really want the real web of the 2000s back, when things felt "real" more or less... how can we even get there.

I would understand that, but it seems they don’t store the stuff but recollect the same content every hour.

I'm assuming a quick hash check to see if there's any change? Between scrapers "most up to date data" is fairly valuable nowadays as well.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#98

Post by tokioyoyo »

n1xis10t wrote: post #95
tokioyoyo wrote: post #91

Earlier quoted context omitted.

Large scale scraping tech is not as sophisticated as you'd think. A significant chunk of it is "get as much as possible, categorize and clean up later". Man, I really want the real web of the 2000s back, when things felt "real" more or less... how can we even get there.

If people start making search engines again and there is more competition for Google, I think things would be pretty sweet.

Because of the financial incentives, it would still end up with people doing things to drive traffic to their website though, no? Maybe because the web was smaller, and people looked at it as means "to explore curiosity" in the olden days it kinda worked differently... maybe I just got old, but I don't want to believe that.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#99

Post by xena »

n1xis10t wrote: post #13
kstrauser wrote: post #11

I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…

*anime jackalgirl Also you mentioned Anubis, so it’s creator will probably read this. Hi Xena!

Ohai! I'm working on dataset poisoning. The early prototype generates vapid LinkedIn posts but future versions will be fully pluggable with WebAssembly.
Post reply on HN