Live data from Hacker News

Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

github.com

41–50 of 288 posts

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#41
post #35

Earlier quoted context omitted.

How do you define a user, and how do you define online? If the forum considers unique cookies to be a user and creates a new cookie for any new cookie-less request, and if it considers a user to be online for 1 hour after their last request, then actually this may be one scraper making ~6 requests per second. That may be a pain in its own way, but it's far from 23k online bots.

That's still 518.400 requests per day . For static content. And it's a niche forum, so it's not exactly going to have millions of pages. Either there are indeed hundreds or thousands of AI bots DDoSing the entire internet, or a couple of bots are needlessly hammering it over and over and over again. I'm not sure which option is worse.

Imagine if all this scraping was going into a search engine with a massive index, or a bunch of smaller search engines that a meta-search engine could be made for. This’d be a lot more cool in that case

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#42

Earlier quoted context omitted.

That’s so many scrapers. There must be a ton of companies with very large document collections at this point, and it really sucks that they don’t at least do us the courtesy of indexing them and making them available for keyword search, but instead only do AI. It’s kind of crazy how much scraping goes on and how little search engine development goes on. I guess search engines aren’t fashionable. Reminds me of this ar…

So! Much! Scraping! They were downloading every commit multiple times, and fetching every file as seen at each of those commits, and trying to download archives of all the code, and hitting `/me/my-repo/blame` endpoints as their IP's first-ever request to my server, and other unlikely stuff. My scraper dudes, it's a git repo. You can fetch the whole freaking thing if you wanna look at it. Of course, that would requir…

Crazy

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#43
post #36
post #31

Any other threads on the prevalence and nuisance of scrapers? I didn’t have any idea it was this bad.

I've been seeing "we had to take the forum/website offline to deal with scrapers" message on quite a few niche websites now. They are an absolute pest.

Really? I haven’t started to see that yet. Weird

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#44

Earlier quoted context omitted.

For “free”.

Did you put “free” in quotes because you need to have paid for stuff from cloudflare to use the “free” thing? If so, I suppose it’s like those magazines that say ”free cd”.

You don't though.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#45

I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.

Are you sure the counter is not broken?

Yes, it’s running on a Woltlab Burning Board since forever.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#46

I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.

How do you define a user, and how do you define online? If the forum considers unique cookies to be a user and creates a new cookie for any new cookie-less request, and if it considers a user to be online for 1 hour after their last request, then actually this may be one scraper making ~6 requests per second. That may be a pain in its own way, but it's far from 23k online bots.

AFAIK it keeps a user counted as online for 5 or 15 minutes (I think 5). It’s a Woltlab Burning Board.

Edit: it’s 15 minutes.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#47

The more ways people mess with scrapers, the better -- let a thousand flowers bloom! You as an individual can't compete with VC-funded looters, but there aren't enough of them to defeat a thousand people resisting in different ways.

We need to find more ways to poison their data.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#48
post #38

There is some irony in using an AI generated banner image for this project... (No, I don't want to defend the poor AI companies. Go for it!)

In the olden days, I used Google an awful lot, but I would still grouse if Google were to drive my server into the ground.

Fair point

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#49

I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.

The bots are exposing themselves as Google, Bing and Yandex. I can’t verify whether it’s being attributed by IP address or whether the forum trusts their user agent. It could basically be anyone.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#50

Earlier quoted context omitted.

Did you put “free” in quotes because you need to have paid for stuff from cloudflare to use the “free” thing? If so, I suppose it’s like those magazines that say ”free cd”.

You don't though.

Good to know thanks
Post reply on HN