Earlier quoted context omitted.
How do you define a user, and how do you define online? If the forum considers unique cookies to be a user and creates a new cookie for any new cookie-less request, and if it considers a user to be online for 1 hour after their last request, then actually this may be one scraper making ~6 requests per second. That may be a pain in its own way, but it's far from 23k online bots.
That's still 518.400 requests per day . For static content. And it's a niche forum, so it's not exactly going to have millions of pages. Either there are indeed hundreds or thousands of AI bots DDoSing the entire internet, or a couple of bots are needlessly hammering it over and over and over again. I'm not sure which option is worse.
Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
41–50 of 288 posts
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#42Earlier quoted context omitted.
That’s so many scrapers. There must be a ton of companies with very large document collections at this point, and it really sucks that they don’t at least do us the courtesy of indexing them and making them available for keyword search, but instead only do AI. It’s kind of crazy how much scraping goes on and how little search engine development goes on. I guess search engines aren’t fashionable. Reminds me of this ar…
So! Much! Scraping! They were downloading every commit multiple times, and fetching every file as seen at each of those commits, and trying to download archives of all the code, and hitting `/me/my-repo/blame` endpoints as their IP's first-ever request to my server, and other unlikely stuff. My scraper dudes, it's a git repo. You can fetch the whole freaking thing if you wanna look at it. Of course, that would requir…
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#43Any other threads on the prevalence and nuisance of scrapers? I didn’t have any idea it was this bad.
I've been seeing "we had to take the forum/website offline to deal with scrapers" message on quite a few niche websites now. They are an absolute pest.
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#44Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#45I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.
Are you sure the counter is not broken?
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#46I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.
How do you define a user, and how do you define online? If the forum considers unique cookies to be a user and creates a new cookie for any new cookie-less request, and if it considers a user to be online for 1 hour after their last request, then actually this may be one scraper making ~6 requests per second. That may be a pain in its own way, but it's far from 23k online bots.
Edit: it’s 15 minutes.
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#47The more ways people mess with scrapers, the better -- let a thousand flowers bloom! You as an individual can't compete with VC-funded looters, but there aren't enough of them to defeat a thousand people resisting in different ways.
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#48Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#49I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.