Any other threads on the prevalence and nuisance of scrapers? I didn’t have any idea it was this bad.
Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
31–40 of 288 posts
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#32I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…
*anime jackalgirl Also you mentioned Anubis, so it’s creator will probably read this. Hi Xena!
Correct; my bad!
And hey, Xena! (And thank you very much!)
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#33Earlier quoted context omitted.
Yeah I guess I don’t know if you can sue someone for using your headers, would be interesting to see how that goes.
i think making the case of "you are acting (sending web requests) while knowingly identifying as another legal entity (and criminally/libelously/etc)" shouldn't be toooo hard
Seems like, but there are tons of things that forge request headers all the time, and I don’t think I’ve heard of anyone getting in legal trouble for it. Now I think most of these are scrapers pretending to be browsers, so it might be different I don’t know.
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#34Any other threads on the prevalence and nuisance of scrapers? I didn’t have any idea it was this bad.
Here’s one from yesterday: https://news.ycombinator.com/item?id=46302496#46306025
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#35I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.
How do you define a user, and how do you define online? If the forum considers unique cookies to be a user and creates a new cookie for any new cookie-less request, and if it considers a user to be online for 1 hour after their last request, then actually this may be one scraper making ~6 requests per second. That may be a pain in its own way, but it's far from 23k online bots.
That's still 518.400 requests per day. For static content. And it's a niche forum, so it's not exactly going to have millions of pages.
Either there are indeed hundreds or thousands of AI bots DDoSing the entire internet, or a couple of bots are needlessly hammering it over and over and over again. I'm not sure which option is worse.
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#36Any other threads on the prevalence and nuisance of scrapers? I didn’t have any idea it was this bad.
I've been seeing "we had to take the forum/website offline to deal with scrapers" message on quite a few niche websites now. They are an absolute pest.
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#37I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…
That’s so many scrapers. There must be a ton of companies with very large document collections at this point, and it really sucks that they don’t at least do us the courtesy of indexing them and making them available for keyword search, but instead only do AI. It’s kind of crazy how much scraping goes on and how little search engine development goes on. I guess search engines aren’t fashionable. Reminds me of this ar…
So! Much! Scraping! They were downloading every commit multiple times, and fetching every file as seen at each of those commits, and trying to download archives of all the code, and hitting `/me/my-repo/blame` endpoints as their IP's first-ever request to my server, and other unlikely stuff.
My scraper dudes, it's a git repo. You can fetch the whole freaking thing if you wanna look at it. Of course, that would require work and context-aware processing on their end, and it's easier for them to shift the expense onto my little server and make me pay for their misbehavior.
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#38There is some irony in using an AI generated banner image for this project...
(No, I don't want to defend the poor AI companies. Go for it!)
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#39Earlier quoted context omitted.
For “free”.
Did you put “free” in quotes because you need to have paid for stuff from cloudflare to use the “free” thing? If so, I suppose it’s like those magazines that say ”free cd”.
I thought they were referring to the indirect costs of supporting monopolistic stuff that enshittifies later.
Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
#40There is some irony in using an AI generated banner image for this project... (No, I don't want to defend the poor AI companies. Go for it!)
In the olden days, I used Google an awful lot, but I would still grouse if Google were to drive my server into the ground.