Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

github.com

Post reply on HN
Print view View original

51–60 of 288 posts

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#51

Post by sethops1 »

thethingundone wrote: post #19

I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.

I have a site with a complete and accurate sitemap.xml describing when its ~6k pages are last updated (on average, maybe weekly or monthly). What do the bots do? They scrape every page continuously 24/7, because of course they do. The amount of waste going into this AI craze is just obscene. It's not even good content.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#52

Post by n1xis10t »

thethingundone wrote: post #49
thethingundone wrote: post #19

I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.

The bots are exposing themselves as Google, Bing and Yandex. I can’t verify whether it’s being attributed by IP address or whether the forum trusts their user agent. It could basically be anyone.

Interesting. When it was just normal search engines I didn’t hear of people having this problem, so this either means that there are a bunch of people pretending to be bing google and yandex, or those companies have gotten a lot more aggressive.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#53

Post by n1xis10t »

sethops1 wrote: post #51
thethingundone wrote: post #19

I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.

I have a site with a complete and accurate sitemap.xml describing when its ~6k pages are last updated (on average, maybe weekly or monthly). What do the bots do? They scrape every page continuously 24/7, because of course they do. The amount of waste going into this AI craze is just obscene. It's not even good content.

It would be interesting if someone made a map that depicts the locations of the ip addresses that are sending so many requests, over the course of a day maybe.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#55

Post by ziml77 »

n1xis10t wrote: post #13
kstrauser wrote: post #11

I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…

*anime jackalgirl Also you mentioned Anubis, so it’s creator will probably read this. Hi Xena!

I checked Xe's profile when I hadn't seen them post here for a while. According to that, they're not really using HN anymore.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#56

Post by n1xis10t »

ziml77 wrote: post #55
n1xis10t wrote: post #13

Earlier quoted context omitted.

*anime jackalgirl Also you mentioned Anubis, so it’s creator will probably read this. Hi Xena!

I checked Xe's profile when I hadn't seen them post here for a while. According to that, they're not really using HN anymore.

See this thread from yesterday or so: https://news.ycombinator.com/item?id=46302496#46306025

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#57

Post by amarant »

owl57 wrote: post #26

> scrapers can ingest them and say "nope we won't scrape there again in the future" Do all the AI scrapers actually do that?

Not all, stuff like unstable diffusion exists.

But a good many, perhaps even most(?), certainly do!

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#58

Post by montroser »

This is a cute idea, but I wonder what is the sustainable solution to this emerging fundamental problem: As content publishers, we want our content to be accessible to everyone, and we're even willing to pay for server costs relative to our intended audience -- but a new outsized flood of scrapers was not part of the cost calculation, and that is messing up the plan.

It seems all options have major trade-offs. We can host on big social media and lose all that control and independence. We can pay for outsized infrastructure just to feed the scrapers, but the cost may actually be prohibitive, and seems such a waste to begin with. We can move as much as possible SSG and put it all behind cloudflare, but this comes with vendor lock in and just isn't architecturally feasible in many applications. We can do real "verified identities" for bots, and just let through the ones we know and like, but this only perpetuates corporate control and makes healthy upstart competition (like Kagi) much more difficult.

So, what are we to do?

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#59

Post by n1xis10t »

montroser wrote: post #58

This is a cute idea, but I wonder what is the sustainable solution to this emerging fundamental problem: As content publishers, we want our content to be accessible to everyone, and we're even willing to pay for server costs relative to our intended audience -- but a new outsized flood of scrapers was not part of the cost calculation, and that is messing up the plan. It seems all options have major trade-offs. We can…

At this point it seems like the problem isn’t internet bandwidth, but just expensive for a server to handle all the requests because it has to process them. Does that seem correct?
Post reply on HN