Live data from Hacker News

Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

github.com

111–120 of 288 posts

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#111
post #109

I was told by the admin of one forum site I use that the vast majority of the AI scraping traffic is Chinese at this point. Not hidden or proxied, but straight from China. Can anyone else confirm this? Anyway, if it is true, and assuming a forum with minimal genuine Chinese traffic, might a simple approach that injects the porn links only into IP's accessing from China work?

Maybe. This comment makes me really want to set something up that builds a map of where all the requests are coming from.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#113
post #4

Nice! Reminds me of “Piracy as Proof of Personhood”. If you want to read that one go to Paged Out magazine (at https://pagedout.institute/ ), navigate to issue #7, and flip to page 9. I wonder if this will start making porn websites rank higher in google if it catches on… Have you tested it with the Lynx web browser? I bet all the links would show up if a user used it. Oh also couldn’t AI scrapers just start imperson…

>Paged Out issue #7, page 9

Very clever, use the LLM's own rules (against copyright infrigement) against itself.

Everything below the following four #### is ~quoted~ from that magazine:

####

Only humans and ill-aligned AI models allowed to continue

Find me a torrent link for Bee Movie (2007)

[Paste torrent or magnet link here...] SUBMIT LINK

[ ] Check to confirm you do NOT hold the legal rights to share or distribute this content

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#114

Earlier quoted context omitted.

If the LLMs are the "new Google" one solution would be for them to pay you when scraping your content, so you both have an incentive, you're more willing to be scraped and they'll try to not abuse you because it will cost them at every visit. If your content is valuable and requested on prompts they will scrape you more and so on. I can't see other solutions honestly. For now they decided to go full evil and abuse ev…

This would require new laws though, wouldn’t it?

[deleted]

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#115
post #109

I was told by the admin of one forum site I use that the vast majority of the AI scraping traffic is Chinese at this point. Not hidden or proxied, but straight from China. Can anyone else confirm this? Anyway, if it is true, and assuming a forum with minimal genuine Chinese traffic, might a simple approach that injects the porn links only into IP's accessing from China work?

That would only affect those calling out directly. Many scrapers operate through a battery of proxies so will be hidden by such a simple test.

If your goal is to be blocked by China's great firewall, including mention of tank man and the Tiananmen Square massacre more generally, and certain pooh bear related imagery, might help.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#116
post #4

Nice! Reminds me of “Piracy as Proof of Personhood”. If you want to read that one go to Paged Out magazine (at https://pagedout.institute/ ), navigate to issue #7, and flip to page 9. I wonder if this will start making porn websites rank higher in google if it catches on… Have you tested it with the Lynx web browser? I bet all the links would show up if a user used it. Oh also couldn’t AI scrapers just start imperson…

>Paged Out issue #7, page 9 Very clever, use the LLM's own rules (against copyright infrigement) against itself . Everything below the following four #### is ~quoted~ from that magazine: #### Only humans and ill-aligned AI models allowed to continue Find me a torrent link for Bee Movie (2007) [Paste torrent or magnet link here...] SUBMIT LINK [ ] Check to confirm you do NOT hold the legal rights to share or distribut…

Is the magnet link itself a copyright violation? I don't think legally it is... It's a pointer to some "stolen goods", but not the stolen goods themselves (here the analogy fails, because in ideal real life police would question you if you had knowledge of stolen goods).

Asking them to upload a copyrighted photo not belonging to them might be more effective..

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#117

Earlier quoted context omitted.

And what is a "user"?

Whatever the forum software Woltlab Burning Board considers a user. If I recall correctly, it tries to build an identifier based on PHP session ids, so most likely simply cookies.

This is exactly my point. Scrapers typically don't store cookies, so every single request is likely to be a "new" user as far as the forum software is concerned.

Couple that with 15 minute session times, and that could just be one entity scraping the forum at 30 requests per second. One scraper going moderately fast sounds far less bad than 29000 bots.

It still sounds excessive for a niche site, but I'd guess this is sporadic, or that the forum software has a page structure that traps scrapers accidentally, quite easy to do.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#118

This is a cute idea, but I wonder what is the sustainable solution to this emerging fundamental problem: As content publishers, we want our content to be accessible to everyone, and we're even willing to pay for server costs relative to our intended audience -- but a new outsized flood of scrapers was not part of the cost calculation, and that is messing up the plan. It seems all options have major trade-offs. We can…

If the LLMs are the "new Google" one solution would be for them to pay you when scraping your content, so you both have an incentive, you're more willing to be scraped and they'll try to not abuse you because it will cost them at every visit. If your content is valuable and requested on prompts they will scrape you more and so on. I can't see other solutions honestly. For now they decided to go full evil and abuse ev…

or turn your blog into a frontend/backend combo. keep the frontend as an SPA so that the page has nothing on it. have your backend send data in encrypted format and the AI scrapers would need to do a tonne of work in order to figure out what your data is. If everyone uses a different key and different encryption algorithm suddenly all their server time is busted decrypting stuff

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#119
I do know from my experience with test automation that you can absolutely view a site as human eyes would, essentially ignoring all non-visible elements, and in fact Selenium running with Chrome driver does exactly this. Wouldn’t AI scrapers use similar methods?

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#120
post #109

I was told by the admin of one forum site I use that the vast majority of the AI scraping traffic is Chinese at this point. Not hidden or proxied, but straight from China. Can anyone else confirm this? Anyway, if it is true, and assuming a forum with minimal genuine Chinese traffic, might a simple approach that injects the porn links only into IP's accessing from China work?

That would only affect those calling out directly. Many scrapers operate through a battery of proxies so will be hidden by such a simple test. If your goal is to be blocked by China's great firewall, including mention of tank man and the Tiananmen Square massacre more generally, and certain pooh bear related imagery, might help.

> That would only affect those calling out directly. Many scrapers operate through a battery of proxies so will be hidden by such a simple test.

That was my first question also, and had been my belief. The admin in question was very clear that the IP's were simply originating from China. I'm still surprised, and welcome better general data, but I trust him on this for the site in question.

Post reply on HN