Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

github.com

Post reply on HN
Print view View original

151–160 of 288 posts

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#151

Post by hollowturtle »

vivzkestrel wrote: post #142
chii wrote: post #129

Earlier quoted context omitted.

How does your normal users get access to the same contents? Or are you having the user solve an encryption puzzle to view it?

- the frontend has a decryption module that ll show users what they want to see, - the backend has an encryption module. - The bots and crawlers will see the encrypted text - Can someone who peeks deeply inside the client side code decrypt it? YES - Will 99% of the scrapers bother doing this? NO - The key can be anything, it could be a per session key agreed upon between the client and the server, a csrf token, or ev…

Ehm what would stop ai scrapers from using a browser like a normal user would? Google bot already does, it can execute js and can read spa client side generated content, so it proves can be done at scale, and I'm pretty sure some ai scrapers already do

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#152

Post by nottorp »

pjc50 wrote: post #150
ddtaylor wrote: post #101

Earlier quoted context omitted.

Cloudflare has created a bit of grief with regular users getting spammed with "prove your human" requests.

All the solutions are going to have a few false positives, sadly.

Or a lot if you use privacy extensions.

Cloudflare's automatic checks (before you get the captcha) must be pretty close to what ad peddlers do.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#153

Post by nottorp »

temporallobe wrote: post #119

I do know from my experience with test automation that you can absolutely view a site as human eyes would, essentially ignoring all non-visible elements, and in fact Selenium running with Chrome driver does exactly this. Wouldn’t AI scrapers use similar methods?

Probably not, because it costs a lot more CPU cycles.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#155

Post by rhdunn »

asphero wrote: post #89

Interesting approach. The scraper-vs-site-owner arms race is real. On the flip side of this discussion - if you're building a scraper yourself, there are ways to be less annoying: 1. Run locally instead of from cloud servers. Most aggressive blocking targets VPS IPs. A desktop app using the user's home IP looks like normal browsing. 2. Respect rate limits and add delays. Obvious but often ignored. 3. Use RSS feeds wh…

Plus simple caching to not redownload the same file/page multiple times.

It should also be easy to detect a forejo, gitea, or similar hosting site, locate the git URL and clone the repo.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#156

Post by PeterStuer »

n1xis10t wrote: post #15
kstrauser wrote: post #11

I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…

That’s so many scrapers. There must be a ton of companies with very large document collections at this point, and it really sucks that they don’t at least do us the courtesy of indexing them and making them available for keyword search, but instead only do AI. It’s kind of crazy how much scraping goes on and how little search engine development goes on. I guess search engines aren’t fashionable. Reminds me of this ar…

Or some anti-ddos/bot companies using ultra cheap scraping services to annoy you enough to get you into their "free" anti bot protection, so they can charge the few real ai scrapers for access to your site.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#157

Post by cakealert »

nospice wrote: post #132
JohnMakin wrote: post #16

Cloudflare offers bot mitigation for free, and pretty generous WAF rules that makes mitigations like this seem a little overblown to me

I'm on the free tier, but I also watch my logs. The vast majority of the traffic I'm getting are scrapers and vulnerability scanners, a lot of them coming through residential proxies and other "laundered" egress points. I honestly don't think that Cloudflare is on top of the problem at all. They claim to be blocking abuse, but in my experience, most of the badness gets through.

when you combine a residential proxy with a tool like curl-impersonate (there are libraries in Go for this type of fingerprint spoofing now) they dont even show up as scrapers anymore, just users. especially when they adjust timings to mimic humans.

clouflare only blocks the most dumb of bots, there are still a lot of them.

this is why cloudflare will issue javascript challenges to you even when you are using google chrome with a VPN, they are desperate to appear to be doing something. and every VPN is used to crawl as well. a slightly more sophisticated bot passes the cloudflare javascript challenge as well, there really is nothing they can do to win here.

i know some teams that got annoyed with residential proxies (they are usually sold as socks5 but can be buggy and low bandwidth) so they invested into defeating the cloudflare javascript challenge and now crawl using 1000's of VPN endpoints at over 100 Gbit/s.

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#158

Post by voodooEntity »

Funny idea, some days ago i was really annoyed again by the idea that these AI crawlers still ignore all code licenses and train their models against any github repo no matter what so i quickly hammerd down this

-> https://github.com/voodooEntity/ghost_trap

basically a github action that extends your README.md with a "polymorphic" prompt injection. I run some "llm"s against it and most cases they just produced garbage.

Thought about also creating a JS variant that you can add to your website that will (not visible for the user) also inject such prompt injections to stop web crwaling like you described

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#159

Post by PunchyHamster »

anonymous908213 wrote: post #67
kstrauser wrote: post #11

I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic…

As someone on the browsing end, I love Anubis. I've only seen it a couple of times, but it sparks joy. It's rather refreshing compared to Cloudfare, which will usually make me immediately close the page and not bother with whatever content was behind it.

My experience with it is that it somehow took 20 seconds to load (site might've been hn-hugged at the time), only to "protect" some fucking static page instead of just serving that shit in the first place rather than wasting CPU on... whatever it was doing to cause delay

Re: Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)

#160

Post by PunchyHamster »

n1xis10t wrote: post #95
tokioyoyo wrote: post #91

Earlier quoted context omitted.

Large scale scraping tech is not as sophisticated as you'd think. A significant chunk of it is "get as much as possible, categorize and clean up later". Man, I really want the real web of the 2000s back, when things felt "real" more or less... how can we even get there.

If people start making search engines again and there is more competition for Google, I think things would be pretty sweet.

it's few orders of magnitude harder given the amount of SEO spam prevalent, and that just gonna get worse with AI
Post reply on HN