Aggressive AI scrapers are making it kinda suck to run wikis
1–5 of 5 posts
Re: Aggressive AI scrapers are making it kinda suck to run wikis
#2DSBL of various sorts work (mostly) for emails.
Site owners could absolutely run a distributed realitime IP/metadata reputation system.
Re: Aggressive AI scrapers are making it kinda suck to run wikis
#3DSBL of various sorts work (mostly) for emails. Site owners could absolutely run a distributed realitime IP/metadata reputation system.
Malicious actors also buy 'aged' accounts, so you'd need to continually monitor this reputation? Seems a hard challenge to solve for individual site owners.
Re: Aggressive AI scrapers are making it kinda suck to run wikis
#4DSBL of various sorts work (mostly) for emails. Site owners could absolutely run a distributed realitime IP/metadata reputation system.
Malicious actors also buy 'aged' accounts, so you'd need to continually monitor this reputation? Seems a hard challenge to solve for individual site owners.
It's easy and effective, aged accounts buying is rare and cumbersome, when the rest are filtered out, the "buyers" stick out like sore thumbs... then wackamole is fun.
Re: Aggressive AI scrapers are making it kinda suck to run wikis
#5Requiring readers to log in to view content doesn't really work. Some will register so that they can use the account to crawl. Some won't even care if they are requesting hundreds of URLs a minute. Sure, that sets off alarms, but it's a game of whack-a-mole again.