I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…
I had to look up what iocaine route is. Here's an article that explains well: https://firesphere.dev/articles/iocaine-the-deadliest-poison...
Creepy Crawlies
541–550 of 707 posts
Re: Creepy Crawlies
#542I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…
Have you considered releasing any of this as a plug for phoenix or similar? I’ve also toyed with this approach and have a similar labyrinth on my website, but it just serves back huge amounts of meaningless text one word at a time.
I do want to add some more gibberish generation to mine. Once a visitor hits the honeypot route it does replace the rest of the site content with nonsense but I think it'd be fun to have it spewing really daft, really long LLM-generated essays on toasters.
Re: Creepy Crawlies
#543Re: Creepy Crawlies
#544Earlier quoted context omitted.
because they do not want to be twitter, reddit, facebook, ...
I'm not suggesting 'login' because facebook/twitter/etc, just as a mechanism to make the bot problem go away. They clearly want it to stop, they tried obvious methods but the AI platforms are circumventing it (deliberately) which is poor form.
Re: Creepy Crawlies
#545Re: Creepy Crawlies
#546Earlier quoted context omitted.
Monero is fine with making people wait, that's literally the whole basis of mining blocks in "crypto currency". It's tuned to the time required. Anubis wants "slow for bots but fast for humans" which is not possible if somebody has a farm of devices that costs them basically zero.
One possible way to do this: rather than simply burning hashes for nothing, Anubis can burn hashes to mine crypto. This way, the bots pay for the infrastructure they consume, at least in part. Combine with heuristics like in the article, so you can crank up the difficulty only for likely bot traffic, and you've got a bot supported site.
Re: Creepy Crawlies
#547Earlier quoted context omitted.
Exactly my observation as well. They devour absolutely everything, no exceptions. No matter how stupid it might be to digest a source code repository via HTTP. They probably don't even recognize what's inside those pages and that there's an easier way to obtain the same result.
The crawlers are not AI. The crawlers are deterministic. They are collecting data to train AIs.
- Respects robots.txt
- Detects if something is likely a git forge and as a result
- Clone the git repository instead of trying to crawl the forge
None of this needs to be happening, and yet it does at an unprecedented scale.