Earlier quoted context omitted.
Real search engines respect robots.txt so you could just tell them not to enter Markov Chain Hell.
I suspect AI crawler would also (quickly learn to) respect it also?
[deleted]
Nepenthes is a tarpit to catch AI web crawlers
31–40 of 290 posts
Re: Nepenthes is a tarpit to catch AI web crawlers
#32Unless this concept becomes a mass phenomenon with many implementations, isn’t this pretty easy to filter out? And furthermore, since this antagonizes billion-dollar companies that can spin up teams doing nothing but browse Github and HN for software like this to prevent polluting their datalakes, I wonder whether this is a very efficient approach.
The idea is that you place this in parallel to the rest of your website routes, that way your entire server might get blacklisted by the bot.
Re: Nepenthes is a tarpit to catch AI web crawlers
#33We had our non-profit website drained out of bandwidth and site closed temporarily (!!) from our hosting deal because of Amazon bot aggressively crawling like ?page=21454 ... etc. Gladly Siteground restored our site without any repercussions as it was not our fault. Added Amazon bot into robots.txt after that one. Don't like how things are right now. Is a tarpit the solution? Or better laws? Would they stop the chine…
For the "good" bots which at least respect robots.txt you can use this list to get ahead of them before they pummel your site.
https://github.com/ai-robots-txt/ai.robots.txt
There's no easy solution for bad bots which ignore robots.txt and spoof their UA though.
Re: Nepenthes is a tarpit to catch AI web crawlers
#34Unless this concept becomes a mass phenomenon with many implementations, isn’t this pretty easy to filter out? And furthermore, since this antagonizes billion-dollar companies that can spin up teams doing nothing but browse Github and HN for software like this to prevent polluting their datalakes, I wonder whether this is a very efficient approach.
If it means it makes your own content safe when you deploy it on a corner of your website: mission accomplished!
>If it means it makes your own content safe
Not really? As mentioned by others, such tarpits are easily mitigated by using a priority queue. For instance, crawlers can prioritize external links over internal links, which means if your blog post makes it to HN, it'll get crawled ahead of the tarpit. If it's discoverable and readable by actual humans, AI bots will be able to scrape it.
Re: Nepenthes is a tarpit to catch AI web crawlers
#35We had our non-profit website drained out of bandwidth and site closed temporarily (!!) from our hosting deal because of Amazon bot aggressively crawling like ?page=21454 ... etc. Gladly Siteground restored our site without any repercussions as it was not our fault. Added Amazon bot into robots.txt after that one. Don't like how things are right now. Is a tarpit the solution? Or better laws? Would they stop the chine…
For the "good" bots which at least respect robots.txt you can use this list to get ahead of them before they pummel your site. https://github.com/ai-robots-txt/ai.robots.txt There's no easy solution for bad bots which ignore robots.txt and spoof their UA though.
Thanks, will look into that!
Re: Nepenthes is a tarpit to catch AI web crawlers
#36Earlier quoted context omitted.
> There are already “infinite” websites like these on the Internet. Cool. And how much of the software driving these websites is FOSS and I can download and run it for my own (popular enough to be crawled more than daily by multiple scrapers) website?
Off the top of my head: https://everyuuid.com/ https://github.com/nolenroyalty/every-uuid
Aren't those finite lists? How is a scraper (normal or LLM) supposed to "get stuck" on those?
Re: Nepenthes is a tarpit to catch AI web crawlers
#37Unless this concept becomes a mass phenomenon with many implementations, isn’t this pretty easy to filter out? And furthermore, since this antagonizes billion-dollar companies that can spin up teams doing nothing but browse Github and HN for software like this to prevent polluting their datalakes, I wonder whether this is a very efficient approach.
Author of a similar tool here[0]. There are a few implementations of this sort of thing that I know of. Mine is different in that the primary purpose is to slightly alter content statically using a Markov generator, mainly to make it useless for content reposters, secondarily to make it useless to LLM crawlers that ignore my robots.txt file[1]. I assume the generated text is bad enough that the LLM crawlers just throw the result out. Other than the extremely poor quality of the text, my tool doesn't leave any fingerprints (like recursive non-sense links.) In any case, it can be run on static sites with no server-side dependencies so long as you have a way to do content redirection based on User-Agent, IP, etc.
My tool does have a second component - linkmaze - which generates a bunch of nonsense text with a Markov generator, and serves infinite links (like Nepthenes does) but I generally only throw incorrigible bots at it (and, at others have noted in-thread, most crawlers already set some kind of limit on how many requests they'll send to a given site, especially a small site.) I do use it for PHP-exploit crawlers as well, though I've seen no evidence those fall into the maze -- I think they mostly just look for some string indicating a successful exploit and move on if whatever they're looking for isn't present.
But, for my use case, I don't really care if someone fingerprints content generated by my tool and avoids it. That's the point: I've set robots.txt to tell these people not to crawl my site.
In addition to Quixotic (my tool) and Napthenes, I know of:
* https://github.com/Fingel/django-llm-poison
* https://codeberg.org/MikeCoats/poison-the-wellms
* https://codeberg.org/timmc/marko/
0 - https://marcusb.org/hacks/quixotic.html
1 - I use the ai.robots.txt user agent list from https://github.com/ai-robots-txt/ai.robots.txt
Re: Nepenthes is a tarpit to catch AI web crawlers
#38Does anyone know if there is anything like Nepenthes but that implements data poisoning attacks like https://arxiv.org/abs/2408.02946
[deleted]
Re: Nepenthes is a tarpit to catch AI web crawlers
#39Does anyone know if there is anything like Nepenthes but that implements data poisoning attacks like https://arxiv.org/abs/2408.02946
I skimmed the paper and the gist seems to be: if you fine-tune a foundation model on bad training data, the resulting model will produce bad outputs. That seems... expected? This makes as much sense as "if you add vulnerable libraries to your app, your app will be vulnerable". I'm not sure how this can turn into an actual attack though.
Re: Nepenthes is a tarpit to catch AI web crawlers
#40Earlier quoted context omitted.
Off the top of my head: https://everyuuid.com/ https://github.com/nolenroyalty/every-uuid
Aren't those finite lists? How is a scraper (normal or LLM) supposed to "get stuck" on those?
even though 2^128 uuids is technically "finite", for all intents and purposes is infinite to a scraper.