Nepenthes is a tarpit to catch AI web crawlers
1–10 of 290 posts
Re: Nepenthes is a tarpit to catch AI web crawlers
#2This keeps generating new pages to keep the crawler occupied.
Looks like this would tarpit any web crawler.
Re: Nepenthes is a tarpit to catch AI web crawlers
#3This keeps generating new pages to keep the crawler occupied. Looks like this would tarpit any web crawler.
It would indeed. Note the warning: "There is not currently a way to differentiate between web crawlers that are indexing sites for search purposes, vs crawlers that are training AI models. ANY SITE THIS SOFTWARE IS APPLIED TO WILL LIKELY DISAPPEAR FROM ALL SEARCH RESULTS."
Re: Nepenthes is a tarpit to catch AI web crawlers
#4This keeps generating new pages to keep the crawler occupied. Looks like this would tarpit any web crawler.
It would indeed. Note the warning: "There is not currently a way to differentiate between web crawlers that are indexing sites for search purposes, vs crawlers that are training AI models. ANY SITE THIS SOFTWARE IS APPLIED TO WILL LIKELY DISAPPEAR FROM ALL SEARCH RESULTS."
It's actually a great idea to spread malware without leaving traces too, it makes content inspection to be very difficult, view-source: to be broken and most of debugging tools, saving to .har, etc.
Re: Nepenthes is a tarpit to catch AI web crawlers
#5Unless this concept becomes a mass phenomenon with many implementations, isn’t this pretty easy to filter out? And furthermore, since this antagonizes billion-dollar companies that can spin up teams doing nothing but browse Github and HN for software like this to prevent polluting their datalakes, I wonder whether this is a very efficient approach.
Re: Nepenthes is a tarpit to catch AI web crawlers
#6Unless this concept becomes a mass phenomenon with many implementations, isn’t this pretty easy to filter out? And furthermore, since this antagonizes billion-dollar companies that can spin up teams doing nothing but browse Github and HN for software like this to prevent polluting their datalakes, I wonder whether this is a very efficient approach.
I am not sure. How would crawlers filter this?
Re: Nepenthes is a tarpit to catch AI web crawlers
#7I have a very vague concept for this, with a different implementation.
Some, uh, sites (forums?) have content that the AI crawlers would like to consume, and, from what I have heard, the crawlers can irresponsibly hammer the traffic of said sites into oblivion.
What if, for the sites which are paywalled, the signup, which invariably comes with a long click-through EULA, had a legal trap within it, forbidding ingestion by AI models on pain of, say, owning ten percent of the company should this be violated. Make sure there is some kind of token payment to get to the content.
Then seed the site with a few instances of hapax legomenon. Trace the crawler back and get the resulting model to vomit back the originating info, as proof.
This should result in either crawlers being more respectful or the end of the hated click-through EULA. We win either way.
Re: Nepenthes is a tarpit to catch AI web crawlers
#8Unless this concept becomes a mass phenomenon with many implementations, isn’t this pretty easy to filter out? And furthermore, since this antagonizes billion-dollar companies that can spin up teams doing nothing but browse Github and HN for software like this to prevent polluting their datalakes, I wonder whether this is a very efficient approach.
I am not sure. How would crawlers filter this?
Check if the response time, the length of the "main text", or other indicators are in the lowest few percentile -> send to the heap for manual review.
Does the inferred "topic" of the domain match the topic of the individual pages? If not -> manual review. And there are many more indicators.
Hire a bunch of student jobbers, have them search github for tarpits, and let them write middleware to detect those.
If you are doing broad crawling, you already need to do this kind of thing anyway.
Re: Nepenthes is a tarpit to catch AI web crawlers
#9Earlier quoted context omitted.
It would indeed. Note the warning: "There is not currently a way to differentiate between web crawlers that are indexing sites for search purposes, vs crawlers that are training AI models. ANY SITE THIS SOFTWARE IS APPLIED TO WILL LIKELY DISAPPEAR FROM ALL SEARCH RESULTS."
It's actually a great idea to spread malware without leaving traces too, it makes content inspection to be very difficult, view-source: to be broken and most of debugging tools, saving to .har, etc.
how is view source broken
Re: Nepenthes is a tarpit to catch AI web crawlers
#10This keeps generating new pages to keep the crawler occupied. Looks like this would tarpit any web crawler.
It would indeed. Note the warning: "There is not currently a way to differentiate between web crawlers that are indexing sites for search purposes, vs crawlers that are training AI models. ANY SITE THIS SOFTWARE IS APPLIED TO WILL LIKELY DISAPPEAR FROM ALL SEARCH RESULTS."
Real search engines respect robots.txt so you could just tell them not to enter Markov Chain Hell.