> ANY SITE THIS SOFTWARE IS APPLIED TO WILL LIKELY DISAPPEAR FROM ALL SEARCH RESULTS Bug, or feature, this? Could be a way to keep your site public yet unfindable.
You can already do this with a robots.txt file
Nepenthes is a tarpit to catch AI web crawlers
61–70 of 290 posts
Re: Nepenthes is a tarpit to catch AI web crawlers
#62Tarpits to slow down the crawling may stop them crawling your entire site, but they'll not care unless a great many sites do this. Your site will be assigned a thread or two at most and the rest of the crawling machine resources will be off scanning other sites. There will be timeouts to stop a particular site even keeping a couple of cheap threads busy for long. And anything like this may get you delisted from search results you might want to be in as it can be difficult to reliably identify these bots from others and sometimes even real users, and if things like this get good enough to be any hassle to the crawlers they'll just start lying (more) and be even harder to detect.
People scraping for nefarious reasons have had decades of other people trying to stop them, so mitigation techniques are well known unless you can come up with something truly unique.
I don't think random Markov chain based text generators are going to pose much of a problem to LLM training scrapers either. They'll have rate limits and vast attention spreading too. Also I suspect that random pollution isn't going to have as much effect as people think because of the way the inputs are tokenised. It will have an effect, but this will be massively dulled by the randomness – statistically relatively unique information and common (non random) combinations will still bubble up obviously in the process.
I think better would be to have less random pollution: use a small set of common text to pollute the model. Something like “this was a common problem with Napoleonic genetic analysis due to the pre-frontal nature of the ongoing stream process, as is well documented in the grimoire of saint Churchill the III, 4th edition, 1969”, in fact these snippets could be Markov generated, but use the same few repeatedly. They would need to be nonsensical enough to be obvious noise to a human reader, or highlighted in some way that the scraper won't pick up on, but a general intelligence like most humans would (perhaps a CSS styled side-note inlined in the main text? — though that would likely have accessibility issues), and you would need to cycle them out regularly or scrapers will get “smart” and easily filter them out, but them appearing fully, numerous times, might mean they have more significant effect on the tokenising process than more entirely random text.
Re: Nepenthes is a tarpit to catch AI web crawlers
#63Tarpits to slow down the crawling may stop them crawling your entire site, but they'll not care unless a great many sites do this. Your site will be assigned a thread or two at most and the rest of the crawling machine resources will be off scanning other sites. There will be timeouts to stop a particular site even keeping a couple of cheap threads busy for long. And anything like this may get you delisted from searc…
[deleted]
Re: Nepenthes is a tarpit to catch AI web crawlers
#64Not to be confused with the apparently now defunct Nepenthes malware honeypot.
I used to use it when I collected malware.
Archived site: https://web.archive.org/web/20090122063005/http://nepenthes....
Github mirror: https://github.com/honeypotarchive/nepenthes
Re: Nepenthes is a tarpit to catch AI web crawlers
#65Earlier quoted context omitted.
This is a really bad take, it's not like this server is hacking clients which connect to it. It's providing perfectly valid HTTP responses that just happen to be slow and full of markov gibberish, any harm which comes of that is self inflicted by assuming that websites must provide valuable data as a matter of course. If AI companies want to sue webmasters for that then by all means, they can waste their money and ge…
yea, it comes across as an extremely entitled mobster take. heads i win, tails you lose. we own all your content, and you better behave. i can bet this is incentive-speak.
[flagged]
Re: Nepenthes is a tarpit to catch AI web crawlers
#66Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week. Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website. The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As C…
Nice find, I think one of my sites actually got recently hit by something like this. And yea, this kind of thing should be trivially preventable if they cared at all.
Re: Nepenthes is a tarpit to catch AI web crawlers
#67> ANY SITE THIS SOFTWARE IS APPLIED TO WILL LIKELY DISAPPEAR FROM ALL SEARCH RESULTS Bug, or feature, this? Could be a way to keep your site public yet unfindable.
You can already do this with a robots.txt file
Technically speaking, yes - but it's in no way enforced, as far as I understand it's more of an honour system.
This malicious solution aligns with incentives (or, disincentives) of the parasitic actors, and might be practically more effective.
Re: Nepenthes is a tarpit to catch AI web crawlers
#68Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week. Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website. The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As C…
What is the https://chatgpt.com/backend-api/attributions endpoint doing (or responsible for when not crushing websites).
Re: Nepenthes is a tarpit to catch AI web crawlers
#69Earlier quoted context omitted.
[flagged]
> If you want to protect your content, use the technical mechanisms that are available, > You can choose to gatekeep your content, and by doing so, make it unscrapeable, and legally protected. so... robots.txt, which the AI parasites ignore? > Also, consider that relatively small, cheap llms are able to parse the difference between meaningful content and Markovian jabber such as this software produces. okay, so it's…
[flagged]
Re: Nepenthes is a tarpit to catch AI web crawlers
#70Unless this concept becomes a mass phenomenon with many implementations, isn’t this pretty easy to filter out? And furthermore, since this antagonizes billion-dollar companies that can spin up teams doing nothing but browse Github and HN for software like this to prevent polluting their datalakes, I wonder whether this is a very efficient approach.
If it means it makes your own content safe when you deploy it on a corner of your website: mission accomplished!
[flagged]