Earlier quoted context omitted.
Aren't those finite lists? How is a scraper (normal or LLM) supposed to "get stuck" on those?
even though 2^128 uuids is technically "finite", for all intents and purposes is infinite to a scraper.
[dead]
Nepenthes is a tarpit to catch AI web crawlers
111–120 of 290 posts
Re: Nepenthes is a tarpit to catch AI web crawlers
#112There are already “infinite” websites like these on the Internet. Crawlers (both AI and regular search) have a set number of pages they want to crawl per domain. This number is usually determined by the popularity of the domain. Unknown websites will get very few crawls per day whereas popular sites millions. Source: I am the CEO of SerpApi.
This certainly violates the TOS for using Google.
what does this have to do with google?
Re: Nepenthes is a tarpit to catch AI web crawlers
#113Earlier quoted context omitted.
Looking at my logs for all of my sites and this isn’t a global truth. I see multiple ai crawlers hammering away requesting the same pages many, many times. Perplexity and Facebook are basically nonstop.
I just looked at the logs for a site, and I saw PerplexityBot is looking at the robots.txt and ignoring it. They don't provide a list of IPs to verify if it is actually them. Anyway, just for anyone with PerplexityBot in their user agent, they can get increasingly bad responses until the abuse stops.
Perplexity is exceptionally bad because they say they respect the robots.txt but clearly don't. When pressed on it they basically shrug and say too bad not put stuff in public if you don't want it crawled. They got a UA block in cloudflare and seems like that did the trick.
Re: Nepenthes is a tarpit to catch AI web crawlers
#114The article claims that using this will "cause your site to disappear from all search results", but the generated pages don't have the traditional "meta" tags that state the intention to block robots.
Are any search engines respecting that classic meta tag?
Re: Nepenthes is a tarpit to catch AI web crawlers
#115Unless this concept becomes a mass phenomenon with many implementations, isn’t this pretty easy to filter out? And furthermore, since this antagonizes billion-dollar companies that can spin up teams doing nothing but browse Github and HN for software like this to prevent polluting their datalakes, I wonder whether this is a very efficient approach.
I forget which fiction book covered this phenomenon ( Rainbow's End? ), but the moment it becomes the basic default install ( ala adblocker in browsers for people ), it does not matter what the bigger players want to do ; they are not actively fighting against determined and possibly radicalized users.
Re: Nepenthes is a tarpit to catch AI web crawlers
#116There are already “infinite” websites like these on the Internet. Crawlers (both AI and regular search) have a set number of pages they want to crawl per domain. This number is usually determined by the popularity of the domain. Unknown websites will get very few crawls per day whereas popular sites millions. Source: I am the CEO of SerpApi.
Looking at my logs for all of my sites and this isn’t a global truth. I see multiple ai crawlers hammering away requesting the same pages many, many times. Perplexity and Facebook are basically nonstop.
What do you mean by many, many times?
Re: Nepenthes is a tarpit to catch AI web crawlers
#117Earlier quoted context omitted.
I am not surprised that OpenAI is not interested if fixing this.
Their security.txt email address replies and asks you to go on BugCrowd. BugCrowd staff is unwilling (or too incompetent) to run a bash curl command to reproduce the issue, while also refusing to forward it to OpenAI. The support@openai.com waits an hour before answering with ChatGPT answer. Issues raised on GitHub directly towards their engineers were not answered. Also Microsoft CERT & Azure security team do not re…
why try this hard for a private company that doesn't employ you?
Re: Nepenthes is a tarpit to catch AI web crawlers
#118Earlier quoted context omitted.
I just looked at the logs for a site, and I saw PerplexityBot is looking at the robots.txt and ignoring it. They don't provide a list of IPs to verify if it is actually them. Anyway, just for anyone with PerplexityBot in their user agent, they can get increasingly bad responses until the abuse stops.
Perplexity is exceptionally bad because they say they respect the robots.txt but clearly don't. When pressed on it they basically shrug and say too bad not put stuff in public if you don't want it crawled. They got a UA block in cloudflare and seems like that did the trick.
User Agent block just means they'd spoof their user agent.
Re: Nepenthes is a tarpit to catch AI web crawlers
#119Earlier quoted context omitted.
Are you a lawyer?
[flagged]
I broadly agree with what you're trying to get across here, but I don't see why I can't set my own standards for what use of my server is authorized or not.
If I publish content at my domain, I can set up blocklists to refuse access to IP ranges I consider more likely to be malicious than not. Is that not already breaking the social contract you're pointing to wrt serving content public ? picking and choosing which parts of the public will get a response from my server ? (I would also be interested to know if there is actual law vs social contracts around behavior) So why shouldn't I be able enforce expectations on how my server is used? The vigilantism aspect of harming the person breaking the rules is another matter, I'm on the fence.
Consider the standard warning posted to most government sites, which is more or less a "no trespassing sign" [0] informing anyone accessing the system what their expectations should be and what counts as authorized use. I suppose it's not a legally binding contract to say "you agree to these terms by requesting this url" but I'm pretty sure convictions have happened with hackers who did not have a contract with the service provider.
Re: Nepenthes is a tarpit to catch AI web crawlers
#120[flagged]
This is a really bad take, it's not like this server is hacking clients which connect to it. It's providing perfectly valid HTTP responses that just happen to be slow and full of markov gibberish, any harm which comes of that is self inflicted by assuming that websites must provide valuable data as a matter of course. If AI companies want to sue webmasters for that then by all means, they can waste their money and ge…
I guess it's an unpopular take but I don't see why it was flagged. It's a good point of discussion.