Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nepenthes is a tarpit to catch AI web crawlers

zadzmo.org

Post reply on HN
Print view View original

81–90 of 290 posts

Re: Nepenthes is a tarpit to catch AI web crawlers

#81

Post by NathanKP »

This looks extremely easy to detect and filter out. For example: https://i.imgur.com/hpMrLFT.png

In short, if the creator of this thinks that it will actually trick AI web crawlers, in reality it would take about 5 mins of time to write a simple check that filters out and bans the site from crawling. With modern LLM workflows its actually fairly simple and cheap to burn just a little bit of GPU time to check if the data you are crawling is decent.

Only a really, really bad crawl bot would fall for this. The funny thing is that in order to make something that an AI crawler bot would actually fall for you'd have to use LLM's to generate realistic enough looking content. Markov chain isn't going to cut it.

Re: Nepenthes is a tarpit to catch AI web crawlers

#82

Post by throw_m239339 »

kerkeslager wrote: post #72

Question: do these bots not respect robots.txt? I haven't added these scrapers to my robots.txt on the sites I work on yet because I haven't seen any problems. I would run something like this on my own websites, but I can't see selling my clients on running this on their websites. The websites I run generally have a honeypot page which is linked in the headers and disallowed to everyone in the robots.txt, and if an I…

> Question: do these bots not respect robots.txt?

No they don't, because there is no potential legal liability for not respecting that file in most countries.

Re: Nepenthes is a tarpit to catch AI web crawlers

#83

Post by jonatron »

kerkeslager wrote: post #72

Question: do these bots not respect robots.txt? I haven't added these scrapers to my robots.txt on the sites I work on yet because I haven't seen any problems. I would run something like this on my own websites, but I can't see selling my clients on running this on their websites. The websites I run generally have a honeypot page which is linked in the headers and disallowed to everyone in the robots.txt, and if an I…

You haven't seen any problems because you created a solution to the problem!

Re: Nepenthes is a tarpit to catch AI web crawlers

#84

Post by bflesch »

michaelbuckbee wrote: post #68
bflesch wrote: post #54

Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week. Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website. The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As C…

What is the https://chatgpt.com/backend-api/attributions endpoint doing (or responsible for when not crushing websites).

When ChatGPT cites web sources in it's output to the user, it will call `backend-api/attributions` with the URL and the API will return what the website is about.

Basically it does HTTP request to fetch HTML `` tag.

They don't check length of supplied `urls[]` array and also don't check if it contains the same URL over and over again (with minor variations).

It's just bad engineering all around.

Re: Nepenthes is a tarpit to catch AI web crawlers

#85

Post by observationist »

jsheard wrote: post #79
observationist wrote: post #69

Earlier quoted context omitted.

[flagged]

> No, put up a loginwall or paywall, authenticate users, and go private. We know for a fact that AI companies don't respect that, if they want data that's behind a paywall then they'll jump through hoops to take it anyway. https://www.theguardian.com/technology/2025/jan/10/mark-zuck... If they don't have to abide by "norms" then we don't have to for their sake. Fuck 'em.

[flagged]

Re: Nepenthes is a tarpit to catch AI web crawlers

#86

Post by palmfacehn »

hartator wrote: post #12

There are already “infinite” websites like these on the Internet. Crawlers (both AI and regular search) have a set number of pages they want to crawl per domain. This number is usually determined by the popularity of the domain. Unknown websites will get very few crawls per day whereas popular sites millions. Source: I am the CEO of SerpApi.

Even a brand new site will get hit heavily by crawlers. Amazonbot, Applebot, LLM bots, scrapers abusing FB's link preview bot, SEO metric bots and more than a few crawlers out of China. The desirable, well behaved crawlers are the only ones who might lose interest.

The typical entry point is a sitemap or RSS feed.

Overall I think the author is misguided in using the tarpit approach. Slow sites get less crawls. I would suggest using easily GZIP'd content and deeply nested tags instead. There are also tricks with XSL, but I doubt many mature crawlers will fall for that one.

Re: Nepenthes is a tarpit to catch AI web crawlers

#87

Post by bflesch »

hassleblad23 wrote: post #74
bflesch wrote: post #54

Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week. Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website. The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As C…

I am not surprised that OpenAI is not interested if fixing this.

Their security.txt email address replies and asks you to go on BugCrowd. BugCrowd staff is unwilling (or too incompetent) to run a bash curl command to reproduce the issue, while also refusing to forward it to OpenAI.

The support@openai.com waits an hour before answering with ChatGPT answer.

Issues raised on GitHub directly towards their engineers were not answered.

Also Microsoft CERT & Azure security team do not reply or care respond to such things (maybe due to lack of demonstrated impact).

Re: Nepenthes is a tarpit to catch AI web crawlers

#89

Post by qwe----3 »

hartator wrote: post #12

There are already “infinite” websites like these on the Internet. Crawlers (both AI and regular search) have a set number of pages they want to crawl per domain. This number is usually determined by the popularity of the domain. Unknown websites will get very few crawls per day whereas popular sites millions. Source: I am the CEO of SerpApi.

This certainly violates the TOS for using Google.

Re: Nepenthes is a tarpit to catch AI web crawlers

#90

Post by blibble »

observationist wrote: post #85
jsheard wrote: post #79

Earlier quoted context omitted.

> No, put up a loginwall or paywall, authenticate users, and go private. We know for a fact that AI companies don't respect that, if they want data that's behind a paywall then they'll jump through hoops to take it anyway. https://www.theguardian.com/technology/2025/jan/10/mark-zuck... If they don't have to abide by "norms" then we don't have to for their sake. Fuck 'em.

[flagged]

this is a very US-ian view of the world

my site is not in the US, I am not a US citizen. US law does not apply to me.

under UK law: robots.txt is an access control mechanism (weak or otherwise)

knowingly bypassing it is likely a criminal offence under the Computer Misuse Act

good luck suing me because you got stuck when you smashed my window and climbed through it

Post reply on HN