Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nepenthes is a tarpit to catch AI web crawlers

zadzmo.org

Post reply on HN
Print view View original

71–80 of 290 posts

Re: Nepenthes is a tarpit to catch AI web crawlers

#71

Post by dylan604 »

captainmuon wrote: post #8
grajaganDev wrote: post #6

Earlier quoted context omitted.

I am not sure. How would crawlers filter this?

Check if the response time, the length of the "main text", or other indicators are in the lowest few percentile -> send to the heap for manual review. Does the inferred "topic" of the domain match the topic of the individual pages? If not -> manual review. And there are many more indicators. Hire a bunch of student jobbers, have them search github for tarpits, and let them write middleware to detect those. If you are…

> Hire a bunch of student jobbers,

Do people still do this, or do they just off shore the task?

Re: Nepenthes is a tarpit to catch AI web crawlers

#72

Post by kerkeslager »

Question: do these bots not respect robots.txt?

I haven't added these scrapers to my robots.txt on the sites I work on yet because I haven't seen any problems. I would run something like this on my own websites, but I can't see selling my clients on running this on their websites.

The websites I run generally have a honeypot page which is linked in the headers and disallowed to everyone in the robots.txt, and if an IP visits that page, they get added to a blocklist which simply drops their connections without response for 24 hours.

Re: Nepenthes is a tarpit to catch AI web crawlers

#74

Post by hassleblad23 »

bflesch wrote: post #54

Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week. Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website. The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As C…

I am not surprised that OpenAI is not interested if fixing this.

Re: Nepenthes is a tarpit to catch AI web crawlers

#75

Post by soupfordummies »

bflesch wrote: post #54

Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week. Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website. The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As C…

Try it and let us know :)

Re: Nepenthes is a tarpit to catch AI web crawlers

#76

Post by dawnerd »

hartator wrote: post #12

There are already “infinite” websites like these on the Internet. Crawlers (both AI and regular search) have a set number of pages they want to crawl per domain. This number is usually determined by the popularity of the domain. Unknown websites will get very few crawls per day whereas popular sites millions. Source: I am the CEO of SerpApi.

Looking at my logs for all of my sites and this isn’t a global truth. I see multiple ai crawlers hammering away requesting the same pages many, many times. Perplexity and Facebook are basically nonstop.

Re: Nepenthes is a tarpit to catch AI web crawlers

#77

Post by WD-42 »

quchen wrote: post #5

Unless this concept becomes a mass phenomenon with many implementations, isn’t this pretty easy to filter out? And furthermore, since this antagonizes billion-dollar companies that can spin up teams doing nothing but browse Github and HN for software like this to prevent polluting their datalakes, I wonder whether this is a very efficient approach.

Does it need to be efficient if it’s easy? I wrote a similar tool except it’s not a performance tarpit. The goal is to slightly modify otherwise organic content so that it is wrong, but only for AI bots. If they catch on and stop crawling the site, nothing is lost. https://github.com/Fingel/django-llm-poison

Re: Nepenthes is a tarpit to catch AI web crawlers

#78

Post by marc »

OpenAI doesn’t take security seriously.

I reported a vulnerability to them that allowed you to get IP addresses of their paying customers.

OpenAI responded “Not applicable” indicating they don’t think it was a serious issue.

The PoC was very easy to understand and simple to replicate.

Edit: I guess I might as well disclose it here since they don’t consider it an issue. They were/are(?) hot linking logo images of third-party plugins. When you open their plugin store it loads a couple dozen of them instantly. This allows those plugin developers (of which there are many) to track the IP addresses and possibly more of who made these requests. It’s straight forward to become a plugin developer and get included. IP tracking is invisible to the user and OpenAI. A simple fix is to proxy these images and/or cache them on the OpenAI server.

Re: Nepenthes is a tarpit to catch AI web crawlers

#79

Post by jsheard »

observationist wrote: post #69
blibble wrote: post #59

Earlier quoted context omitted.

> If you want to protect your content, use the technical mechanisms that are available, > You can choose to gatekeep your content, and by doing so, make it unscrapeable, and legally protected. so... robots.txt, which the AI parasites ignore? > Also, consider that relatively small, cheap llms are able to parse the difference between meaningful content and Markovian jabber such as this software produces. okay, so it's…

[flagged]

> No, put up a loginwall or paywall, authenticate users, and go private.

We know for a fact that AI companies don't respect that, if they want data that's behind a paywall then they'll jump through hoops to take it anyway.

https://www.theguardian.com/technology/2025/jan/10/mark-zuck...

If they don't have to abide by "norms" then we don't have to for their sake. Fuck 'em.

Post reply on HN