Nepenthes is a tarpit to catch AI web crawlers • Hacker News Board

Re: Nepenthes is a tarpit to catch AI web crawlers

#91

Post by jonatron » Thu, Jan 16, 2025, 5:02 PM UTC

dawnerd wrote: post #76

hartator wrote: post #12

There are already “infinite” websites like these on the Internet. Crawlers (both AI and regular search) have a set number of pages they want to crawl per domain. This number is usually determined by the popularity of the domain. Unknown websites will get very few crawls per day whereas popular sites millions. Source: I am the CEO of SerpApi.

Looking at my logs for all of my sites and this isn’t a global truth. I see multiple ai crawlers hammering away requesting the same pages many, many times. Perplexity and Facebook are basically nonstop.

I just looked at the logs for a site, and I saw PerplexityBot is looking at the robots.txt and ignoring it. They don't provide a list of IPs to verify if it is actually them. Anyway, just for anyone with PerplexityBot in their user agent, they can get increasingly bad responses until the abuse stops.

Re: Nepenthes is a tarpit to catch AI web crawlers

#92

Post by dewey » Thu, Jan 16, 2025, 5:06 PM UTC

JohnMakin wrote: post #66

bflesch wrote: post #54

Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week. Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website. The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As C…

Nice find, I think one of my sites actually got recently hit by something like this. And yea, this kind of thing should be trivially preventable if they cared at all.

> And yea, this kind of thing should be trivially preventable if they cared at all.

Most of the time when someone says something is "trivial" without knowing anything about the internals, it's never trivial.

As someone working close to the b2c side of a business, I can’t count the amount of times I've heard that something should be trivial while it's something we've thought about for years.

Re: Nepenthes is a tarpit to catch AI web crawlers

#93

Post by 0xf00ff00f » Thu, Jan 16, 2025, 5:09 PM UTC

kerkeslager wrote: post #72

Question: do these bots not respect robots.txt? I haven't added these scrapers to my robots.txt on the sites I work on yet because I haven't seen any problems. I would run something like this on my own websites, but I can't see selling my clients on running this on their websites. The websites I run generally have a honeypot page which is linked in the headers and disallowed to everyone in the robots.txt, and if an I…

> The websites I run generally have a honeypot page which is linked in the headers and disallowed to everyone in the robots.txt, and if an IP visits that page, they get added to a blocklist which simply drops their connections without response for 24 hours.

I love this idea!

Re: Nepenthes is a tarpit to catch AI web crawlers

#94

Post by grahamj » Thu, Jan 16, 2025, 5:11 PM UTC

That’s so funny, I’ve thought of this exact idea several times over the last couple of weeks. As usual someone beat me to it :D

Re: Nepenthes is a tarpit to catch AI web crawlers

#95

Post by tofof » Thu, Jan 16, 2025, 5:12 PM UTC

observationist wrote: post #51

jsheard wrote: post #48

Earlier quoted context omitted.

This is a really bad take, it's not like this server is hacking clients which connect to it. It's providing perfectly valid HTTP responses that just happen to be slow and full of markov gibberish, any harm which comes of that is self inflicted by assuming that websites must provide valuable data as a matter of course. If AI companies want to sue webmasters for that then by all means, they can waste their money and ge…

[flagged]

He's not interfering with any normal operation of any system. He is offering links. You can follow them or not, entirely at your own discretion. Those links load slowly. You can wait for them to complete or not, entirely at your own discretion.

The crawler's normal operation is not interfered with in any way: the crawler does exactly what it's programmed to do. If its programmers decided it should exhaustively follow links, he's not preventing it from doing that operation.

Legally, at best you'd be looking to warp the concept of attractive nuisance to apply to a crawler. As that legal concept is generally intended to prevent bodily harm to children, however, good luck.

Re: Nepenthes is a tarpit to catch AI web crawlers

#96

Post by DigiEggz » Thu, Jan 16, 2025, 5:13 PM UTC

Amazing project. I hope to see this put to serious use.

As a quick note and not sure if it's already been mentioned, but the main blurb has a typo: "... go back into a the tarpit"

Re: Nepenthes is a tarpit to catch AI web crawlers

#97

Post by grahamj » Thu, Jan 16, 2025, 5:14 PM UTC

dewey wrote: post #92

JohnMakin wrote: post #66

Earlier quoted context omitted.

Nice find, I think one of my sites actually got recently hit by something like this. And yea, this kind of thing should be trivially preventable if they cared at all.

> And yea, this kind of thing should be trivially preventable if they cared at all. Most of the time when someone says something is "trivial" without knowing anything about the internals, it's never trivial. As someone working close to the b2c side of a business, I can’t count the amount of times I've heard that something should be trivial while it's something we've thought about for years.

If you’re unable to throttle your own outgoing requests you shouldn’t be making any

Re: Nepenthes is a tarpit to catch AI web crawlers

#98

Post by reginald78 » Thu, Jan 16, 2025, 5:17 PM UTC

Is there a reason people can't use hashcash or some other proof of work system on these bad citizen crawlers?

Re: Nepenthes is a tarpit to catch AI web crawlers

#99

Post by bflesch » Thu, Jan 16, 2025, 5:17 PM UTC

dewey wrote: post #92

JohnMakin wrote: post #66

Earlier quoted context omitted.

Nice find, I think one of my sites actually got recently hit by something like this. And yea, this kind of thing should be trivially preventable if they cared at all.

> And yea, this kind of thing should be trivially preventable if they cared at all. Most of the time when someone says something is "trivial" without knowing anything about the internals, it's never trivial. As someone working close to the b2c side of a business, I can’t count the amount of times I've heard that something should be trivial while it's something we've thought about for years.

The technical flaws are quite trivial to spot, if you have the relevant experience:

- urls[] parameter has no size limit

- urls[] parameter is not deduplicated (but their cache is deduplicating, so this security control was there at some point but is ineffective now)

- their requests to same website / DNS / victim IP address rotate through all available Azure IPs, which gives them risk of being blocked by other hosters. They should come from the same IP address. I noticed them changing to other Azure IP ranges several times, most likely because they got blocked/rate limited by Hetzner or other counterparties from which I was playing around with this vulnerabilities.

But if their team is too limited to recognize security risks, there is nothing one can do. Maybe they were occupied last week with the office gossip around the sexual assault lawsuit against Sam Altman. Maybe they still had holidays or there was another, higher-risk security vulnerability.

Having interacted with several bug bounties in the past, it feels OpenAI is not very mature in that regard. Also why do they choose BugCrowd when HackerOne is much better in my experience.

Re: Nepenthes is a tarpit to catch AI web crawlers

#100

Post by bflesch » Thu, Jan 16, 2025, 5:20 PM UTC

grahamj wrote: post #97

dewey wrote: post #92

Earlier quoted context omitted.

> And yea, this kind of thing should be trivially preventable if they cared at all. Most of the time when someone says something is "trivial" without knowing anything about the internals, it's never trivial. As someone working close to the b2c side of a business, I can’t count the amount of times I've heard that something should be trivial while it's something we've thought about for years.

If you’re unable to throttle your own outgoing requests you shouldn’t be making any

I assume it'll be hard for them to notice because it's all coming from Azure IP ranges. OpenAI has very big credit card behind this Azure account so this vulnerability might only be limited by Azure capacity.

I noticed they switched their crawler to new IP ranges several times, but unfortunately Microsoft CERT / Azure security team didn't answer to my reports.

If this vulnerability is exploited, it hits your server with MANY requests per second, right from the hearts of Azure cloud.