Nepenthes is a tarpit to catch AI web crawlers • Hacker News Board

Re: Nepenthes is a tarpit to catch AI web crawlers

#191

Post by asah » Fri, Jan 17, 2025, 7:48 AM UTC

orf wrote: post #158

bflesch wrote: post #153

Earlier quoted context omitted.

Ego, curiosity, potential bug bounty & this was a low hanging fruit: I was just watching API request in Devtools while using ChatGPT. It took 10 minutes to spot it, and a week of trying to reach a human being. Iterating on the proof-of-concept code to increase potency is also a nice hobby. These kinds of vulnerabilities give you good idea if there could be more to find, and if their bug bounty program actually is wor…

Make it reflective, reflect it back onto an OpenAI API route.

Lol but actually this is a good way to escalate priority. Better yet, point it at various Microsoft sites that aren't provisioned to handle the traffic and let them internally escalate.

Re: Nepenthes is a tarpit to catch AI web crawlers

#192

Post by bflesch » Fri, Jan 17, 2025, 7:54 AM UTC

mitjam wrote: post #179

bflesch wrote: post #54

Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week. Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website. The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As C…

How can it reach localhost or is this only a placeholder for a real address?

The code in the github repo has some errors to prevent script kiddies from directly copy/pasting it.

Obviously the proof-of-concept shared with OpenAI/BugCrowd didn't have such errors.

Re: Nepenthes is a tarpit to catch AI web crawlers

#193

Post by bflesch » Fri, Jan 17, 2025, 7:56 AM UTC

dangoodmanUT wrote: post #175

bflesch wrote: post #54

Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week. Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website. The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As C…

has anyone tested this working? I get a 301 in my terminal trying to send a request to my site

Hopefully they'd have it fixed by now. The magic of HN exposure...

Re: Nepenthes is a tarpit to catch AI web crawlers

#194

Post by Brian_K_White » Fri, Jan 17, 2025, 8:44 AM UTC

permo-w wrote: post #117

bflesch wrote: post #87

Earlier quoted context omitted.

Their security.txt email address replies and asks you to go on BugCrowd. BugCrowd staff is unwilling (or too incompetent) to run a bash curl command to reproduce the issue, while also refusing to forward it to OpenAI. The support@openai.com waits an hour before answering with ChatGPT answer. Issues raised on GitHub directly towards their engineers were not answered. Also Microsoft CERT & Azure security team do not re…

why try this hard for a private company that doesn't employ you?

At least one time it's worth going through all the motions to prove whether it is or is not actually functional, so that they can not say "no one reported a problem..." about all the problems.

You can't say they don't have a funtional process, and they are lying or disingenuous when they claim to, if you never actually tried for real for yourself at least once.

Re: Nepenthes is a tarpit to catch AI web crawlers

#195

Post by mrweasel » Fri, Jan 17, 2025, 8:48 AM UTC

taikahessu wrote: post #30

We had our non-profit website drained out of bandwidth and site closed temporarily (!!) from our hosting deal because of Amazon bot aggressively crawling like ?page=21454 ... etc. Gladly Siteground restored our site without any repercussions as it was not our fault. Added Amazon bot into robots.txt after that one. Don't like how things are right now. Is a tarpit the solution? Or better laws? Would they stop the chine…

> We had our non-profit website drained out of bandwidth

There is a number of sites which are having issues with scrapers (AI and others) generating so much traffic that transit providers are informing them that their fees will go up with the next contract renewal, if the traffic is not reduced. It's just very hard for the individual sites to do much about it, as most of the traffic stems from AWS, GCP or Azure IP ranges.

It is a problem and the AI companies do not care.

Re: Nepenthes is a tarpit to catch AI web crawlers

#196

Post by a_c » Fri, Jan 17, 2025, 9:05 AM UTC

We need a tarpit that feed AI their own hallucination. Make the habsburg dynasty of AI a reality

Re: Nepenthes is a tarpit to catch AI web crawlers

#197

Post by Cthulhu_ » Fri, Jan 17, 2025, 9:32 AM UTC

a_c wrote: post #196

We need a tarpit that feed AI their own hallucination. Make the habsburg dynasty of AI a reality

There was an article about that the other day having to do with image generation, and while it didn't exactly create Hapsburg chins there was definite problems after a few generations. I can't find it though :/

Re: Nepenthes is a tarpit to catch AI web crawlers

#198

Post by flir » Fri, Jan 17, 2025, 9:39 AM UTC

WD-42 wrote: post #146

pmarreck wrote: post #132

Earlier quoted context omitted.

It's not. It's rather pointless and frankly, nearsighted. And we can DDoS sites like this just as offensively as well simply by making many requests to it since its own docs say its Markov generation is computationally expensive, but it is NOT expensive for even 1 person to make many requests to it. Just expensive to host. So feel free to use this bash function to defeat these: httpunch() { local url=$1 local connect…

You called the parent unintelligent yet need an LLM to show you how to run curl in a loop. Yikes.

"I'm not lazy, I'm efficient" - Heinlein

Re: Nepenthes is a tarpit to catch AI web crawlers

#199

Post by EdwardDiego » Fri, Jan 17, 2025, 10:28 AM UTC

phito wrote: post #56

As a carnivorous plant enthusiast, I love the name.

I was just reading about one of these today, that occasionally eats small mammals.

https://en.wikipedia.org/wiki/Nepenthes_attenboroughii

Re: Nepenthes is a tarpit to catch AI web crawlers

#200

Post by netdevphoenix » Fri, Jan 17, 2025, 10:37 AM UTC

permo-w wrote: post #117

bflesch wrote: post #87

Earlier quoted context omitted.

Their security.txt email address replies and asks you to go on BugCrowd. BugCrowd staff is unwilling (or too incompetent) to run a bash curl command to reproduce the issue, while also refusing to forward it to OpenAI. The support@openai.com waits an hour before answering with ChatGPT answer. Issues raised on GitHub directly towards their engineers were not answered. Also Microsoft CERT & Azure security team do not re…

why try this hard for a private company that doesn't employ you?

I always wonder why people not working or planning to work in infosec do this. I get giving up your free time to build open source functionality used by rich for-profit companies that will just make them rich because that's the nature of open source. But literally giving your free time to help a rich company get richer that I do not get. My only explanation is that they enjoy the process. It's like people spending their free time giving information and resources when they would not do that if that person was in front of them.