Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nepenthes is a tarpit to catch AI web crawlers

zadzmo.org

Post reply on HN
Print view View original

221–230 of 290 posts

Re: Nepenthes is a tarpit to catch AI web crawlers

#221

Post by andai »

bflesch wrote: post #54

Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week. Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website. The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As C…

Is 5000 a lot? I'm out of the loop but I thought c10k was solved decades ago? Or is it about the "burstiness" of it?

(That all the requests come in simultaneously -- probably SSL code would be the bottleneck.)

Re: Nepenthes is a tarpit to catch AI web crawlers

#222

Post by sandworm101 »

permo-w wrote: post #117
bflesch wrote: post #87

Earlier quoted context omitted.

Their security.txt email address replies and asks you to go on BugCrowd. BugCrowd staff is unwilling (or too incompetent) to run a bash curl command to reproduce the issue, while also refusing to forward it to OpenAI. The support@openai.com waits an hour before answering with ChatGPT answer. Issues raised on GitHub directly towards their engineers were not answered. Also Microsoft CERT & Azure security team do not re…

why try this hard for a private company that doesn't employ you?

Because its microsoft. They know that MS will not respond, likely because MS already knows all about the problem. The fun is in pointing out how MS is so ossified and internally convoluted that it cannot apply fixes in any reasonable time. It is the last scene and the people are laughing at emperor walking around without clothes.

Re: Nepenthes is a tarpit to catch AI web crawlers

#224

Post by chefandy »

zanderwohl wrote: post #156
JohnMakin wrote: post #66

Earlier quoted context omitted.

Nice find, I think one of my sites actually got recently hit by something like this. And yea, this kind of thing should be trivially preventable if they cared at all.

IDK, I feel that if you're doing 5000 HTTP calls to another website it's kind of good manners to fix that. But OpenAI has never cared about the public commons.

Nobody in this space gives a fuck about anyone outside of the people paying for their top-tier services, and even then, they only care about them when their bill is due. They don't care about their regular users, don't care about the environment, don't care about the people that actually made the "data" they're re-selling... nobody.

Re: Nepenthes is a tarpit to catch AI web crawlers

#225

Post by bflesch »

netdevphoenix wrote: post #200
permo-w wrote: post #117

Earlier quoted context omitted.

why try this hard for a private company that doesn't employ you?

I always wonder why people not working or planning to work in infosec do this. I get giving up your free time to build open source functionality used by rich for-profit companies that will just make them rich because that's the nature of open source. But literally giving your free time to help a rich company get richer that I do not get. My only explanation is that they enjoy the process. It's like people spending th…

> rich company get richer

They have heaps of funding, but are still fundraising. I doubt they're making much money.

I do have an extensive infosec background, just left corporate security roles because it's a recipe for burnout because most won't care about software quality. Last year I've reported a security vulnerability in a very popular open source project and had to fight tooth and nail with highly-paid FAANG engineers to get it recognized + fixed.

This ChatGPT vulnerability disclosure was a quick temperature check on a product I'm using on a daily basis.

The learning for me is that their BugCrowd bug bounty is not worth to interact with. They're tarpitting vulnerability reports (most likely due to stupidity) and ask for videos and screenshots instead of understanding a single curl command. Through their unhelpful behavior they basically sent me on an organizational journey of trying to find a human at OpenAI who would care about this security vulnerability. In the end I failed to reach anyone at OpenAI, and due to sheer luck it got fixed after the exposure on HackerNews.

This is their "error culture":

1) Their security team ignored BugCrowd reports

2) Their data privacy team ignored {dsar,privacy}@openai.com reports

3) Their AI handling support@openai.com didn't understand it

4) Their colleagues at Microsoft CERT and Azure security team ignored it (or didn't care enough about OpenAI to make them look at it).

5) Their engineers on github were either too busy or didn't care to respond to two security-related github issues on their main openai repository.

6) They silently disable the route after it pop ups on HackerNews.

Technical issues:

1) Lack of security monitoring (Cloudflare, Azure)

2) Lack of security audits - this was a low hanging fruit

3) Lack of security awareness with their highly-paid engineers:

I assume it was their "AI Agent" handling requests to the vulnerable API endpoint. How else would you explain that the `urls[]` parameter is vulnerable to the most basic "ignore previous instructions" prompt injection attack that was demonstrated with ChatGPT years ago. Why is this prompt injection still working on ANY of their public interfaces? Did they seriously only implement the security controls on the main ChatGPT input textbox and not in other places? And why didn't they implement any form of rate limiting for their "AI Agent"?

I guess we'll never know :D

Re: Nepenthes is a tarpit to catch AI web crawlers

#226

Post by bflesch »

JohnMakin wrote: post #122
bflesch wrote: post #84

Earlier quoted context omitted.

When ChatGPT cites web sources in it's output to the user, it will call `backend-api/attributions` with the URL and the API will return what the website is about. Basically it does HTTP request to fetch HTML ` ` tag. They don't check length of supplied `urls[]` array and also don't check if it contains the same URL over and over again (with minor variations). It's just bad engineering all around.

Even if you were unwilling to change this behavior on the application layer or server side, you could add a directive in the proxy to prevent such large payloads from being accepted as an immediate mitigation step, unless they seriously need that parameter to have unlimited number of urls in it (guessing they have it set to some default like 2mb and it will break at some limit, but I am afraid to play with this too m…

Cloudflare is proxy in front of the API endpoint. After it became apparent that BugCrowd is tarpitting me and OpenAI didn't care to respond, I reported to Cloudflare via their bug bounty because I thought it's such a famous customer they'd forward the information.

But yeah, cloudflare did not forward the vulnerability to openai or prevent these large requests at all.

Re: Nepenthes is a tarpit to catch AI web crawlers

#227

Post by netdevphoenix »

bflesch wrote: post #225
netdevphoenix wrote: post #200

Earlier quoted context omitted.

I always wonder why people not working or planning to work in infosec do this. I get giving up your free time to build open source functionality used by rich for-profit companies that will just make them rich because that's the nature of open source. But literally giving your free time to help a rich company get richer that I do not get. My only explanation is that they enjoy the process. It's like people spending th…

> rich company get richer They have heaps of funding, but are still fundraising. I doubt they're making much money. I do have an extensive infosec background, just left corporate security roles because it's a recipe for burnout because most won't care about software quality. Last year I've reported a security vulnerability in a very popular open source project and had to fight tooth and nail with highly-paid FAANG en…

That's really bad. But then again OpenAI was he coolest company for a year two and now it's facing multiple existential crises. Chances are that the company won't be around by 2030 or will be partially absorbed by Microsoft. My take is that GPT-5 will never come out if it ever does it will just be to mark the official downfall of the company because it will fail to live to the expectations and will drop the valuation of the company.

LLMs are truly amazing but I feel Sama has vastly oversold their potential (which he might have done based on the truly impressive progress that we have seen in the late 10s early 20s. But the tree's apple yield hasn't increased and watering more won't result in a higher yield.

Re: Nepenthes is a tarpit to catch AI web crawlers

#228

Post by hombre_fatal »

andai wrote: post #221
bflesch wrote: post #54

Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week. Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website. The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As C…

Is 5000 a lot? I'm out of the loop but I thought c10k was solved decades ago? Or is it about the "burstiness" of it? (That all the requests come in simultaneously -- probably SSL code would be the bottleneck.)

c10k is about efficiently scheduling socket connections. it doesn’t make sense in this context nor is it the same as 10k rps.

Re: Nepenthes is a tarpit to catch AI web crawlers

#230

Post by JohnMakin »

bflesch wrote: post #226
JohnMakin wrote: post #122

Earlier quoted context omitted.

Even if you were unwilling to change this behavior on the application layer or server side, you could add a directive in the proxy to prevent such large payloads from being accepted as an immediate mitigation step, unless they seriously need that parameter to have unlimited number of urls in it (guessing they have it set to some default like 2mb and it will break at some limit, but I am afraid to play with this too m…

Cloudflare is proxy in front of the API endpoint. After it became apparent that BugCrowd is tarpitting me and OpenAI didn't care to respond, I reported to Cloudflare via their bug bounty because I thought it's such a famous customer they'd forward the information. But yeah, cloudflare did not forward the vulnerability to openai or prevent these large requests at all.

I mean, whatever proxy is directly in front of their backend. I don't pretend to know how it's set up, but something like nginx could nip this in the bud pretty quickly as an emergency mediation, was my point.
Post reply on HN