Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI – vulnerability responsible disclosure

requilence.any.org

Post reply on HN
Print view View original

41–50 of 87 posts

Re: OpenAI – vulnerability responsible disclosure

#41

Post by milkshakes »

pyman wrote: post #20
999900000999 wrote: post #11

Earlier quoted context omitted.

Users should always avoid sharing sensitive data. A lot of AI products straight up have plan text logs available for everyone at the company to view.

It's not just about sensitive data like passwords, contracts, or IP. It's also about the personal conversations people have with ChatGPT. Some are depressed, some are dealing with bullying, others are trying to figure out how to come out to their parents. For them, this isn't just sensitive, it's life-changing if it gets leaked. It's like Meta leaking their WhatsApp messages. I really hope they fix this bug and start…

maybe you should stop trusting random people on the internet making extraordinary claims without proof then?

Re: OpenAI – vulnerability responsible disclosure

#42

Post by baby_souffle »

milkshakes wrote: post #41
pyman wrote: post #20

Earlier quoted context omitted.

It's not just about sensitive data like passwords, contracts, or IP. It's also about the personal conversations people have with ChatGPT. Some are depressed, some are dealing with bullying, others are trying to figure out how to come out to their parents. For them, this isn't just sensitive, it's life-changing if it gets leaked. It's like Meta leaking their WhatsApp messages. I really hope they fix this bug and start…

maybe you should stop trusting random people on the internet making extraordinary claims without proof then?

Isn't "assume vulnerable" The only prudent thing to do here?

Re: OpenAI – vulnerability responsible disclosure

#43

Post by jojobas »

Sebguer wrote: post #40
requilence wrote: post #32

Earlier quoted context omitted.

In one of the responses, it provided the financial analysis of a not well-known company with a non-Latin name located in a small country. I found this company; it is real and numbers in the response are real. When I asked my ChatGPT to provide a financial report for this company without using web tools, it responded: `Unfortunately, I don’t have specific financial statements for “xxx” for 2021 and 2022 in my training…

Do you understand what a hallucination is?

Coming up with accurate financial data that you can't get it to report outright doesn't seem like one.

Re: OpenAI – vulnerability responsible disclosure

#44

Post by refulgentis »

jojobas wrote: post #43
Sebguer wrote: post #40

Earlier quoted context omitted.

Do you understand what a hallucination is?

Coming up with accurate financial data that you can't get it to report outright doesn't seem like one.

I don't understand the wording

Accurate financial data?

How do we know?

What does using not-web-search not having the data have to do with the claim that private chats with the data are being leaked?

Re: OpenAI – vulnerability responsible disclosure

#45

Post by refulgentis »

baby_souffle wrote: post #42
milkshakes wrote: post #41

Earlier quoted context omitted.

maybe you should stop trusting random people on the internet making extraordinary claims without proof then?

Isn't "assume vulnerable" The only prudent thing to do here?

No? Yes? Mu?

After some hemming and hawing, my most cromulent thought is, having good security posture isn't synonymous with accepting every claim you get from the firehose

Re: OpenAI – vulnerability responsible disclosure

#46

Post by milkshakes »

baby_souffle wrote: post #42
milkshakes wrote: post #41

Earlier quoted context omitted.

maybe you should stop trusting random people on the internet making extraordinary claims without proof then?

Isn't "assume vulnerable" The only prudent thing to do here?

everything is vulnerable. the question is, has this researcher demonstrated that they have discovered and successfully exploited such a vulnerability. what exactly in this post makes you believe that this is the case?

Re: OpenAI – vulnerability responsible disclosure

#47

Post by astrange »

thorum wrote: post #22

> The leaked responses show clear signs of being real conversations: they start with contextually appropriate replies, sometimes reference the original user question, appear in various languages, and maintain coherent conversational flow. This pattern is inconsistent with random model hallucinations but matches exactly what you'd expect from misdirected user sessions. A model like GPT-4o can hallucinated responses th…

GPT-4o's writing style is so specific that I find it hard to believe it could fake a user query.

You can spot anyone using AI writing a mile away. It stopped saying "delve" but started saying stuff like "It's not X–it's Y" and "check out the vibes (string of wacky emoji)" constantly.

Re: OpenAI – vulnerability responsible disclosure

#48

Post by Sebguer »

jojobas wrote: post #43
Sebguer wrote: post #40

Earlier quoted context omitted.

Do you understand what a hallucination is?

Coming up with accurate financial data that you can't get it to report outright doesn't seem like one.

Models do not possess awareness of their training data. Also you are taking at face value that it is "accurate".

Re: OpenAI – vulnerability responsible disclosure

#49

Post by wavemode »

astrange wrote: post #47
thorum wrote: post #22

> The leaked responses show clear signs of being real conversations: they start with contextually appropriate replies, sometimes reference the original user question, appear in various languages, and maintain coherent conversational flow. This pattern is inconsistent with random model hallucinations but matches exactly what you'd expect from misdirected user sessions. A model like GPT-4o can hallucinated responses th…

GPT-4o's writing style is so specific that I find it hard to believe it could fake a user query. You can spot anyone using AI writing a mile away. It stopped saying "delve" but started saying stuff like "It's not X–it's Y" and "check out the vibes (string of wacky emoji)" constantly.

LLMs are trained and fine-tuned on real conversations, so resembling a real conversation doesn't really rule out hallucination.

If the story in OP about getting a company's private financial data is true (i.e. the numbers are correct and nonpublic) that could be a smoking gun.

Either way it's a bad look for OpenAI to have not responded to this. Even if the resolution turns out to be that these are just hallucinations, it should've been investigated and responded to by now if OpenAI actually care about security.

Re: OpenAI – vulnerability responsible disclosure

#50

Post by postalcoder »

jonrouach wrote: post #28
requilence wrote: post #10

Earlier quoted context omitted.

No, definitely not the empty string hallucination bug. These are clearly real user conversations. They start like proper replies to requests, sometimes reference the original question, and appear in different languages.

i had the exact same behavior back in 2023, it seemed like clearly leakage of user conversations - but it was just a bug with api calls in the software i was using. https://snipboard.io/FXOkdK.jpg

There was an issue with conversation leakage, though. It involved some bug with Redis.

I felt like it was a huge deal at the time but it’s surprisingly hard to quickly google it.

Post reply on HN