Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unexpected responses from ChatGPT: Incident Report

status.openai.com

Post reply on HN
Print view View original

131–140 of 277 posts

Re: Unexpected responses from ChatGPT: Incident Report

#131

Post by _heimdall »

astrange wrote: post #118
_heimdall wrote: post #84

This is exactly the kind of issue that can lead to unintended consequences. What if, instead of spewing out seemingly nonsense answers, the LLM spewed out very real answers that violated built-in moderation protocols? Or shared secrets or other users chats? What if a bug released accidentally stumbled upon how to allow the LLM to become self aware? Or paranoid? These potentials seem outlandish, but we honestly don't…

The LLM doesn't have secrets or other users' chats in it. Why would they put that in there?

LLMs can have secrets if they were scraped in the training data.

And how do we know definitively what is done with chat logs? The LLM model is a black box for OpenAI (they don't know what was learned or why it was learned), and OpenAI is a black box for users (we don't know what data they collect or how they use it).

Re: Unexpected responses from ChatGPT: Incident Report

#132

Post by mewpmewp2 »

block_dagger wrote: post #108
fsmv wrote: post #106

Earlier quoted context omitted.

I just mean, given that LLMs exist this isn't a surprising result. It only looks surprising because the UI makes you forget that each prompt is a completely new universe to the model.

I wouldn't call a step in a history-aware conversation a completely new universe. By that logic, every single time a token is generated is a new universe even though the token is largely dependent on the prompt, which includes custom instructions, chat history, and all tokens generated in the response so far.

Well, I would have also said or thought that each token actually is a new universe in a sense. You could rotate between different LLMs for each token for example or instances of the same LLM or branch into different possibilities. Input gets cycled again as a whole.

Re: Unexpected responses from ChatGPT: Incident Report

#133

Post by mempko »

block_dagger wrote: post #31

I experienced this personally and it kinda freaked me out. Here is the chat in question, it occurs about halfway through (look for ChatGPT using emojis) https://chat.openai.com/share/74bd7c02-79b5-4c99-a3a5-97b83f... EDIT: Note that my personal instructions tell ChatGPT to refer to itself as Chaz in the third person. I find this fun. EDIT2: Here is a snippet of the conversation on pastebin: https://pastebin.com/AXzd6…

We need more Chaz in our lives.

Re: Unexpected responses from ChatGPT: Incident Report

#134

Post by bitzun »

okdood64 wrote: post #10

Say what you will about Google, but bugs such as these, released to the public (or their enterprise customers) this debilitating to a core product, are exceedingly rare.

Google can't figure out how to make an mp3 player that doesn't shit itself randomly (Youtube Music, which I pay for).

Re: Unexpected responses from ChatGPT: Incident Report

#135

Post by gedy »

toddmorey wrote: post #48
block_dagger wrote: post #31

I experienced this personally and it kinda freaked me out. Here is the chat in question, it occurs about halfway through (look for ChatGPT using emojis) https://chat.openai.com/share/74bd7c02-79b5-4c99-a3a5-97b83f... EDIT: Note that my personal instructions tell ChatGPT to refer to itself as Chaz in the third person. I find this fun. EDIT2: Here is a snippet of the conversation on pastebin: https://pastebin.com/AXzd6…

I mean, this is almost poetic: Classic plays, wholly told. Shadow into form, hands as cue. Keep it at the dial, in right on Pitch.

Transcript kind of works as a song/rap:

https://app.suno.ai/song/57892fb8-753b-4f91-a690-49f491f782b...

or Reggae:

https://app.suno.ai/song/97dbc42b-3004-4d2e-8b59-0f758cbad33...

Re: Unexpected responses from ChatGPT: Incident Report

#136

Post by fs_tab »

bbor wrote: post #36
cm2012 wrote: post #2

This should maybe help out the people who think ChatGPT has actual consciousness. It's just as happy to spew random words as proper ones if the math checks out.

Posting one more time: this is proof that AI is connected to human-like linguistic patterns, IMO. No, it obviously doesn’t have “consciousness” in the sense of an ongoing stream-of-consciousness monologue, but that doesn’t mean it’s not mimicking some real part of human cognition. https://en.wikipedia.org/wiki/Colorless_green_ideas_sleep_fu...

"We find that the larger neural language models get, the more their representations are structurally similar to neural response measurements from brain imaging."

https://arxiv.org/abs/2306.01930

Re: Unexpected responses from ChatGPT: Incident Report

#137

Post by astrange »

_heimdall wrote: post #131
astrange wrote: post #118

Earlier quoted context omitted.

The LLM doesn't have secrets or other users' chats in it. Why would they put that in there?

LLMs can have secrets if they were scraped in the training data. And how do we know definitively what is done with chat logs? The LLM model is a black box for OpenAI (they don't know what was learned or why it was learned), and OpenAI is a black box for users (we don't know what data they collect or how they use it).

You can probe what was learned if you have access to the model; it'll tell you, especially if you do it before applying the safety features.

A good heuristic for whether they would train user chats into the model is whether this makes any sense. But it doesn't; it's not valuable. They could be saying anything in there, it's likely private, and it's probably not truthful information.

Presumably they do do something with responses you've marked thumbs up/thumbs down to, but there are ways of using those that aren't directly putting them in the training data. After all, that feedback isn't trustworthy either.

Re: Unexpected responses from ChatGPT: Incident Report

#138

Post by astrange »

ummonk wrote: post #126
astrange wrote: post #118

Earlier quoted context omitted.

The LLM doesn't have secrets or other users' chats in it. Why would they put that in there?

If they use batching during inference (which they very probably do), then some kind of coding mistake of the sort that happened with this bug absolutely could result in leakage between chats.

One thing that did happen is there was a bug in the website for a day that really did show you other user's chat history.

IIRC some reporting confused this with "Samsung had some employees upload internal PDFs to ChatGPT" to produce the claim that ChatGPT was leaking internal Samsung information via training, which it wasn't.

Re: Unexpected responses from ChatGPT: Incident Report

#139

Post by skissane »

bongodongobob wrote: post #111
skissane wrote: post #68

Earlier quoted context omitted.

Whether "ChatGPT has actual consciousness" depends on what you consider "consciousness" to be, and what are your criteria for deciding whether something has it. Panpsychists [0] claim that everything is actually conscious, even inanimate objects such as rocks. If rocks have actual consciousness, why can't ChatGPT have it too? And the fact that ChatGPT sometimes talks gibberish would be irrelevant, since rocks never s…

Saying rocks are conscious is a poor summary. It's more like adult human > human child > dolphin > dog > human infant > bird > snake > spider > ant > dust mite etc. The whole thing is a continuum and everything is made of matter, so there's a little bit of potential consciousness in all matter.

> Saying rocks are conscious is a poor summary

> The whole thing is a continuum and everything is made of matter, so there's a little bit of potential consciousness in all matter

I disagree that my summary is "poor"–because some panpsychists do say that rocks are actually conscious individuals, as opposed to merely containing "a little bit of potential consciousness". The IEP article I linked contains this quote from the early 20th century Anglo-German philosopher F. C. S. Schiller (not to be confused with the much more famous 18th century German philosopher Schiller): "A stone, no doubt, does not apprehend us as spiritual beings… But does this amount to saying that it does not apprehend us at all, and takes no note whatever of our existence? Not at all; it is aware of us and affected by us on the plane on which its own existence is passed… It faithfully exercises all the physical functions, and influences us by so doing. It gravitates and resists pressure, and obstructs…vibrations, and so forth, and makes itself respected as such a body. And it treats us as if of a like nature with itself, on the level of its understanding…"

Of course, panpsychism has never been a single theory, it is a family of related theories, and so not every panpsychist would agree with that quote from Schiller–but I don't believe his view on rocks is unique to him either.

Re: Unexpected responses from ChatGPT: Incident Report

#140

Post by skissane »

card_zero wrote: post #127
bongodongobob wrote: post #111

Earlier quoted context omitted.

Saying rocks are conscious is a poor summary. It's more like adult human > human child > dolphin > dog > human infant > bird > snake > spider > ant > dust mite etc. The whole thing is a continuum and everything is made of matter, so there's a little bit of potential consciousness in all matter.

I'm trying to think of a civil and constructive way to say "bullshit". I guess an obvious objection is "why?". Then something about Russell's teapot. There could be consciousness hidden in every atom, in some physics-defying way that we can't yet comprehend: there could also be garden furniture hidden up my nose, or a teapot hidden far out in the solar system, which is the most likely of the three since it's at least…

> I guess an obvious objection is "why?"

Because coming up with criteria for determining what is and isn't conscious, and justifying the particular criterion you choose (as opposed to all the alternatives) is hard. Faced with the difficulty of that problem, the two simplest solutions are the extremes of panpsychism (everything is conscious) and eliminativism (nothing is conscious)–since for both the "criterion of consciousness" is maximally simple. One might even argue that, by the principle of parsimony, that ceteris paribus we ought to prefer the simpler theory, we should prefer the simpler theories of panpsychism and eliminativism to the more complex theories of "some things are conscious but other things aren't", unless some good reason can be identified for us doing so.

> in some physics-defying way that we can't yet comprehend

How is panpsychism "physics-defying"? Mainstream physics doesn't deal in "consciousness", so theories such as "leptons and quarks are conscious individuals" doesn't contradict mainstream physics. Well, it quite possibly would contradict the von Neumann-Wigner interpretation of QM, but I don't think many would consider that "mainstream"

Post reply on HN