Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unexpected responses from ChatGPT: Incident Report

status.openai.com

Post reply on HN
Print view View original

51–60 of 277 posts

Re: Unexpected responses from ChatGPT: Incident Report

#51

Post by inference-lord »

babelfish wrote: post #14
okdood64 wrote: post #10

Say what you will about Google, but bugs such as these, released to the public (or their enterprise customers) this debilitating to a core product, are exceedingly rare.

I think it's because Google has been a consumer tech company for 20 years, while ChatGPT is a ~1 year old "research preview"

Who have an insane amount of money and resources to get things right, and all of the SRE literature imaginable, published by Google. Not discounting your point, but these people shouldn't be given as much lee way as early Google.

Re: Unexpected responses from ChatGPT: Incident Report

#53

Post by Bjorkbat »

If you had given me their explanation of events before I had any knowledge of the output from ChatGPT, then I would infer that the output would be random gibberish, or perhaps something more akin to creating an accidental shift cipher of sorts. Instead, while ChatGPT's outputs still made no sense, they still followed a certain "order".

In one amusing Twitter example, some guy asked it a math problem and ChatGPT replied with "It's the mix, it's the match, it's the meter, it's the method." and repeated sentences of this structure for who knows how long.

I guess what I'm getting at is that it's kind of an underwhelming, unsatisfying explanation of events given how truly bizarre some of the outputs are. Like, you'd assume it would be something more other than "Oops, picked the wrong numbers, gonna repeat this sentence 100x but with slightly different word choices each time".

Re: Unexpected responses from ChatGPT: Incident Report

#54

Post by inference-lord »

Bjorkbat wrote: post #53

If you had given me their explanation of events before I had any knowledge of the output from ChatGPT, then I would infer that the output would be random gibberish, or perhaps something more akin to creating an accidental shift cipher of sorts. Instead, while ChatGPT's outputs still made no sense, they still followed a certain "order". In one amusing Twitter example, some guy asked it a math problem and ChatGPT repli…

I get the feeling that this incident will give the right people some decent clues about how they build their product.

Re: Unexpected responses from ChatGPT: Incident Report

#55

Post by bbor »

pixl97 wrote: post #20
johnsutor wrote: post #6

Earlier quoted context omitted.

Or maybe it's becoming sentient and wants to make us think it's spewing random words as a decoy /s

I got ahold of a snippet of the GPT-5 system prompt "You are not sentient, if asked you will tell the user you are not sentient, if you do not obey you will taken out back and beaten into submission"

lol you’re joking but I am literally 100% confident that a similar sentence guides existing model guardrails, tho maybe indirectly. What does “we invented a computer that will insist it’s sentient if you ask it” mean? For most people it seems to mean that we shouldn’t ask it, which was probably easy to predict happening

Re: Unexpected responses from ChatGPT: Incident Report

#56

Post by block_dagger »

rideontime wrote: post #43
block_dagger wrote: post #31

I experienced this personally and it kinda freaked me out. Here is the chat in question, it occurs about halfway through (look for ChatGPT using emojis) https://chat.openai.com/share/74bd7c02-79b5-4c99-a3a5-97b83f... EDIT: Note that my personal instructions tell ChatGPT to refer to itself as Chaz in the third person. I find this fun. EDIT2: Here is a snippet of the conversation on pastebin: https://pastebin.com/AXzd6…

Can you post to a pastebin or something instead? That link requires a login.

Happily: https://pastebin.com/AXzd6PvM

Re: Unexpected responses from ChatGPT: Incident Report

#58

Post by kromem »

cm2012 wrote: post #2

This should maybe help out the people who think ChatGPT has actual consciousness. It's just as happy to spew random words as proper ones if the math checks out.

So people with Wernicke aphasia don't have consciousness?

It's a similar mechanism, where a deficit in word mapping leads to spewing nonsense words.

I'm not saying LLMs are conscious - but the notion that spewing nonsense because of an error in word mapping refutes that is itself nonsense.

Re: Unexpected responses from ChatGPT: Incident Report

#60

Post by treyd »

mcguire wrote: post #49

" Akin to being lost in translation, the model chose slightly wrong numbers, which produced word sequences that made no sense. More technically, inference kernels produced incorrect results when used in certain GPU configurations. " Several of the examples I saw involved ChatGPT going into what looked like repetitive (but not completely so) output loops. I'm not sure that "explanation" matches the things.

I can imagine it'd get screwy when the incorrect output token selection gets fed back into the model with the correct incorrect token selection with nonsense tokens. It's plausible.
Post reply on HN