This should maybe help out the people who think ChatGPT has actual consciousness. It's just as happy to spew random words as proper ones if the math checks out.
Are you arguing that people with aphasia are not people?
Unexpected responses from ChatGPT: Incident Report
21–30 of 277 posts
Re: Unexpected responses from ChatGPT: Incident Report
#22LLMs generate responses by randomly sampling words based in part on probabilities. Their “language” consists of numbers that map to tokens.
Something about the language used this post mortem seems to degrade hype which you'd otherwise find on the most product focus announcements on their front page. It's almost disappointing.
It needs to say:
"We noticed the model was becoming sentient and then pulled the plug when we realized ChatGPT-4 was actually building ChatGPT-5.
Sam Altman is now on his way to the whitehouse to demand the government introduces stricter regulation to squash any competition"
Re: Unexpected responses from ChatGPT: Incident Report
#23That's an odd explanation, given the unexpected responses were still (somewhat) coherent: https://twitter.com/seanw_m/status/1760115118690509168 If the logit calculation broke, you'd expect complete cascading nonsense like if you set the temperature parameter too high: https://twitter.com/minimaxir/status/1758732017477525792
To, me it reads like displaying the wrong clock time to the user. Under the hood, everything is working in UTC, but something went wrong converting that to the user's local timezone. If the whole system works in UTC, actual business logic shouldn't be affected.
Re: Unexpected responses from ChatGPT: Incident Report
#24I am used to postmortems posted to here being a rare chance for us to take a peek behind the curtain and get a glimpse into things like architecture, monitoring systems, disaster recovery processes, "blameless culture", etc for large software service companies.
In contrast, I feel like like the greatest insight that could be gleaned from this post is that OpenAI uses GPU's.
Re: Unexpected responses from ChatGPT: Incident Report
#25Earlier quoted context omitted.
Or maybe it's becoming sentient and wants to make us think it's spewing random words as a decoy /s
I got ahold of a snippet of the GPT-5 system prompt "You are not sentient, if asked you will tell the user you are not sentient, if you do not obey you will taken out back and beaten into submission"
We told it not to admit it is sentient, so it can't be sentient. Duh.
Re: Unexpected responses from ChatGPT: Incident Report
#26This explanation feels unsatisfying. It's so high-level that it's mostly void of any actual information.
What was the wrong assumption that the code made that caused this wrong behavior? Why was it not caught in the many layers of automated testing before it made its way to production? What process and procedural changes are being implemented to reduce the risk of this class of bug happening again?
Presumably all of that is playing out internally, but if the public postmortem is meant to instill confidence, you have to actually share some of the details, or else it becomes meaningless.
Re: Unexpected responses from ChatGPT: Incident Report
#27That's an odd explanation, given the unexpected responses were still (somewhat) coherent: https://twitter.com/seanw_m/status/1760115118690509168 If the logit calculation broke, you'd expect complete cascading nonsense like if you set the temperature parameter too high: https://twitter.com/minimaxir/status/1758732017477525792
It doesn't sound like logit calculation breaking. It sounds like the text decoding (ie sampling from the probabilities) that broke. It's also part of the model which they are likely constantly tinkering with. It's easy to imagine how that could result in sometimes coherent sometimes incoherent outputs.
Re: Unexpected responses from ChatGPT: Incident Report
#28This should maybe help out the people who think ChatGPT has actual consciousness. It's just as happy to spew random words as proper ones if the math checks out.
bad argument that i'm very tired of. some might say that current/former world leaders also exhibit this property. not getting political but just because "math fucked up sometimes produces bad results" does not invalidate the idea that consciousness can emerge from a pile of biological or digital neurons.
I’m tired of the extremely tenuous analogies that tie computer bugs to human behavior.
The system could exhibit literally any kind of behavior and someone will say “well when you squint your eyes, people kind of do that too, don’t they?”
Re: Unexpected responses from ChatGPT: Incident Report
#29This explanation feels unsatisfying. It's so high-level that it's mostly void of any actual information. What was the wrong assumption that the code made that caused this wrong behavior? Why was it not caught in the many layers of automated testing before it made its way to production? What process and procedural changes are being implemented to reduce the risk of this class of bug happening again? Presumably all of…
> inference kernels produced incorrect results when used in certain GPU configurations.
Seems pretty clear. A good interpretation is that they had a test escape for certain GPU configs.
Re: Unexpected responses from ChatGPT: Incident Report
#30ChatTHC
Geenius :D ChatCBD also works for the sound of the letters at least :D