Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT unexpectedly began speaking in a user's cloned voice during testing

arstechnica.com

Post reply on HN
Print view View original

51–60 of 164 posts

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#51

Post by AmericanChopper »

skybrian wrote: post #27

It’s “unexpected” because their early training didn’t get rid of it as well as they hoped. LLM’s are good at detecting patterns and like to continue the pattern. They’re starting with autocomplete for voice and training it to do something else. For now, it’s fairly harmless since it’s only a blooper in a lab, but there will likely be open-weights versions of this sort of thing eventually. And there will probably be p…

> there will likely be open-weights versions of this sort of thing eventually. And there will probably be people who argue that it’s a good thing, somehow.

If this technology is going to exist (which it is), do you think it would be more harmful for everybody to have access to it, or for only some group of elites (big business, your own government, somebody else government, criminal groups, whoever) to have access to it? Because those are the only two choices as far as I can tell.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#53

Post by visarga »

EGreg wrote: post #9
golergka wrote: post #2

This looks extremely similar to what often happened with chat implementations in 3 and 3.5 era: GPT world generate it's answer and then go on and generate the next input by the user as well.

I want to know, if LLMs were unleashed on simulating real-world people, how well could they predict everything they are going to say? In battles or business, how well would LLMs predict what people will do? Seems to me that a billion parameters can model decision making by a group of humans pretty well. Especially strategies to psych them out like anticipating everything they try ahead of time, and showing them it’s…

> how well could they predict everything they are going to say?

Maybe they can predict preferences, not everything we are going to say

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#54

Post by throwaway48540 »

AISnakeOil wrote: post #44

So basically the model took user data and used it as training data in realtime? Big if true.

It did the same thing ChatGPT does when it picks up your writing style and exact words/sentences after a few messages. Literally - the audio is encoded as tokens and fed to the LLM, there is no distinction between text and audio from the model's point of view.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#55

Post by Jensson »

AISnakeOil wrote: post #44

So basically the model took user data and used it as training data in realtime? Big if true.

No, the whole point of transformer architecture is that it can do stuff like this without any extra training, an LLM can copy your writing pattern etc.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#56

Post by aimazon »

c0mbonat0r wrote: post #38
skybrian wrote: post #27

It’s “unexpected” because their early training didn’t get rid of it as well as they hoped. LLM’s are good at detecting patterns and like to continue the pattern. They’re starting with autocomplete for voice and training it to do something else. For now, it’s fairly harmless since it’s only a blooper in a lab, but there will likely be open-weights versions of this sort of thing eventually. And there will probably be p…

Can someone pleasee convince why i shouldn't be absolutely shit out of my mind cynical about this innovation? we are literally seeing the downfall of trust in society. and no, i dont believe i am exaggerating

Attacks on trust have been around for as long as we have had trust. Generative AI makes some attack vectors easier but it's nothing new: if your trust is earned using a voice you recognise then you model for trust has been broken since before most of us were born.

https://en.wikipedia.org/wiki/Impersonator

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#58

Post by throwaway48540 »

bee_rider wrote: post #33
sigmoid10 wrote: post #7

This problem appeared during pre-release testing and has since been solved post-generation using an output classifier that verifies responses, according to the system card release. It was predictable that someone would spin this into a black mirror-esque clickbait story.

We built the torment nexus and then configured it not to use any of the torment functionality that showed up in testing. It was predictable that someone would turn this into a “don’t build the torment nexus”-esque clickbait story.

Only a HN user could twist a text/audio generator to present it as "torment nexus".

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#59

Post by Jensson »

kmeisthax wrote: post #40
Jensson wrote: post #8

Earlier quoted context omitted.

The capability is real even if it wont happen with current model censorship. Just have to wait until we get an open source version, I bet Meta is working on one right now.

Even the FOSS models will be "censored". Icky implications aside, a voice assistant that starts making up its own prompts instead of listening to you is not a useful product.

> a voice assistant that starts making up its own prompts instead of listening to you is not a useful product

That would be way more useful than a voice assistant, with that you can replace voice actors for cheap! Many would pay good money for that product.

The important part is that you can ask it to do specific voice lines with specific feelings and voices. If it then starts to copy your voice lines afterwards that wont take from the fact that it generated the voice lines you wanted first. The censor that prevents it from using other voices prevents this use case.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#60

Post by AmericanChopper »

c0mbonat0r wrote: post #38
skybrian wrote: post #27

It’s “unexpected” because their early training didn’t get rid of it as well as they hoped. LLM’s are good at detecting patterns and like to continue the pattern. They’re starting with autocomplete for voice and training it to do something else. For now, it’s fairly harmless since it’s only a blooper in a lab, but there will likely be open-weights versions of this sort of thing eventually. And there will probably be p…

Can someone pleasee convince why i shouldn't be absolutely shit out of my mind cynical about this innovation? we are literally seeing the downfall of trust in society. and no, i dont believe i am exaggerating

The way I see it LLMs are just making it more obvious to see all the flaws with the existing levels of trust. Humans have never had access to universal truths, or universal ways of validating anything. Any claim anybody makes could be intentionally or unintentionally deceitful or untrue. The idea that there are sources you can trust to do your thinking for you is the more dangerous illusion in my opinion, and I’m not convinced that society will be harmed by poking some holes through it.
Post reply on HN