Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT unexpectedly began speaking in a user's cloned voice during testing

arstechnica.com

Post reply on HN
Print view View original

21–30 of 164 posts

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#21

Post by ec109685 »

klyrs wrote: post #15
peddling-brink wrote: post #6

So Open AI has the capability to deep fake any of its users that use the voice chat capability? Yikes.

Did anybody with a passing familiarity with the technology not know this? There's a reason people don't trust their IP to public chat assistants, ya know. It's because they're remorselessly siphoning our data for training purposes. And you've agreed to let them use your inputs for anything they please.

That’s not what is happening here. Instead, the user’s voice output during the session is used to generate later sound by the model accidentally.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#22

Post by xyst »

Now all we need to get is a 3D printer that can print a mask of anybody’s face and now we can have MI style operations

https://youtu.be/v1Y4CubBi60 (5:30)

Imagine a world where world dictators are replaced rather than killed. Rollback dictatorship over years, install democratic process, then magically commit seppuku in a plane crash.

Brilliant. What could go wrong? /s

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#23

Post by xyst »

defamation wrote: post #20
throw101010 wrote: post #14

Earlier quoted context omitted.

A couple seconds? Which project does this?

every single voice cloning project I'm sure you could have used google https://elevenlabs.io/app/voice-lab https://app.resemble.ai/users/sign_in https://github.com/neonbjb/tortoise-tts https://coqui.ai/blog/tts/open_xtts it was even possible 5 years ago https://github.com/CorentinJ/Real-Time-Voice-Cloning

Mission Impossible 3 was only the proof of concept

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#25

Post by grugagag »

throw101010 wrote: post #14
cal85 wrote: post #13

Earlier quoted context omitted.

Anyone can clone a voice with access to a couple of seconds of it.

A couple seconds? Which project does this?

Cloning voice signature or timbre may need a bit more for a good quality. Then there are idiosyncracies in one’s voice. In addition to that, there are tiny verbal tics, expressions, cadence, feel, and some more to be able to say you have properly cloned someone’s voice. The two second sample is like a shallow clone of sorts and is indeed vector space.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#26

Post by BoredPositron »

grugagag wrote: post #25
throw101010 wrote: post #14

Earlier quoted context omitted.

A couple seconds? Which project does this?

Cloning voice signature or timbre may need a bit more for a good quality. Then there are idiosyncracies in one’s voice. In addition to that, there are tiny verbal tics, expressions, cadence, feel, and some more to be able to say you have properly cloned someone’s voice. The two second sample is like a shallow clone of sorts and is indeed vector space.

The last sentence is hilarious. What do you think "properly" cloned voices are? Not every model is few shot and not every model relies on their training set for paralanguage anymore. Easiest way to try it out is properly the pro voice cloning from elevenlabs.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#27

Post by skybrian »

It’s “unexpected” because their early training didn’t get rid of it as well as they hoped. LLM’s are good at detecting patterns and like to continue the pattern. They’re starting with autocomplete for voice and training it to do something else.

For now, it’s fairly harmless since it’s only a blooper in a lab, but there will likely be open-weights versions of this sort of thing eventually. And there will probably be people who argue that it’s a good thing, somehow.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#28

Post by dylan604 »

surfingdino wrote: post #24

Someone needs to tell OpenAI marketing that the mood is changing and creepy features like this one may be used against AI.

Up until the singularity, that's fine. After that, it won't care about the moods of the hairless apes.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#29

Post by bee_rider »

mjburgess wrote: post #19
mitthrowaway2 wrote: post #16

Earlier quoted context omitted.

We'll have to gradually get used to the notion that a person's voice, like so many other things we once thought of as intimately personal, is just a coordinate in a high-dimensional vector space.

Those are not contradictions.

But it is unintuitive to those of us who are from the past.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#30

Post by trescenzi »

mjburgess wrote: post #19
mitthrowaway2 wrote: post #16

Earlier quoted context omitted.

We'll have to gradually get used to the notion that a person's voice, like so many other things we once thought of as intimately personal, is just a coordinate in a high-dimensional vector space.

Those are not contradictions.

One of the common bad takes is that since personal things can be distilled to “just math” they are no longer personal or valuable in the sort of way we value personal things. Increasing our understanding of how the world works shouldn’t devalue the world.

To put it another way, people having souls is not the only reason to treat people like people. Or as Dr Seuss says: A person’s a person no matter how small.

Post reply on HN