Live data from Hacker News

ChatGPT unexpectedly began speaking in a user's cloned voice during testing

arstechnica.com

31–40 of 164 posts

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#31

Earlier quoted context omitted.

A couple seconds? Which project does this?

Cloning voice signature or timbre may need a bit more for a good quality. Then there are idiosyncracies in one’s voice. In addition to that, there are tiny verbal tics, expressions, cadence, feel, and some more to be able to say you have properly cloned someone’s voice. The two second sample is like a shallow clone of sorts and is indeed vector space.

[deleted]

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#32

Earlier quoted context omitted.

Cloning voice signature or timbre may need a bit more for a good quality. Then there are idiosyncracies in one’s voice. In addition to that, there are tiny verbal tics, expressions, cadence, feel, and some more to be able to say you have properly cloned someone’s voice. The two second sample is like a shallow clone of sorts and is indeed vector space.

The last sentence is hilarious. What do you think "properly" cloned voices are? Not every model is few shot and not every model relies on their training set for paralanguage anymore. Easiest way to try it out is properly the pro voice cloning from elevenlabs.

"Expressions" in particular is about choice of words. A few seconds definitely isn't enough to duplicate that.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#33

This problem appeared during pre-release testing and has since been solved post-generation using an output classifier that verifies responses, according to the system card release. It was predictable that someone would spin this into a black mirror-esque clickbait story.

We built the torment nexus and then configured it not to use any of the torment functionality that showed up in testing. It was predictable that someone would turn this into a “don’t build the torment nexus”-esque clickbait story.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#34
post #13

Earlier quoted context omitted.

Anyone can clone a voice with access to a couple of seconds of it.

We'll have to gradually get used to the notion that a person's voice, like so many other things we once thought of as intimately personal, is just a coordinate in a high-dimensional vector space.

> is just a coordinate in a high-dimensional vector space

Everything is, including types of personality, physiognomy and demographics. Anything that can be related to anything else is embeddable.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#35
post #13

Earlier quoted context omitted.

Anyone can clone a voice with access to a couple of seconds of it.

We'll have to gradually get used to the notion that a person's voice, like so many other things we once thought of as intimately personal, is just a coordinate in a high-dimensional vector space.

Even simpler, mathematically anything computable is a natural number.

That doesn’t remove privacy and other rights.

https://en.m.wikipedia.org/wiki/G%C3%B6del_numbering

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#36

Earlier quoted context omitted.

Those are not contradictions.

One of the common bad takes is that since personal things can be distilled to “just math” they are no longer personal or valuable in the sort of way we value personal things. Increasing our understanding of how the world works shouldn’t devalue the world. To put it another way, people having souls is not the only reason to treat people like people. Or as Dr Seuss says: A person’s a person no matter how small.

But I do think there are some psychological hurdles we’ll have to overcome, as it becomes possible to mechanically copy individual personal aspects. A person’s a person no matter how small, but we aren’t all used to thinking of ourselves as very small.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#37

This problem appeared during pre-release testing and has since been solved post-generation using an output classifier that verifies responses, according to the system card release. It was predictable that someone would spin this into a black mirror-esque clickbait story.

I am absolutely baffled how you don't see the contradiction in your 2 sentences lol

What contradiction?

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#38

It’s “unexpected” because their early training didn’t get rid of it as well as they hoped. LLM’s are good at detecting patterns and like to continue the pattern. They’re starting with autocomplete for voice and training it to do something else. For now, it’s fairly harmless since it’s only a blooper in a lab, but there will likely be open-weights versions of this sort of thing eventually. And there will probably be p…

Can someone pleasee convince why i shouldn't be absolutely shit out of my mind cynical about this innovation? we are literally seeing the downfall of trust in society. and no, i dont believe i am exaggerating

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#39
post #32

Earlier quoted context omitted.

The last sentence is hilarious. What do you think "properly" cloned voices are? Not every model is few shot and not every model relies on their training set for paralanguage anymore. Easiest way to try it out is properly the pro voice cloning from elevenlabs.

"Expressions" in particular is about choice of words. A few seconds definitely isn't enough to duplicate that.

The choice of words is usually yours for a tts model.

Re: ChatGPT unexpectedly began speaking in a user's cloned voice during testing

#40
post #8

This problem appeared during pre-release testing and has since been solved post-generation using an output classifier that verifies responses, according to the system card release. It was predictable that someone would spin this into a black mirror-esque clickbait story.

The capability is real even if it wont happen with current model censorship. Just have to wait until we get an open source version, I bet Meta is working on one right now.

Even the FOSS models will be "censored". Icky implications aside, a voice assistant that starts making up its own prompts instead of listening to you is not a useful product.
Post reply on HN