Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenVoice: Versatile Instant Voice Cloning

arxiv.org

Post reply on HN
Print view View original

141–150 of 207 posts

Re: OpenVoice: Versatile Instant Voice Cloning

#141

Post by refulgentis »

anotherevan wrote: post #128

Is it possible to use this (or Eleven Labs) to generate a voice model to plug into an Android phone's TTS? I have a friend with a paralysed larynx who is often using his phone or a small laptop to type in order to communicate. I know he would love it if it was possible to take old recordings of him speaking and use that to give him back "his" voice, at least in some small measure.

Alas, no (made some contributions to TTS at G, and worked on Android).

iOS has this built in :/ which may bode well, there's no greater Google product manager than "whatever apple just shipped."

I'm doing some xplatform on device inference stuff (see FONNX on GitHub) and it'll be one of 100 items that'll stick on my mind for a while, I hope I find time and I'll try to ping you

Edit: is an Android app with a keyboard and "speak" button that does API calls to eleven labs sufficient for something worth trying?

Re: OpenVoice: Versatile Instant Voice Cloning

#142

Post by abetusk »

paradox460 wrote: post #139
abetusk wrote: post #91

Earlier quoted context omitted.

This is an exceptional use case! Mr Beast talked about translating his videos to other languages to get more reach. This can be done for people with limited budget or just in general so people can watch videos without needing subtitles. I wouldn't be surprised if we saw this incorporated into YT in the near future.

Another really good one would be for RPGs. Instead of clumsy approaches like "Hey Dragonborn" and whatnot, they could actually say your character's name out loud.

Right, and taking it one step further, LLMs in games with voice actors providing the basis for dynamic dialogue that sounds like it's coming from a person.

Re: OpenVoice: Versatile Instant Voice Cloning

#143

Post by grayhatter »

lbrunson wrote: post #124
grayhatter wrote: post #62

Earlier quoted context omitted.

No. The real answer is yes, I could probably come up with some contrived examples, like I lost my voice in a freak LLM accident and now want to clone my old voice. But this doesn't (you don't?) really need a net benefit reason to figure it out and publish it. Because why? I assume, because "this shouldn't exist!" which is just a more palatable wa to phrase "won't someone think of the children". Society doesn't benefi…

By this logic there shouldn’t be regulation on anything, because the bad guys will have it any way. While you can’t make it go away, you can disincentivize propagation and use which can be the difference between thousands of cases of scams/extortions and millions. Until there’s a stronger argument for voice cloning models (talking to a dead loved one is creepy and not a positive argument) then we shouldn’t encourage…

That's correct, I believe it shouldn't be illegal to know anything. Nor do I think science needs any kind of regulation.

Hurting people, lying, that's already illegal.

I think Maybe you misunderstood my argument. My argument isn't that good guy with a voice cloaner is the only thing that can stop a bad guy with a voice cloaner. That's, as you pointed out, stupid. My argument is that no one benefits if how easy it is to make one remains a secret to everyone but the bad guys.

Re: OpenVoice: Versatile Instant Voice Cloning

#144

Post by grayhatter »

johnnyworker wrote: post #77
grayhatter wrote: post #62

Earlier quoted context omitted.

No. The real answer is yes, I could probably come up with some contrived examples, like I lost my voice in a freak LLM accident and now want to clone my old voice. But this doesn't (you don't?) really need a net benefit reason to figure it out and publish it. Because why? I assume, because "this shouldn't exist!" which is just a more palatable wa to phrase "won't someone think of the children". Society doesn't benefi…

> Why does it need a practical reason? To at least give us something as a consolation for all the havoc all sorts of deep fakes will wreak on societies. It's like asking what a knife can be used for other than murder. It's a valid question.

It's a valid question, but not a good one. The implication is something needs a reason to exist. This is a new technology, and just like all new technologies, the fear of it is spreading faster than the understanding.

Scams aren't going away. Will this make it easier to scam some people? Absolutely, so did the internet. I'm not claiming this is anything like the internet. My argument closer to, the reason people get scammed isn't because [thing exists] it's because bad people lie, and kind people trust them. We can all wring our hands in fear over what the new technology might do, or we can solve the problems we care about. Authenticity was hard before this, and it'll be hard after.

Re: OpenVoice: Versatile Instant Voice Cloning

#145

Post by anotherevan »

refulgentis wrote: post #141
anotherevan wrote: post #128

Is it possible to use this (or Eleven Labs) to generate a voice model to plug into an Android phone's TTS? I have a friend with a paralysed larynx who is often using his phone or a small laptop to type in order to communicate. I know he would love it if it was possible to take old recordings of him speaking and use that to give him back "his" voice, at least in some small measure.

Alas, no (made some contributions to TTS at G, and worked on Android). iOS has this built in :/ which may bode well, there's no greater Google product manager than "whatever apple just shipped." I'm doing some xplatform on device inference stuff (see FONNX on GitHub) and it'll be one of 100 items that'll stick on my mind for a while, I hope I find time and I'll try to ping you Edit: is an Android app with a keyboard…

> I'll try to ping you

Thanks. Use the email address in my profile if anything eventuates.

> is an Android app with a keyboard and "speak" button that does API calls to eleven labs sufficient for something worth trying?

Maybe. Obviously something with local processing would be preferred, but it might be an option when internet connectivity is good. Is there such an app?

Re: OpenVoice: Versatile Instant Voice Cloning

#146

Post by echelon »

cchance wrote: post #135
echelon wrote: post #101

Earlier quoted context omitted.

RVC is voice conversion (audio to audio), and it's typically finetuned. This is zero shot TTS. Samples create vector encodings that serve as input to inference. There's no retraining the model unless you want it to generalize or perform better.

It isn't though, people need to read the paper and the comments from the author they aren't actually doing the voice generation they pass the text off to VITS, and then they're sauce is that they are doing tone mapping on that VITS output, so if anything they're a competitor to RVC, it's just that the version they published includes VITS also

Interesting.

Funny enough, a lot of RVC packages are using VITS to do RVC for TTS.

Re: OpenVoice: Versatile Instant Voice Cloning

#147

Post by booleandilemma »

peddling-brink wrote: post #140
IshKebab wrote: post #130

Earlier quoted context omitted.

We understand exactly the implications of that action. There are no implications. Simply downloading a zip from Amazon has zero risk. Even opening an arbitrary zip has essentially zero risk. RCE from opening a zip is obviously a really critical and valuable vulnerability and would not be wasted with a public link. Combine that with the fact that this comes from a voice cloning GitHub repo and the chance of this havin…

I'll be honest. You've put way more thought into this then I did. But in the spirit of hacker news, I'll continue the argument. > There are no implications. Untrue and absolutist. > Simply downloading a zip from Amazon has zero risk. Agreed. > Even opening an arbitrary zip has essentially zero risk. RCE from opening a zip is obviously a really critical and valuable vulnerability and would not be wasted with a public…

People like the parent routinely download all of the random zip files off the web that they can get their mouse cursors on. Nothing is going to stop them.

Re: OpenVoice: Versatile Instant Voice Cloning

#148

Post by anotherevan »

klankbrouwerij wrote: post #137
anotherevan wrote: post #128

Is it possible to use this (or Eleven Labs) to generate a voice model to plug into an Android phone's TTS? I have a friend with a paralysed larynx who is often using his phone or a small laptop to type in order to communicate. I know he would love it if it was possible to take old recordings of him speaking and use that to give him back "his" voice, at least in some small measure.

Your friend can take a look at solutions from Acapela [0], SpeakUnique [1] or VOCALiD [2]. Not sure whether they have a solution for Android though. I recently saw a video from google about a custom voice they created for somebody with ALS but I can't seem to find it online (Does anybody have a link?). Creating custom voices is not yet available on Android though. The latest iOS release (iOS 17) does support creating…

Hi and thanks for the suggestions. Looking through them, it looks like you need to do what they call "voice banking" before you lose your voice. Basically reading a script they provide.

Unfortunately my friend's voice is too far gone for that to be possible. Hoping for something where they can use old recordings to generate a voice.

Re: OpenVoice: Versatile Instant Voice Cloning

#149

Post by uoaei »

dijksterhuis wrote: post #129
z991 wrote: post #75

I commend the authors on making this easy to try! However it doesn't work very well for me for general voice cloning. I read the first paragraph of the wikipedia page on books and had it generate the next sentence. It's obviously computer generated to my ear. Audio sample: https://storage.googleapis.com/dalle-party/sample.mp3 Cloned voice (converted to mp3): https://storage.googleapis.com/dalle-party/output_en_defaul…

> It's obviously computer generated to my ear. From the README Disclaimer This is an open-source implementation that approximates the performance of the internal voice clone technology of myshell.ai. The online version in myshell.ai has better 1) audio quality, 2) voice cloning similarity, 3) speech naturalness and 4) computational efficiency.

So this paper is a thinly veiled ad of myshell.ai's services?

Re: OpenVoice: Versatile Instant Voice Cloning

#150

Post by CaptainFever »

RagnarD wrote: post #106

My first and ongoing thought is that immoral/criminal uses of voice cloning vastly exceed any legitimate ones.

My first thought is anonymity. I can make YouTube videos without needing to use my real voice... while being able to keep my personal inflections and emphasis, something TTS (AI) voices can't do.

Or...! Indie game development. I can learn basic voice acting (to get rid of the cringe), and act out all of my characters using different voices.

Post reply on HN