Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

waxy.org

Post reply on HN
Print view View original

191–200 of 207 posts

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#191

Post by lhl »

modeless wrote: post #186
lhl wrote: post #185

Earlier quoted context omitted.

Just a followup for those interested, inference implementation notes and comparison clip between StyleTTS2, TTS VITS, and XTTS: https://fediverse.randomfoo.net/notice/AaOgprU715gcT5GrZ2

Wow you got it working so fast! I'm still stuck in package manager hell trying to debug a million little issues.

In my post I link to my issue where I outline what I needed to do from a clean mamba env that might help.

Pytorch nightly (I use for cuda-12) doesn't work w Python 3.12, but if you stick w 3.11 or 3.10 you should be ok. Rest was just w/o version numbers if you're on a clean venv should be fine, however there's a bug in the Utils lib that requires a 1-line fix if you're trying to inference (also linked). nltk was the only dependency not listed so not bad compared to most code drops!

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#192

Post by cm2012 »

Koffiepoeder wrote: post #162
cm2012 wrote: post #137

Earlier quoted context omitted.

There's little nothing more important to the happiness of humanity than increased productivity per capita. That sounds crazy but when you think about it it's true.

Well, this is a very one sided view on the world I'd say. From personal experience, I can surely tell you that I was much happier in countries where productivity was lower. The people there are just so much more pure of heart.

It's fine to visit, but in every measure of happiness people in poor countries are more lonely and report worse life satisfaction.

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#193

Post by modeless »

lhl wrote: post #191
modeless wrote: post #186

Earlier quoted context omitted.

Wow you got it working so fast! I'm still stuck in package manager hell trying to debug a million little issues.

In my post I link to my issue where I outline what I needed to do from a clean mamba env that might help. Pytorch nightly (I use for cuda-12) doesn't work w Python 3.12, but if you stick w 3.11 or 3.10 you should be ok. Rest was just w/o version numbers if you're on a clean venv should be fine, however there's a bug in the Utils lib that requires a 1-line fix if you're trying to inference (also linked). nltk was the…

I spent a couple of hours debugging why jupyter's debugger wasn't working right, so not exactly related to the code. I did also find and fix that utils bug you mentioned. But my current issue is that phonemizer won't find espeak even though I set the environment variables that are supposed to work. I'll figure it out eventually...

Thanks for writing up your experience! Good to know it works! And it's fast!

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#194

Post by joenot443 »

tavavex wrote: post #165
joenot443 wrote: post #65

Earlier quoted context omitted.

Something I think we're slowly coming to terms with is that the current generation of techies (the ones who can afford to spend hours upon hours tweaking models and sharing results) really prefer Discord over our Web 2.0 forum type communities like this one. Even reddit on, which is lagging in popularity amongst Gen-Z when compared to Discord or TikTok, you can immediately tell upon reading /r/LocalLLMs that a really…

I kind of disagree? I am gen Z myself, and have used reddit extensively. While I like Discord a lot, I strongly disagree with using it to host content, essentially gating non-members from getting what they want (which is what leads to these communities with ludicrously inflated member counts). And this sentiment definitely isn't just me, a lot of the techie "CS major" people I know lean towards using slightly older s…

That's good to know! Yeah - I shouldn't imply that these preferences are universal or absolute, just trends I've personally noticed.

Glad to hear you and your peers are still posting on the open web!

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#195

Post by BeFlatXIII »

LordDragonfang wrote: post #89
throwaway290 wrote: post #68

Earlier quoted context omitted.

If this community was available for crawling then LLM would crawl it and there would be no value in participating in the community because you can just ask the LLM about all that, no?

If the value your community provides is low enough that it can be effectively replaced by a general purpose LLM, then it should be. The value of a community should be pushing the boundaries of knowledge, not gatekeeping it. C'mon, this is hacker news, what happened to "information should be free"?

> C'mon, this is hacker news, what happened to "information should be free"?

We've had an infestation of "pay me or I won't share" types.

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#196

Post by Bilal_io »

modeless wrote: post #193
lhl wrote: post #191

Earlier quoted context omitted.

In my post I link to my issue where I outline what I needed to do from a clean mamba env that might help. Pytorch nightly (I use for cuda-12) doesn't work w Python 3.12, but if you stick w 3.11 or 3.10 you should be ok. Rest was just w/o version numbers if you're on a clean venv should be fine, however there's a bug in the Utils lib that requires a 1-line fix if you're trying to inference (also linked). nltk was the…

I spent a couple of hours debugging why jupyter's debugger wasn't working right, so not exactly related to the code. I did also find and fix that utils bug you mentioned. But my current issue is that phonemizer won't find espeak even though I set the environment variables that are supposed to work. I'll figure it out eventually... Thanks for writing up your experience! Good to know it works! And it's fast!

Are you on Windows? I've had the issue and was able to fix it by manually adding these system variables:

  PHONEMIZER_ESPEAK_LIBRARY = c:\Program Files\eSpeak NG\libespeak-ng.dll

  PHONEMIZER_ESPEAK_PATH = c:\Program Files\eSpeak NG

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#197

Post by hinkley »

blagie wrote: post #100
hinkley wrote: post #73

> Artifacts aside, it sounds like Michael Jackson doing a Weird Al impression?! Every line has a distinctly “white and nerdy” vibe: it loses any seriousness and edge, exaggerating words for comic effect and enunciating lyrics really clearly so the punchlines can be heard. No, it sounds like someone doing doing an impression of Weird Al doing an impression of Michael Jackson. Someone whose mom told them they were spec…

... they're good enough. I have an accent. If not for that, I'd be a great presenter. If I could translate my voice into a poor Neil deGrasse Tyson, a poor Patrick Steward, a poor Carl Sagan, a poor Morgan Freeman, etc., my presentations would be... better.

If it makes you more comfortable and confident, that is helping you.

This isn't autotune for the spoken word, though. It's not fixing pacing or vocabulary, and in the audio above it isn't even fixing intonation. Yes, a thick German accent will give you away as being of German extraction. But you're also using the word 'since' when Brits and Americans would use 'for', and it's not going to fix that. Any more than it'll fix my french when I make the exact same mistake going the other direction (for=duration vs for=purpose vs for=interval). If I hear 'since one month' you're likely German or Indian. If you ask how long I've been in Marseille you'll know I'm American in about half that time.

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#198

Post by matteoraso »

sublinear wrote: post #157
minimaxir wrote: post #10

This article only covers the musical aspects of AI voice cloning, but there's another dynamic to AI voice cloning that's more complicated: replacing general voice actors in movies/video games/anime (example: https://www.axios.com/2023/07/24/ai-voice-actors-victoria-at... ) Unlike musicians who can't be replaced without significant postprocessing, have enough money to not be impacted by competition, and have legal mus…

I have a different take on this. AI voice is cheaper, but it's also a more boring and generic performance. There is zero progress made towards any sort of creative AI that produces good unique work. The market for this then is small businesses who can't afford a professional voice actor. AI is opening up new markets, not killing the jobs of the truly talented.

>There is zero progress made towards any sort of creative AI that produces good unique work.

It's only been a year. Give it some time and I'm sure AI will have much better results. Right now, you can get some of that unique work by finetuning the AI off of a person's existing portfolio.

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#199

Post by gwern »

techdragon wrote: post #190
gwern wrote: post #189

Earlier quoted context omitted.

Don't expect that to last more than a year or two, assuming it's even still a problem for the best voice-generation AIs. Generating high-quality is the hard problem; generating specific high-quality samples is, by comparison, a lot easier. Remember when Stable Diffusion was released a year ago and one of the big artist copes was "sure, it can generate random images, but it'll never be able to generate the same charac…

The issue is more that I can’t get the equivalent of a slider control to adjust one or more properties of the voice from the AI in real time… like a vocal fry slider to use an example of something most people are capable of deliberately doing when they want to… but the currently available models are pre-trained to sound like the average/median of one specific person (or character) and while I imagine tools will impro…

No, that's easy. We had the equivalent of that in GANs many years ago. If you've never seen GAN editing, here's a quick video: https://www.youtube.com/watch?v=Z1-3JKDh0nI (Background: https://gwern.net/face#reversing-stylegan-to-control-modify-... ) You just classify the latents and then you can edit it. These days, with pretrained models like CLIP, you don't necessarily even need a latent space: you can take a model which has been trained on sound/text descriptions, like AudioCLIP, prompt it with a text like "vocal fry", and then the generated samples are subtly skewed to try to maximize similarity with "vocal fry". You put a slider on that for how much weight/skewing it does, and now you have a slider control to adjust properties of the voice from the AI. If something like this doesn't exist, it's obvious how to do it. (Even the realtime problem is being solved by figuring out how to train diffusion models to do a GAN-like single pass: https://arxiv.org/abs/2309.06380 )

Re: Weird A.I. Yankovic: a cursed deep dive into the world of voice cloning

#200

Post by techdragon »

gwern wrote: post #199
techdragon wrote: post #190

Earlier quoted context omitted.

The issue is more that I can’t get the equivalent of a slider control to adjust one or more properties of the voice from the AI in real time… like a vocal fry slider to use an example of something most people are capable of deliberately doing when they want to… but the currently available models are pre-trained to sound like the average/median of one specific person (or character) and while I imagine tools will impro…

No, that's easy. We had the equivalent of that in GANs many years ago. If you've never seen GAN editing, here's a quick video: https://www.youtube.com/watch?v=Z1-3JKDh0nI (Background: https://gwern.net/face#reversing-stylegan-to-control-modify-... ) You just classify the latents and then you can edit it. These days, with pretrained models like CLIP, you don't necessarily even need a latent space: you can take a model…

I didn’t get to really explore the GAN generation of ML work particularly well since I had no supported hardware (no desire to support the nVidia monopoly on ML work) and refused to blow money on cloud instances I’d probably forget at some point and wind up with a giant bill.

It’s a really different world now I’ve got massive models running on my laptop thanks to Apple Silicon and the unified memory architecture, and the c++ ports of various diffusion image models and several families of large language text models work well on my AMD gpu too… it’s so much easier to participate in the current generation of applied ML work without having to go out of my way to have specific ML supported hardware.

Post reply on HN