Ah, yes, the Furious 7 soundtrack. Definitely something everyone recalls
VibeVoice: A Frontier Open-Source Text-to-Speech Model
51–60 of 177 posts
Re: VibeVoice: A Frontier Open-Source Text-to-Speech Model
#52Unfortunately it's not usable if you're GPU-poor. Couldn't figure out how to run this with an old 1080. I tried VibeVoice-1.5B on my old CPU with torch.float32 and it took 832 seconds to generate a 66 second audio clip. Switching from torch.bfloat16 also introduced some weird sound artifacts in the audio output. If you're GPU-poor the best TTS model I've tried so far is Kokoro.
Someone else mentioned in this thread that you cannot add annotations to the text to control the output. I think for these models to really level up there will have to be an intermediate step that takes your regular text as input and it generates an annotated output, which can be passed to the TTS model. That would give users way more control over the final output, since they would be able to inspect and tweak any details instead of expecting the model to get everything correctly in a single pass.
Re: VibeVoice: A Frontier Open-Source Text-to-Speech Model
#53Ah, yes, the Furious 7 soundtrack. Definitely something everyone recalls
The most popular song of the year from one of the most popular movie franchises that had been in the global news due to the death of its star. Probably the most memorable song from a soundtrack of the century so far.
Re: VibeVoice: A Frontier Open-Source Text-to-Speech Model
#54I read the comments praising these voices as very life like, and went to the page primed to hear very convincing voices. That is not at all what I heard though. The voices are decent, but the intonation is off on almost every phrase, and there is a very clear robotic-sounding modulation. It's generally very impressive compared to many text-to-speech solutions from a few years ago, but for today, I find it very uninsp…
This is close to SOTA emotional performance, at least the female voices.
I trust the human scores in the paper. At least my ear aligns with that figure.
With stuff like this coming out in the open, I wonder if ElevenLabs will maintain its huge ARR lead in the field. I really don't see how they can continue to maintain a lead when their offering is getting trounced by open models.
Re: VibeVoice: A Frontier Open-Source Text-to-Speech Model
#55The first example sounds like a cry for help.
Some of them have tone wobbles which iirc was more common in early TTS models. Looks like the huge context window is really helping out here.
Re: VibeVoice: A Frontier Open-Source Text-to-Speech Model
#56what's the relationship between this work and the recently announced voice models from Microsoft AI? https://microsoft.ai/news/two-new-in-house-models/
Re: VibeVoice: A Frontier Open-Source Text-to-Speech Model
#57I tried some TTS models a while ago, but I noticed that none of them allowed to put markup statements in the text. For example, it would be nice to do something like: Hey look! [enthusiastic] Should we tell the others? Maybe not ... [giggles] etc. In fact, I think this kind of thing is absolutely necessary if you want to use this to replace a voice actor.
Eleven labs has some models with support for that.
Re: VibeVoice: A Frontier Open-Source Text-to-Speech Model
#58The Spontaneous Emotion dailog sounds like a team member venting through LLMs. They could have skipped the singing part, it would be better if the model did not try to do that :)
It did get me to look up the song [1] again though, which is a great stimulator of emotion. The robot singing has a long way to go.
1. https://music.youtube.com/watch?v=xl8thVrlvjI&si=dU6aIJIPWSs...
Re: VibeVoice: A Frontier Open-Source Text-to-Speech Model
#59Wow. I admit that I am not a native speaker, but this looks (or rather, sounds) VERY impressive and I could mistake it for hearing two people talking.
The giveaway is they will never talk over each other. Only one speaker at a time, consistently.
And longer pause between turns than humans would do.
Re: VibeVoice: A Frontier Open-Source Text-to-Speech Model
#60The examples are kind of off-putting. We're definitely in uncanny valley territory here.