StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

Thanks, fascinating insights. I think an everything-to-everything multimodal model could work if it's big enough because of transfer learning (but then there are latency issues), and so could a refined system built on LLMs/LMMs with TTS (like what you are using), but I haven't seen any good research on audio-to-audio language models. My suspicion is that that would take a lot of compute, much more than text, and that the amount of semantically meaningful accessible data might be much lower as well. And if you do manage to get to the same level of quality as text, what is latency like then? Not 100% sure, just intuitions, but I doubt it's great.

I like the idea of an RL predictor for interruption timing, although I think it might struggle with factual-correction interruptions. It could be a good way to make a very fast system, and if latency on the rest of the system is low enough you could probably start slipping in your "Of course", "Yeah, I agree", and "It was in March, but yeah" for truly natural speech. If latency is low you could just use the RL system to find opportunities to interrupt, give them to the LLM/LMM, and it decides how to interrupt, all the way from "mhm", to "Yep, sounds good to me", to "Not quite, it was the 3rd entry, but yeah otherwise it makes sense", to "Actually can I quickly jump on that? I just wanted to quickly [make a point]/[ask a question] about [some thing that requires exploration before the conversation continues]".

Tuning a system like this would be the most annoying activity in human history, but something like this has to be achieved for truly natural conversation so we gotta do it lol.

It threw a python exception for me and didn't generate speech

meh this is not that good. Sounds quite boring.

> I'm half kidding, I get what you mean

I mean yeah, I'm not saying that they shouldn't be able to use the name. There's no control of "StyleX" but it certainly is a poor choice that can lead to confusion. That's all I'm getting at. 100% this is an opinion (would be insane if believed to be anything else).

I don't think it is just a "sub-field" as you mention and it definitely isn't like StyleGAN isn't known by nearly every person that learns ML (I have seen very few courses that do not mention it, but those tend to be ones that don't discuss generation at all). StyleGAN is one of the most well known models that exist. Up there with GPT, YOLO, and ViT. Realistically we use these names as a style of model now rather than the actual original model themselves (or somewhat interchangeably).

The original StyleTTS's abstract has the line

> Here, we propose StyleTTS, a style-based generative model for parallel TTS

And I certainly would not blame anyone for thinking "Oh, they're using a StyleGAN". That's all I'm saying. Their style encoder looks nothing like the StyleGAN's style encoder. It looks a bit closer to the synthesis network but that's just because they're using Leaky ReLUs and AdaIN, but like we said before, that's not really a StyleGAN specific thing. There are also other parts we could say look similar but they are pretty generic sections that I wouldn't particularly think uniquely pertains to StyleGAN architectures (or StyleDiffusion ones that do make this callback).

It other words, it's like naming something iX. Sure, Apple doesn't have complete control over a leading letter but I also understand Apple's claim that such a naming pattern can confuse people. Certainly a name collision. Hell, I'll say that the authors that made this paper knew what they were doing https://arxiv.org/abs/2212.01452

I just think they can come up with a better name that has worse chance of collision. It's not like StyleTTS is a particularly creative name or even that apt of a description either. Names are important because they do mean things. You may think it is not a poor choice of naming and that's okay too. But we also work in different fields too and I'd argue that research papers are aimed at other researchers, where I would be surprised if anyone works in generations (image, voice, language, data, whatever) is not well aware of the Style based networks. Because I can use that sentence and it make sense to most ML people.