Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

github.com

Post reply on HN
Print view View original

161–170 of 245 posts

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#161

Post by modeless »

Sean-Der wrote: post #158
modeless wrote: post #153

Earlier quoted context omitted.

I plan to change the audio input to use WebRTC, then I get echo cancellation and network transparency for free. Although dealing with WebRTC is a headache harder than doing the AI parts.

What do you find hard about WebRTC? I would love to help. Would even code up a prototype if you wanted :)

For starters, every WebRTC demo I've tried has at least 400ms of round trip latency even on a loopback connection. Shoot me an email if you know WebRTC, would be good to chat with someone who knows stuff!

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#162

Post by refulgentis »

IshKebab wrote: post #115
refulgentis wrote: post #92

Earlier quoted context omitted.

It imposes a cost on others when when you makes false claims like I said or felt the question was unreasonable. I didn't and don't. It is a hard question to understand and an interesting mind-bender to answer. Less policing of the metacontext and more focusing on the discussion at hand will help ensure there's interlocutors around to, at the very least, continue policing.

Sorry but it was pretty obvious what he meant.

It's not, at all.

He could have meant speed, text, audio, words, or phonemes, with least probably images.

He probably didn't mean phonemes or he wouldn't be asking.

He probably didn't mean arbitrarily slicing 'real' audio and stitching on fake audio - he made repeated references to a video game.

He probably didn't mean inpainting and outpainting imagery, even though he made reference to a video game, because its an audio model.

Thank you for explaining I deserve to get downvoted through the floor multiple times for asking a question because it's "obvious". Maybe you can explain to the rest of the class what he meant then? If it was obviously phonemes, will you then advocate for them being downvoted through the floor since the answer was obvious? Or is it only people who assume good faith and ask what they meant who deserve downvotes?

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#163

Post by eigenvalue »

modeless wrote: post #149
eigenvalue wrote: post #137

Earlier quoted context omitted.

Tried it but it seems it only works with Cuda 11 and I have 12 installed. Not really willing to potentially screw up my Cuda environment to try it.

Thanks for trying, what error message did you get? It works without CUDA installed at all on my test machine.

  Process Process-2:
  Traceback (most recent call last):
    File "multiprocessing\process.py", line 314, in _bootstrap
    File "multiprocessing\process.py", line 108, in run
    File "chirp.py", line 126, in whisper_process
    File "chirp.py", line 126, in 
    File "faster_whisper\transcribe.py", line 426, in generate_segments
    File "faster_whisper\transcribe.py", line 610, in encode
  RuntimeError: Library cublas64_11.dll is not found or cannot be loaded
  tts initialized

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#164

Post by modeless »

visarga wrote: post #121

Yes, please integrate it with Mistral and Whisper. This has got to get into the LLM frontends.

Done: https://apps.microsoft.com/detail/9NC624PBFGB7

It's mostly just a demo for now and a little bit janky but it's fun to chat with and you can see the promise for 100% local voice AI in the future.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#165

Post by modeless »

eigenvalue wrote: post #163
modeless wrote: post #149

Earlier quoted context omitted.

Thanks for trying, what error message did you get? It works without CUDA installed at all on my test machine.

Process Process-2: Traceback (most recent call last): File "multiprocessing\process.py", line 314, in _bootstrap File "multiprocessing\process.py", line 108, in run File "chirp.py", line 126, in whisper_process File "chirp.py", line 126, in File "faster_whisper\transcribe.py", line 426, in generate_segments File "faster_whisper\transcribe.py", line 610, in encode RuntimeError: Library cublas64_11.dll is not found or…

Hmm, the dll is included in the app package but maybe there is a conflict with other installed DLLs on some machines. When releasing PC software I always expect this type of issue unfortunately. I plan to move away from faster_whisper which may fix this.

I have to say that the Python ecosystem is just awful for distribution purposes and I spent a lot longer on packaging issues than I did on the actual AI parts. And clearly didn't find all of the issues :)

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#166

Post by taneq »

modeless wrote: post #108
lucubratory wrote: post #104

Earlier quoted context omitted.

How hard on your end does the task of making the chatbot converse naturally look? Specifically I'm thinking about interruptions, if it's talking too long I would like to be able to start talking and interrupt it like in a normal conversation, or if I'm saying something it could quickly interject something. Once you've got the extremely high speed, theoretically faster than real time, you can start doing that stuff ri…

Yes, I implemented the ability to interrupt the chatbot while it is talking. It wasn't too hard, although it does require you to wear headphones so the bot doesn't hear itself and get interrupted. The other way around (bot interrupting the user) is hard. Currently the bot starts processing a response after every word that the voice recognition outputs, to reduce latency. When new words come in before the response is…

Deciding when someone is done speaking is hard to do well and impossible to do perfectly. Some people finish speaking, then think of something else to say and pretend they were still talking.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#167

Post by modeless »

taneq wrote: post #166
modeless wrote: post #108

Earlier quoted context omitted.

Yes, I implemented the ability to interrupt the chatbot while it is talking. It wasn't too hard, although it does require you to wear headphones so the bot doesn't hear itself and get interrupted. The other way around (bot interrupting the user) is hard. Currently the bot starts processing a response after every word that the voice recognition outputs, to reduce latency. When new words come in before the response is…

Deciding when someone is done speaking is hard to do well and impossible to do perfectly. Some people finish speaking, then think of something else to say and pretend they were still talking.

True, perfection isn't achievable but human level performance is all you need and it may be possible to do better than that.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#168

Post by taneq »

huac wrote: post #76
kelseyfrog wrote: post #30

Earlier quoted context omitted.

> inferences at up to 15-95X (!) RT on my 4090 That's incredible! Are infill and outpainting equivalents possible? Super-RT TTS at this level of quality opens up a diverse array of uses esp for indie/experimental gamedev that I'm excited for.

It is theoretically possible to train a model that, given some speech, attempts to continue the speech, e.g. Spectron: https://michelleramanovich.github.io/spectron/spectron/ . Similarly, it is possible to train a model to edit the content, a la Voicebox: https://voicebox.metademolab.com/edit.html .

Great. :P

Me: Won’t it be great when AI can-

Computer: Finish your sentences for you? OMG that’s exactly what I was thinking!

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#169

Post by eigenvalue »

modeless wrote: post #165
eigenvalue wrote: post #163

Earlier quoted context omitted.

Process Process-2: Traceback (most recent call last): File "multiprocessing\process.py", line 314, in _bootstrap File "multiprocessing\process.py", line 108, in run File "chirp.py", line 126, in whisper_process File "chirp.py", line 126, in File "faster_whisper\transcribe.py", line 426, in generate_segments File "faster_whisper\transcribe.py", line 610, in encode RuntimeError: Library cublas64_11.dll is not found or…

Hmm, the dll is included in the app package but maybe there is a conflict with other installed DLLs on some machines. When releasing PC software I always expect this type of issue unfortunately. I plan to move away from faster_whisper which may fix this. I have to say that the Python ecosystem is just awful for distribution purposes and I spent a lot longer on packaging issues than I did on the actual AI parts. And c…

Agree completely. But in this case the fault is with CUDA which never ever works without a struggle. It’s insane how hard it is to get stuff that works cross-platform without a lot of work using CUDA. Even PyTorch has an awkward way of dealing with it and they have more resources to figure it out than just about anyone.

Re: StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

#170

Post by aik »

modeless wrote: post #101

I made a 100% local voice chatbot using StyleTTS2 and other open source pieces (Whisper and OpenHermes2-Mistral-7B). It responds so much faster than ChatGPT. You can have a real conversation with it instead of the stilted Siri-style interaction you have with other voice assistants. Fun to play with! Anyone who has a Windows gaming PC with a 12 GB Nvidia GPU (tested on 3060 12GB) can install and converse with StyleTTS…

Hey modeless. Love it. Is your project open source by any chance? Would love to see it.
Post reply on HN