Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Chatterbox TTS

github.com

Post reply on HN
Print view View original

31–40 of 197 posts

Re: Chatterbox TTS

#31

Post by vunderba »

ianbicking wrote: post #13
Mizza wrote: post #8

Demos here: https://resemble-ai.github.io/chatterbox_demopage/ (not mine) This is a good release if they're not too cherry picked! I say this every time it comes up, and it's not as sexy to work on, but in my experiments voice AI is really held back by transcription, not TTS. Unless that's changed recently.

FWIW in my recent experience I've found LLMs are very good at reading through the transcription errors (I've yet to experiment with giving the LLM alternate transcriptions or confidence levels, but I bet they could make good use of that too)

Pairing speech recognition with a LLM acting as a post-processor is a pretty good approach.

I put together a script a while back which converts any passed audio file (wav, mp3, etc.), normalizes the audio, passes it to ggerganov whisper for transcription, and then forwards to an LLM to clean the text. I've used it with a pretty high rate of success on some of my very old and poorly recorded voice dictation recordings from over a decade ago.

Public gist in case anyone finds it useful:

https://gist.github.com/scpedicini/455409fe7656d3cca8959c123...

Re: Chatterbox TTS

#33

Post by cyanf »

az226 wrote: post #22

How does one train a TTS model with an LLM backbone? Practically, how does this work?

you use a neural audio codec to encode audio into codebooks

then you could treat the codebook entries as tokens and treat audio generation as a next token prediction task

you then take the codebook entries generated and run it through the codec’s decoder and yield audio

it works surprisingly well

speech text models (tts model with an llm as backbone) is the current meta

Re: Chatterbox TTS

#34

Post by vunderba »

abraxas wrote: post #7

Are these things good enough to narrate a book convincingly or does the voice lose coherence after a few paragraphs being spoken?

Most of these TTS systems tend to fall apart the longer the text - it's a good idea to just wrap any longform text into separate paragraph segmented batches and then stitch them back together again at the end.

I've also found that if your one-shot sample wave isn't really clean that sometimes Chatterbox produces random unholy whooshing sounds at the end of the generated audio which is an added bonus if you're recording Dante's Inferno.

Re: Chatterbox TTS

#35

Post by teraflop »

> Every audio file generated by Chatterbox includes Resemble AI's Perth (Perceptual Threshold) Watermarker - imperceptible neural watermarks that survive MP3 compression, audio editing, and common manipulations while maintaining nearly 100% detection accuracy.

Am I misunderstanding, or can you trivially disable the watermark by simply commenting out the call to the apply_watermark function in tts.py? https://github.com/resemble-ai/chatterbox/blob/master/src/ch...

I thought the point of this sort of watermark was that it was embedded somehow in the model weights, so that it couldn't easily be separated out. If you're going to release an open-source model that adds a watermark as a separate post-processing step, then why bother with the watermark at all?

Re: Chatterbox TTS

#36

Post by Tokumei-no-hito »

vunderba wrote: post #31
ianbicking wrote: post #13

Earlier quoted context omitted.

FWIW in my recent experience I've found LLMs are very good at reading through the transcription errors (I've yet to experiment with giving the LLM alternate transcriptions or confidence levels, but I bet they could make good use of that too)

Pairing speech recognition with a LLM acting as a post-processor is a pretty good approach. I put together a script a while back which converts any passed audio file (wav, mp3, etc.), normalizes the audio, passes it to ggerganov whisper for transcription, and then forwards to an LLM to clean the text. I've used it with a pretty high rate of success on some of my very old and poorly recorded voice dictation recordings…

thanks for sharing. are some local models better than others? can small models work well or do you want 8B+?

Re: Chatterbox TTS

#37

Post by jchw »

teraflop wrote: post #35

> Every audio file generated by Chatterbox includes Resemble AI's Perth (Perceptual Threshold) Watermarker - imperceptible neural watermarks that survive MP3 compression, audio editing, and common manipulations while maintaining nearly 100% detection accuracy. Am I misunderstanding, or can you trivially disable the watermark by simply commenting out the call to the apply_watermark function in tts.py? https://github.c…

Possibly a sort of CYA gesture, kinda like how original Stable Diffusion had a content filter IIRC. Could also just be to prevent people from accidentally getting peanut butter in the toothpaste WRT training data, too.

Re: Chatterbox TTS

#39

Post by andy_xor_andrew »

in my experience, TTS has been a "pick two" situation:

- fast / cheap to run

- can clone voices

- sounds super realistic

from what I can tell, Chatterbox is the first that apparently lets you pick 3! (have not tried it myself yet, this is just what I can deduce)

Re: Chatterbox TTS

#40

Post by lukax »

Mizza wrote: post #8

Demos here: https://resemble-ai.github.io/chatterbox_demopage/ (not mine) This is a good release if they're not too cherry picked! I say this every time it comes up, and it's not as sexy to work on, but in my experiments voice AI is really held back by transcription, not TTS. Unless that's changed recently.

[flagged]
Post reply on HN