[flagged]
How we made a text-to-speech model respond in sub-50 ms
11–20 of 49 posts
Re: How we made a text-to-speech model respond in sub-50 ms
#12Having built my own voice assistant (https://github.com/acatovic/ova) and having tried many other services and models, I feel the real win is when this is on-device, and by "on-device" I mean being very inexpensive to run on a phone, and not H100. I've now been using Pocket TTS which is super fast, and also Chatterbox and Fish Audio S2 Pro (on the Mac/PC), I feel we are so close, yet so far. The quality is amazing, but can we take this to the next level and make it run on mobile? What would it take?
Re: How we made a text-to-speech model respond in sub-50 ms
#13this is cool but for agent scenarios unless an LLM bakes in the speech tokens directly, the latency is lost to inference, and this is what makes openai's voice model so interesting
also sweet spot is under 150ms so the remainder is inference latency turn around, a 50ms turnaround including tts-stt would ofc be the dream
that is "this ai agent is indistinguishably present and sentient" area
Re: How we made a text-to-speech model respond in sub-50 ms
#14chatgpt responds super fast but says filler words like 'hmm..' 'let me think' and responds later with delay
Isn't ChatGPT benchmaxxing, then ?
Responding "hmm…" isn't actually responding and latency should time to first relevant phoneme.
Re: How we made a text-to-speech model respond in sub-50 ms
#15[flagged]
Re: How we made a text-to-speech model respond in sub-50 ms
#16[dead]
Re: How we made a text-to-speech model respond in sub-50 ms
#17no video demonstration?
Re: How we made a text-to-speech model respond in sub-50 ms
#18time-to-first-audio (TTFA) is critical for realtime voice applications. open source implementations (e.g. vLLM-Omni, SGLang-Omni) are often too slow for production and can have issues with realtime playback if you push for lower latency. we wanted to fix that. we optimized qwen3-tts, a popular OSS TTS model, to achieve 34 ms p95 TTFA at 10 requests per second on 1 x H100. we open source the implementation and benchma…
Hi there! I actually thought your Dia models were amazing and very natural sounding, I haven’t tried qwen 3 tts yet - has your focus shifted away from building your Dia models and shifted more towards hosting and infrastructure?
Hey thank you for your kind words! Yes, we’ve shifted to inference but will also continue doing finetuning etc. on top of open models. Don’t have plans to do pretraining though.
Re: How we made a text-to-speech model respond in sub-50 ms
#19time-to-first-audio (TTFA) is critical for realtime voice applications. open source implementations (e.g. vLLM-Omni, SGLang-Omni) are often too slow for production and can have issues with realtime playback if you push for lower latency. we wanted to fix that. we optimized qwen3-tts, a popular OSS TTS model, to achieve 34 ms p95 TTFA at 10 requests per second on 1 x H100. we open source the implementation and benchma…
How fast is it on consumer-level hardware?
We got a rtx 4090 handling around 10 concurrent requests at 50 ms TTFA after some config changes / adjustment as it doesn’t have FP8. So this 50 ms TTFA thing is very much possible on consumer hardware.
Re: How we made a text-to-speech model respond in sub-50 ms
#20Of course speed is good, but if you don't add an artificial latency (or better, use the extra time for some QA, guardrails, etc...), the model will come off as creepy at best, and the conversation will feel awkward for the user.
Humans have a roughly 200ms auditive processing latency, (audio input to neural response), in conversation we know and account for this, such that if someone responds in 100ms, we interpret that we interrupted them and that their message doesn't come in response to what we just said, but what we said before.
This can be especially relevant in sentences where an interruption would sharply contrast.
"I think murder is bad, but.."
If someone cuts of right after the but, a human would interpret that the interjection responds to the fact that someone thinks murder is bad. Which is starkly different than interrupting someone after they are about to excuse murder.