How we made a text-to-speech model respond in sub-50 ms
1–10 of 49 posts
Re: How we made a text-to-speech model respond in sub-50 ms
#2time-to-first-audio (TTFA) is critical for realtime voice applications. open source implementations (e.g. vLLM-Omni, SGLang-Omni) are often too slow for production and can have issues with realtime playback if you push for lower latency. we wanted to fix that.
we optimized qwen3-tts, a popular OSS TTS model, to achieve 34 ms p95 TTFA at 10 requests per second on 1 x H100. we open source the implementation and benchmark, as well as a breakdown of how it was done.
Re: How we made a text-to-speech model respond in sub-50 ms
#3time-to-first-audio (TTFA) is critical for realtime voice applications. open source implementations (e.g. vLLM-Omni, SGLang-Omni) are often too slow for production and can have issues with realtime playback if you push for lower latency. we wanted to fix that. we optimized qwen3-tts, a popular OSS TTS model, to achieve 34 ms p95 TTFA at 10 requests per second on 1 x H100. we open source the implementation and benchma…
How fast is it on consumer-level hardware?
Re: How we made a text-to-speech model respond in sub-50 ms
#4time-to-first-audio (TTFA) is critical for realtime voice applications. open source implementations (e.g. vLLM-Omni, SGLang-Omni) are often too slow for production and can have issues with realtime playback if you push for lower latency. we wanted to fix that. we optimized qwen3-tts, a popular OSS TTS model, to achieve 34 ms p95 TTFA at 10 requests per second on 1 x H100. we open source the implementation and benchma…
Hi there! I actually thought your Dia models were amazing and very natural sounding, I haven’t tried qwen 3 tts yet - has your focus shifted away from building your Dia models and shifted more towards hosting and infrastructure?
Re: How we made a text-to-speech model respond in sub-50 ms
#5time-to-first-audio (TTFA) is critical for realtime voice applications. open source implementations (e.g. vLLM-Omni, SGLang-Omni) are often too slow for production and can have issues with realtime playback if you push for lower latency. we wanted to fix that. we optimized qwen3-tts, a popular OSS TTS model, to achieve 34 ms p95 TTFA at 10 requests per second on 1 x H100. we open source the implementation and benchma…
Haven't read the full report yet, just a quick question. Are your numbers for cold start without pre fill or is it after warmed cache?
Re: How we made a text-to-speech model respond in sub-50 ms
#6chatgpt responds super fast but says filler words like 'hmm..' 'let me think' and responds later with delay
Re: How we made a text-to-speech model respond in sub-50 ms
#7chatgpt responds super fast but says filler words like 'hmm..' 'let me think' and responds later with delay
its backchanneling frequently makes me laugh to the point of forgetting what i wanted to say. i do like it, it just takes some getting used to, especially since i've been keeping things nice and simple and taking it one step at a time for so long.
Re: How we made a text-to-speech model respond in sub-50 ms
#8chatgpt responds super fast but says filler words like 'hmm..' 'let me think' and responds later with delay
But even then, it's targeting like 300ms not 30ms, right?
Re: How we made a text-to-speech model respond in sub-50 ms
#9GPT‑Realtime‑2 is really weird. Perhaps just because it's bidirectional and now has the failure mode as a possibility, it responds too soon with filler at awkward times, and it's generally overeager. I feel like there was plenty of opportunity to just work on latency engineering like this effort.
Re: How we made a text-to-speech model respond in sub-50 ms
#10This is right up my alley as ive been building a local voice agent for a year now. Ive tried many different models and have a custom implementation for omni voice that ive tuned for over many months. Ive never been able to achieve faster then 200ms ttfa for that model at 24 steps, but the reason is .... quality. I find that there is a lot of room for improvement in many tts models out there by a huge margin. But there is also a quality hard wall that you eventually hit that the tradeoff of faster latency but lower quality is not worth it. When making a really well sounding voice agent quality of voice, cadence, expression, etc... matters a lot. It will be interesting to try this implementation and see if its quality outputs match my expectations, if so great job indeed.