Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we made a text-to-speech model respond in sub-50 ms

nari-labs.com

Post reply on HN
Print view View original

31–40 of 49 posts

Re: How we made a text-to-speech model respond in sub-50 ms

#31

Post by toebee »

zuzululu wrote: post #28
toebee wrote: post #21

Earlier quoted context omitted.

Qwen3 TTS has input streaming mode: you can stream LLM output into the speech model. So don’t need to wait for a full sentence. We also implement this websocket variant, and it also runs at sub 50 ms. LLM TTFT is still a big issue, and we might tackle that problem as well.

huh that is crazy fast, demo ?

will try to record a video soon, in the meantime you can grab a h100 from somewhere like runpod or modal and test it out:

docker run --rm --gpus all \ -p 8000:8000 \ -e HF_TOKEN \ -e QWEN3_TTS_PROFILE=ttfa \ -v nari-qwen3-tts-cache:/home/nari/.cache \ ghcr.io/nari-labs/nari-qwen3-tts:latest

Re: How we made a text-to-speech model respond in sub-50 ms

#32

Post by kshmir »

toebee wrote: post #2

time-to-first-audio (TTFA) is critical for realtime voice applications. open source implementations (e.g. vLLM-Omni, SGLang-Omni) are often too slow for production and can have issues with realtime playback if you push for lower latency. we wanted to fix that. we optimized qwen3-tts, a popular OSS TTS model, to achieve 34 ms p95 TTFA at 10 requests per second on 1 x H100. we open source the implementation and benchma…

Thanks for your article I set it in my 5090 server and took some ideas to improve the whisper inference I also have.

Running asr+llm+tts now :)

Re: How we made a text-to-speech model respond in sub-50 ms

#33

Post by sciencesama »

toebee wrote: post #2

time-to-first-audio (TTFA) is critical for realtime voice applications. open source implementations (e.g. vLLM-Omni, SGLang-Omni) are often too slow for production and can have issues with realtime playback if you push for lower latency. we wanted to fix that. we optimized qwen3-tts, a popular OSS TTS model, to achieve 34 ms p95 TTFA at 10 requests per second on 1 x H100. we open source the implementation and benchma…

interesting

Re: How we made a text-to-speech model respond in sub-50 ms

#35

Post by vivzkestrel »

- a bit unrelated but still had to ask

- when recording gaming footage with OBS studio with my mic plugged in, i want to convert my voice to a tts type voice in real time

- Basically I speak in my tone but the output is one of your GPT voices

- Anyone know of a library or plugin that can accomplish this in real time

Re: How we made a text-to-speech model respond in sub-50 ms

#36

Post by pcvetkovski »

Sub-50ms on text-to-speech running on an LLM is commendable.

We recently shipped text-to-speech and speech-to-text support inside Finsight (Maxint). We tapped into the platform’s native speech capabilities, which were integrated with the user’s preferred LLM inference endpoint (including local on-device models). This approach enabled us to eliminate latency and preserve user’s privacy, delivering human-like conversational experience.

Since your approach involves running the model directly, did you run into memory bandwidth contention or audio buffer underruns during bursty generation, when both the LLM and TTS models are executing concurrently?

Re: How we made a text-to-speech model respond in sub-50 ms

#38

Post by rullelito »

toebee wrote: post #2

time-to-first-audio (TTFA) is critical for realtime voice applications. open source implementations (e.g. vLLM-Omni, SGLang-Omni) are often too slow for production and can have issues with realtime playback if you push for lower latency. we wanted to fix that. we optimized qwen3-tts, a popular OSS TTS model, to achieve 34 ms p95 TTFA at 10 requests per second on 1 x H100. we open source the implementation and benchma…

Feels like caching the top 1000 most common beginnings would solve this for 99% of cases?

Re: How we made a text-to-speech model respond in sub-50 ms

#40

Post by totetsu »

toebee wrote: post #2

time-to-first-audio (TTFA) is critical for realtime voice applications. open source implementations (e.g. vLLM-Omni, SGLang-Omni) are often too slow for production and can have issues with realtime playback if you push for lower latency. we wanted to fix that. we optimized qwen3-tts, a popular OSS TTS model, to achieve 34 ms p95 TTFA at 10 requests per second on 1 x H100. we open source the implementation and benchma…

I just tried it out on runpod [1] on the lady of shalott. Its got a funny sing song affect https://pastewaves.com/player/cdc4693a-a818-4d34-8566-f0207d...

that one took maybe 12.677831s to generate..

DNS: 0.003747s Connect: 0.044571s TLS: 0.115428s First byte: 12.677831s Total curl: 26.268993s HTTP status: 200 Downloaded: 7695404 bytes

[1] with this template https://console.runpod.io/hub/template/pahlszv0ul?ref=0xiswp...

Post reply on HN