Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we made a text-to-speech model respond in sub-50 ms

nari-labs.com

Post reply on HN
Print view View original

21–30 of 49 posts

Re: How we made a text-to-speech model respond in sub-50 ms

#21

Post by toebee »

zuzululu wrote: post #13

this is cool but for agent scenarios unless an LLM bakes in the speech tokens directly, the latency is lost to inference, and this is what makes openai's voice model so interesting also sweet spot is under 150ms so the remainder is inference latency turn around, a 50ms turnaround including tts-stt would ofc be the dream that is "this ai agent is indistinguishably present and sentient" area

Qwen3 TTS has input streaming mode: you can stream LLM output into the speech model. So don’t need to wait for a full sentence. We also implement this websocket variant, and it also runs at sub 50 ms.

LLM TTFT is still a big issue, and we might tackle that problem as well.

Re: How we made a text-to-speech model respond in sub-50 ms

#22

Post by toebee »

narrationbox wrote: post #5
toebee wrote: post #2

time-to-first-audio (TTFA) is critical for realtime voice applications. open source implementations (e.g. vLLM-Omni, SGLang-Omni) are often too slow for production and can have issues with realtime playback if you push for lower latency. we wanted to fix that. we optimized qwen3-tts, a popular OSS TTS model, to achieve 34 ms p95 TTFA at 10 requests per second on 1 x H100. we open source the implementation and benchma…

Haven't read the full report yet, just a quick question. Are your numbers for cold start without pre fill or is it after warmed cache?

We do graph capture etc at startup (same as vLLM) but this model variant doesn’t require prefix caching - the prefix is just 10 tokens.

Re: How we made a text-to-speech model respond in sub-50 ms

#23

Post by bt1a »

toebee wrote: post #19
bityard wrote: post #3

Earlier quoted context omitted.

How fast is it on consumer-level hardware?

We got a rtx 4090 handling around 10 concurrent requests at 50 ms TTFA after some config changes / adjustment as it doesn’t have FP8. So this 50 ms TTFA thing is very much possible on consumer hardware.

I'm going to see how this shakes out on my machine with a few 3090s. I see you all are leveraging some custom cuda kernels, so it may not work out of the box on Ampere (30xx) architecture yeah?

Re: How we made a text-to-speech model respond in sub-50 ms

#24

Post by toebee »

bt1a wrote: post #23
toebee wrote: post #19

Earlier quoted context omitted.

We got a rtx 4090 handling around 10 concurrent requests at 50 ms TTFA after some config changes / adjustment as it doesn’t have FP8. So this 50 ms TTFA thing is very much possible on consumer hardware.

I'm going to see how this shakes out on my machine with a few 3090s. I see you all are leveraging some custom cuda kernels, so it may not work out of the box on Ampere (30xx) architecture yeah?

Yep, might need some changes.

Re: How we made a text-to-speech model respond in sub-50 ms

#25

Post by toebee »

nowittyusername wrote: post #10

This is right up my alley as ive been building a local voice agent for a year now. Ive tried many different models and have a custom implementation for omni voice that ive tuned for over many months. Ive never been able to achieve faster then 200ms ttfa for that model at 24 steps, but the reason is .... quality. I find that there is a lot of room for improvement in many tts models out there by a huge margin. But ther…

we continuously compared output to qwen's original implementation and do not see differences in output quality. let us know if it works well for you!

Re: How we made a text-to-speech model respond in sub-50 ms

#26

Post by toebee »

armcat wrote: post #12

Having built my own voice assistant ( https://github.com/acatovic/ova ) and having tried many other services and models, I feel the real win is when this is on-device, and by "on-device" I mean being very inexpensive to run on a phone, and not H100. I've now been using Pocket TTS which is super fast, and also Chatterbox and Fish Audio S2 Pro (on the Mac/PC), I feel we are so close, yet so far. The quality is amazing,…

we haven't tried so can't say for sure, but if optimized for a scenario where the batch size is 1 and max concurrency is 1, it seems possible to get something pretty fast. i'm guessing mobile hardware specific optimizations will be important but we are not experts in that field.

Re: How we made a text-to-speech model respond in sub-50 ms

#28

Post by zuzululu »

toebee wrote: post #21
zuzululu wrote: post #13

this is cool but for agent scenarios unless an LLM bakes in the speech tokens directly, the latency is lost to inference, and this is what makes openai's voice model so interesting also sweet spot is under 150ms so the remainder is inference latency turn around, a 50ms turnaround including tts-stt would ofc be the dream that is "this ai agent is indistinguishably present and sentient" area

Qwen3 TTS has input streaming mode: you can stream LLM output into the speech model. So don’t need to wait for a full sentence. We also implement this websocket variant, and it also runs at sub 50 ms. LLM TTFT is still a big issue, and we might tackle that problem as well.

huh that is crazy fast, demo ?

Re: How we made a text-to-speech model respond in sub-50 ms

#29

Post by dang »

elixirbash wrote: post #15

[flagged]

Can you please not post AI-generated or AI-edited comments to HN? It's not allowed here - see https://news.ycombinator.com/newsguidelines.html#generated and https://news.ycombinator.com/item?id=47340079.

Of course, it's impossible to know for sure what was LLM processed or not, but some of your posts (like this one) have been getting classified that way.

Re: How we made a text-to-speech model respond in sub-50 ms

#30

Post by toebee »

MaxikCZ wrote: post #17

no video demonstration?

will try to record something - in the meantime you can spin up a machine on runpod or modal to quickly test it out.

docker run --rm --gpus all \ -p 8000:8000 \ -e HF_TOKEN \ -e QWEN3_TTS_PROFILE=ttfa \ -v nari-qwen3-tts-cache:/home/nari/.cache \ ghcr.io/nari-labs/nari-qwen3-tts:latest

Post reply on HN