Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we made a text-to-speech model respond in sub-50 ms

nari-labs.com

Post reply on HN
Print view View original

41–49 of 49 posts

Re: How we made a text-to-speech model respond in sub-50 ms

#41

Post by totetsu »

zuzululu wrote: post #28
toebee wrote: post #21

Earlier quoted context omitted.

Qwen3 TTS has input streaming mode: you can stream LLM output into the speech model. So don’t need to wait for a full sentence. We also implement this websocket variant, and it also runs at sub 50 ms. LLM TTFT is still a big issue, and we might tackle that problem as well.

huh that is crazy fast, demo ?

I tried it out on runpod for about $4 using this template. you might need to override the environment varrs and put in a hugging face token for it to pull model. https://console.runpod.io/hub/template/pahlszv0ul?ref=0xiswp...

Re: How we made a text-to-speech model respond in sub-50 ms

#42

Post by totetsu »

totetsu wrote: post #40
toebee wrote: post #2

time-to-first-audio (TTFA) is critical for realtime voice applications. open source implementations (e.g. vLLM-Omni, SGLang-Omni) are often too slow for production and can have issues with realtime playback if you push for lower latency. we wanted to fix that. we optimized qwen3-tts, a popular OSS TTS model, to achieve 34 ms p95 TTFA at 10 requests per second on 1 x H100. we open source the implementation and benchma…

I just tried it out on runpod [1] on the lady of shalott. Its got a funny sing song affect https://pastewaves.com/player/cdc4693a-a818-4d34-8566-f0207d... that one took maybe 12.677831s to generate.. DNS: 0.003747s Connect: 0.044571s TLS: 0.115428s First byte: 12.677831s Total curl: 26.268993s HTTP status: 200 Downloaded: 7695404 bytes [1] with this template https://console.runpod.io/hub/template/pahlszv0ul?ref=0xisw…

Is this kind of failure typical?

On the websockts mode it ends up turning into gibberish. (after "such that") https://pastewaves.com/player/4786c14f-3370-4b71-ab1b-1aa422...

but not in POST mode. https://pastewaves.com/player/aa61a44f-5abb-475c-8df4-b21944...

Re: How we made a text-to-speech model respond in sub-50 ms

#43

Post by techwizrd »

vivzkestrel wrote: post #35

- a bit unrelated but still had to ask - when recording gaming footage with OBS studio with my mic plugged in, i want to convert my voice to a tts type voice in real time - Basically I speak in my tone but the output is one of your GPT voices - Anyone know of a library or plugin that can accomplish this in real time

Like a real-time voice changer? I'm sure some such thing exists. I'm building a real-time voice changer for RPG sessions because I'm terrible at doing voices. Half my NPCs all sound the same!

Re: How we made a text-to-speech model respond in sub-50 ms

#44

Post by vivzkestrel »

techwizrd wrote: post #43
vivzkestrel wrote: post #35

- a bit unrelated but still had to ask - when recording gaming footage with OBS studio with my mic plugged in, i want to convert my voice to a tts type voice in real time - Basically I speak in my tone but the output is one of your GPT voices - Anyone know of a library or plugin that can accomplish this in real time

Like a real-time voice changer? I'm sure some such thing exists. I'm building a real-time voice changer for RPG sessions because I'm terrible at doing voices. Half my NPCs all sound the same!

exactly! you get my point, what kinda whisper based or otherwise library have you managed to find for doing this

Re: How we made a text-to-speech model respond in sub-50 ms

#45

Post by jubilanti »

toebee wrote: post #19
bityard wrote: post #3

Earlier quoted context omitted.

How fast is it on consumer-level hardware?

We got a rtx 4090 handling around 10 concurrent requests at 50 ms TTFA after some config changes / adjustment as it doesn’t have FP8. So this 50 ms TTFA thing is very much possible on consumer hardware.

I can buy a used car for the price of a used RTX 4090 ($2500-$3000), I wouldn't consider it consumer hardware. Prosumer maybe. Almost no consumer needs 10 concurrent requests. How fast does this run on a 3060 or CPU/iGPU only like an Intel Iris or AMD Navi? Or is your priority more commercial cloud services instead of local self hosted?

Re: How we made a text-to-speech model respond in sub-50 ms

#46

Post by ompogUe »

vivzkestrel wrote: post #35

- a bit unrelated but still had to ask - when recording gaming footage with OBS studio with my mic plugged in, i want to convert my voice to a tts type voice in real time - Basically I speak in my tone but the output is one of your GPT voices - Anyone know of a library or plugin that can accomplish this in real time

Does hardware count? Like a Roland VT-4?

Re: How we made a text-to-speech model respond in sub-50 ms

#47

Post by toebee »

totetsu wrote: post #42
totetsu wrote: post #40

Earlier quoted context omitted.

I just tried it out on runpod [1] on the lady of shalott. Its got a funny sing song affect https://pastewaves.com/player/cdc4693a-a818-4d34-8566-f0207d... that one took maybe 12.677831s to generate.. DNS: 0.003747s Connect: 0.044571s TLS: 0.115428s First byte: 12.677831s Total curl: 26.268993s HTTP status: 200 Downloaded: 7695404 bytes [1] with this template https://console.runpod.io/hub/template/pahlszv0ul?ref=0xisw…

Is this kind of failure typical? On the websockts mode it ends up turning into gibberish. (after "such that") https://pastewaves.com/player/4786c14f-3370-4b71-ab1b-1aa422... but not in POST mode. https://pastewaves.com/player/aa61a44f-5abb-475c-8df4-b21944...

will look into this and get it fixed

Re: How we made a text-to-speech model respond in sub-50 ms

#48

Post by totetsu »

toebee wrote: post #47
totetsu wrote: post #42

Earlier quoted context omitted.

Is this kind of failure typical? On the websockts mode it ends up turning into gibberish. (after "such that") https://pastewaves.com/player/4786c14f-3370-4b71-ab1b-1aa422... but not in POST mode. https://pastewaves.com/player/aa61a44f-5abb-475c-8df4-b21944...

will look into this and get it fixed

The source text is just copy pasted from another comment in this page. I have no idea of the expected inputs to this kind of model, if new line chars etc are not expected.

Re: How we made a text-to-speech model respond in sub-50 ms

#49

Post by danamit »

vivzkestrel wrote: post #35

- a bit unrelated but still had to ask - when recording gaming footage with OBS studio with my mic plugged in, i want to convert my voice to a tts type voice in real time - Basically I speak in my tone but the output is one of your GPT voices - Anyone know of a library or plugin that can accomplish this in real time

I feel like accent removers is what you need.
Post reply on HN