Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How OpenAI delivers low-latency voice AI at scale

openai.com

Post reply on HN
Print view View original

141–150 of 172 posts

Re: How OpenAI delivers low-latency voice AI at scale

#141

Post by futureshock »

legohead wrote: post #22

The low latency is more of a pain point than a good thing, the way they have it implemented. Trying to have a casual conversation with it, as humans we naturally pause, and GPT will take this as you are "done" and start blabbing away. I also suffer from finding the appropriate word I want as I've gotten older and slower, and this fast-voice-gpt just ends up frustrating me more than helping. I have to sit there and th…

Reducing the network latency helps with this exactly. OpenAI can make better timed decisions when to begin responding so it'll feel less like an interruption. I've also seen some research on full duplex voice models that handle interruption more like an organic conversation and low latency will help there as well

Re: How OpenAI delivers low-latency voice AI at scale

#142

Post by hobofan »

angry_octet wrote: post #123
jameshush wrote: post #113

Earlier quoted context omitted.

This is more of a VAD/turn detection issue. It's gotten a lot better over the last few years, but it's a hard problem. The extra ~100ms of latency makes a huge difference otherwise, especially when you have use cases that require tool calling that can easily add 500ms+ of latency.

It seems that tool calling shouldn't be 500ms of latency?

If you have tool calling complex enough that it necessitates a higher reasoning level, and you would otherwise have reasoning set to "none", this can easily come out to 500ms.

Re: How OpenAI delivers low-latency voice AI at scale

#143

Post by aleda145 »

Sean-Der wrote: post #99
aleda145 wrote: post #55

Earlier quoted context omitted.

Appreciate you putting the entire book online! I read parts of it a while ago when I had an idea on using webRTC data channels to pass data from databases to browser clients via a CLI. Your book made me understand that it's probably not a great fit for my use case. I just used a centralized control plane and websockets instead. I still feel like there is something fun that we can do with webRTC data channels + zero c…

Thanks for reading it! You can't beat Websockets :) Especially since you have so much tooling/existing stuff that works with HTTP. I have been trying to get a website off the ground that does Datachannels + SQlite in the browser and then users sync between each other. I have gotten distracted so many times though.

Super cool! Please give me a ping if you ever launch that (my email is on my website (in my profile))

Re: How OpenAI delivers low-latency voice AI at scale

#145

Post by TomGarden »

Lucasoato wrote: post #65

Wait a minute... I’m genuinely happy that they are sharing this, but keep in mind that realtime audio model from OpenAI are still stuck with the 4o family in terms of capabilities, sadly. I still find them so useful, such a pity that there’s no real competitor in this segment, having the experience a real conversation has helped me so much in expressing ideas and concepts. Still, it’s worth to keep in mind that these…

Claude voice mode has come a long way! I'd say it's smarter than CGPT AVM last time i tried it.

But personally I've settled on just speaking to the slower models over a custom tts app, I find it being instant was not actually that important, and in the silence I find myself marinating in the discussion more anyway

Re: How OpenAI delivers low-latency voice AI at scale

#146

Post by user_7832 »

dharma1 wrote: post #76
Lucasoato wrote: post #75

Earlier quoted context omitted.

Thanks, I’ll try it, even if my experience wasn’t that great with Google models lately (503s)

Give it a shot, 3.1 live one in AI studio/API and max out reasoning - not the one in Gemini app it’s an older model. Another option is to use pipecat with their VAD and separate STT and TTS and any (fast) LLM of your choice - but it’s more plumbing and not a true speech to speech model

> Give it a shot, 3.1 live one in AI studio/API and max out reasoning - not the one in Gemini app it’s an older model.

Do you know why this is a thing? Despite the app technically being Gemini, I find it quite crap, while the AI Studio thing with thinking is my favorite LLM. Very jarring tbh.

Re: How OpenAI delivers low-latency voice AI at scale

#147

Post by spongebobstoes »

ButlerianJihad wrote: post #126
theptip wrote: post #125

Earlier quoted context omitted.

By you want to be able to interject “hold on…” and have it immediately stop talking, when it goes off the rails. And GP is correctly pointing out that the only negative here (silence waiting latency maybe being too low) is tunable separately from the network latency number.

I want to be able to click the "Stop" button on my earphones remote. I want to be able to interject "woah" or "stop!" or "wait!" or that it would detect that I've inhaled a breath, or that my eyes glazed over. I want the LLM to figure out that every speed setting for its voice output is in "auctioneer" territory rather than "lecturing university professor with tenure and a pension" pacing. But we won't get any of tha…

newer models tend to use fewer thinking tokens to solve the same problems, and is a strong counterexample to your entire comment

Re: How OpenAI delivers low-latency voice AI at scale

#149

Post by shevy-java »

I don't like AI in general, and on youtube there are soooooo many horrible videos with voice AI. Having said that, I did notice AI has actually worked for some hobbyist-maintained games for the most part. Example: BG2EE (Baldur's Gate 2 Enhanced Edition). Yes, this is a forgotten game; and I actually have background music as audio rather than listen to the dialogue, save for testing it, but for the most part it worked here. So for poor-ressource hobbyists, AI is actually not totally useless. For youtube I find only horribly crap examples. I don't watch any AI-involved videos (if I can spot it; so much fake on youtube these days, Google does not realise how AI is killing many old users and visitors here).

Re: How OpenAI delivers low-latency voice AI at scale

#150

Post by pk-voice »

Aeroi wrote: post #6

if anyone is looking to get into this. pipecat is a great open-source repo and community. https://github.com/pipecat-ai/pipecat

If you like Pipecat’s focus on speed, you might also try out our open source, which comes with all the batteries included (knowledge base, telephony/SIP, variables, BYOK any LLM STT TTS, Speech to Speech, etc )

And it's fully OSS- like n8n for voice AI, and you can use it with OpenClaw or Claude code - recently launched MCPs.Github- https://github.com/dograh-hq/dograh, Youtube -https://www.youtube.com/watch?v=sxiSp4JXqws&list=PLDqzGuN7B1...

Post reply on HN