Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How OpenAI delivers low-latency voice AI at scale

openai.com

Post reply on HN
Print view View original

91–100 of 172 posts

Re: How OpenAI delivers low-latency voice AI at scale

#91

Post by wnmurphy »

legohead wrote: post #22

The low latency is more of a pain point than a good thing, the way they have it implemented. Trying to have a casual conversation with it, as humans we naturally pause, and GPT will take this as you are "done" and start blabbing away. I also suffer from finding the appropriate word I want as I've gotten older and slower, and this fast-voice-gpt just ends up frustrating me more than helping. I have to sit there and th…

Strongly agree, some of us like to choose our words more carefully when interacting with an LLM.

I've tried to convey this to OpenAI through various available channels (dev forums, app feedback, etc.).

Grok solves this by having an optional push-to-talk mode, but this is not hands-free and thus more cumbersome than just having a user-configurable variable like seconds_delay_before_sending_voice_input.

Re: How OpenAI delivers low-latency voice AI at scale

#92

Post by [deleted] »

ericmcer wrote: post #83
zamadatix wrote: post #35

Earlier quoted context omitted.

I think these are 2 different layers of "latency". The latency in the article is referring to the transport of the audio stream itself while the latency in your scenario is about how quickly to start responding inside the audio stream.

I think he’s saying they are doing an insane level of complexity to shave ~100ms off response times in a scenario where that isn’t important and might even be a negative

[deleted]

Re: How OpenAI delivers low-latency voice AI at scale

#93

Post by wnmurphy »

janalsncm wrote: post #56
legohead wrote: post #22

The low latency is more of a pain point than a good thing, the way they have it implemented. Trying to have a casual conversation with it, as humans we naturally pause, and GPT will take this as you are "done" and start blabbing away. I also suffer from finding the appropriate word I want as I've gotten older and slower, and this fast-voice-gpt just ends up frustrating me more than helping. I have to sit there and th…

I’ve also experienced this and it’s really annoying. There is this pressure to keep talking if I’m not done with my thought that feels pretty unnatural at least for me. If I’m searching for the right word, I want the opportunity to find it. I think the solution is to handle pauses more intelligently rather than having a higher latency protocol. With low latency you can interrupt and the bot can immediately stop rambl…

100%. I have to hold the floor by filling the space with "ummmmmmmm.... uhhhh...." which inevitably distracts me from my point altogether. Poor user experience.

Re: How OpenAI delivers low-latency voice AI at scale

#94

Post by wnmurphy »

dtran wrote: post #69
legohead wrote: post #22

The low latency is more of a pain point than a good thing, the way they have it implemented. Trying to have a casual conversation with it, as humans we naturally pause, and GPT will take this as you are "done" and start blabbing away. I also suffer from finding the appropriate word I want as I've gotten older and slower, and this fast-voice-gpt just ends up frustrating me more than helping. I have to sit there and th…

This has more to do with Voice Activity Detection (VAD) than the latency described in the article

Exactly. It's a tangent, but clearly a pain point for enough users.

Re: How OpenAI delivers low-latency voice AI at scale

#95

Post by [deleted] »

ericmcer wrote: post #83
zamadatix wrote: post #35

Earlier quoted context omitted.

I think these are 2 different layers of "latency". The latency in the article is referring to the transport of the audio stream itself while the latency in your scenario is about how quickly to start responding inside the audio stream.

I think he’s saying they are doing an insane level of complexity to shave ~100ms off response times in a scenario where that isn’t important and might even be a negative

[deleted]

Re: How OpenAI delivers low-latency voice AI at scale

#96

Post by Sean-Der »

ericmcer wrote: post #81
Sean-Der wrote: post #23

Very grateful that OpenAI published the article/publicized their usage of Pion[0] a library I work on. If you aren't familiar with WebRTC it's a super fun space. I work on a book WebRTC for the Curious [1] that details how it works. [0] https://github.com/pion/webrtc [1] https://webrtcforthecurious.com

I use pion thanks for making it! Curious if you thought their approach was necessary, it seemed like a ton of complexity to reduce one of the faster parts of a voice AI setup. Having a fast model and accurate VAD seems way more important than fine tuning WebRTC transit times.

Thanks for using it :)

I think It’s a case of you improve what you own. The owners of WebRTC servers were aggressively improving their part. They don’t own the inference servers.

Re: How OpenAI delivers low-latency voice AI at scale

#97

Post by zamadatix »

ericmcer wrote: post #83
zamadatix wrote: post #35

Earlier quoted context omitted.

I think these are 2 different layers of "latency". The latency in the article is referring to the transport of the audio stream itself while the latency in your scenario is about how quickly to start responding inside the audio stream.

I think he’s saying they are doing an insane level of complexity to shave ~100ms off response times in a scenario where that isn’t important and might even be a negative

When GP mentioned reducing conversational latency as a negative that made sense (and should probably be done IMO), it just wasn't the same category of latency the article talks about reducing. I.e. increasing "network latency" just makes the conversation feel more and more out of sync, it doesn't change the rate at which the AI will interrupt ("turn latency") because the latter is based on the duration of the pause in the audio stream, not the duration it took to deliver that audio stream.

If you meant there is a case where reducing the network latency at the same delivery reliability for a given audio stream is actually a negative then I'd love to hear more about it as I'm a network guy always in search of an excuse for latency :D.

Re: How OpenAI delivers low-latency voice AI at scale

#98

Post by artdigital »

Lucasoato wrote: post #65

Wait a minute... I’m genuinely happy that they are sharing this, but keep in mind that realtime audio model from OpenAI are still stuck with the 4o family in terms of capabilities, sadly. I still find them so useful, such a pity that there’s no real competitor in this segment, having the experience a real conversation has helped me so much in expressing ideas and concepts. Still, it’s worth to keep in mind that these…

This is what makes their voice mode unusable to me. I can’t stand the way 4o replies and it’s such a big jump in quality from text mode

Re: How OpenAI delivers low-latency voice AI at scale

#99

Post by Sean-Der »

aleda145 wrote: post #55
Sean-Der wrote: post #23

Very grateful that OpenAI published the article/publicized their usage of Pion[0] a library I work on. If you aren't familiar with WebRTC it's a super fun space. I work on a book WebRTC for the Curious [1] that details how it works. [0] https://github.com/pion/webrtc [1] https://webrtcforthecurious.com

Appreciate you putting the entire book online! I read parts of it a while ago when I had an idea on using webRTC data channels to pass data from databases to browser clients via a CLI. Your book made me understand that it's probably not a great fit for my use case. I just used a centralized control plane and websockets instead. I still feel like there is something fun that we can do with webRTC data channels + zero c…

Thanks for reading it!

You can't beat Websockets :) Especially since you have so much tooling/existing stuff that works with HTTP.

I have been trying to get a website off the ground that does Datachannels + SQlite in the browser and then users sync between each other. I have gotten distracted so many times though.

Re: How OpenAI delivers low-latency voice AI at scale

#100

Post by Sean-Der »

lxgr wrote: post #85
dtran wrote: post #69

Earlier quoted context omitted.

This has more to do with Voice Activity Detection (VAD) than the latency described in the article

That seems to be the issue: VAD is insufficient here. Knowing when to respond requires semantic understanding, which probably only the model itself is capable enough. Maybe it’s hard for them to train it to only respond once it seems appropriate to do so?

I am excited for VAD to go away. PersonaPlex totally seems like the future.

However things like 'Call center helpline' turn based actually seems better! You don't want to be interrupted when giving information back and forth (I think?)

Post reply on HN