>feels natural if conversation moves at the speed of speech
As someone use to podcast at 3x speech and sapi text to speech at much higher rate, listening to AI at human speech is a chore.
How OpenAI delivers low-latency voice AI at scale
151–160 of 172 posts
Re: How OpenAI delivers low-latency voice AI at scale
#152Very grateful that OpenAI published the article/publicized their usage of Pion[0] a library I work on. If you aren't familiar with WebRTC it's a super fun space. I work on a book WebRTC for the Curious [1] that details how it works. [0] https://github.com/pion/webrtc [1] https://webrtcforthecurious.com
slightly unrelated but what’s with storing the entire codebase in the root directory instead of a nested src folder? It makes getting to the README a lot more difficult
I assume this is why GitHub has the annoying #readme-ov-file slug
Re: How OpenAI delivers low-latency voice AI at scale
#153Earlier quoted context omitted.
I’ve tried this and it says it will but just keeps cutting in. I hate this feature so much.
If anyone has an alternative I’m all ears. This would be a killer feature for me and something I’ve tried to use on cross-country road trips.
If you're setting this up yourself instead of using a lab's built-it speech functionality, you can run a small LLM in parallel, on a local model or small model like Haiku, that acts as a gate for either doing TTS on the response or not. Its only job is to decide if the transcription it receives is of someone being done talking or if that person is likely to still be mid-thought or mid-sentence.
Re: How OpenAI delivers low-latency voice AI at scale
#154I never use the voice mode in the phone app, it's stuck with 4o for some reason. Same with Claude, that uses Haiku. Why can't they use a better model with thinking disabled?
Re: How OpenAI delivers low-latency voice AI at scale
#155Very grateful that OpenAI published the article/publicized their usage of Pion[0] a library I work on. If you aren't familiar with WebRTC it's a super fun space. I work on a book WebRTC for the Curious [1] that details how it works. [0] https://github.com/pion/webrtc [1] https://webrtcforthecurious.com
For those unfamiliar with WebRTC, the Pion FAQ page has a good description:
> WebRTC is a standardized protocol for P2P communication. It allows two peers to exchange media and data. It is encrypted by default, and handles connectivity establishment in many different network conditions. It is supported in browsers, and has multiple out of browser implementations.[0]
Re: How OpenAI delivers low-latency voice AI at scale
#156The low latency is more of a pain point than a good thing, the way they have it implemented. Trying to have a casual conversation with it, as humans we naturally pause, and GPT will take this as you are "done" and start blabbing away. I also suffer from finding the appropriate word I want as I've gotten older and slower, and this fast-voice-gpt just ends up frustrating me more than helping. I have to sit there and th…
Agreed. It’s stressful. I think they need to have an option to adopt a suffix, so they don’t start babbling until there is an “over” followed by a pause like in the old army walkie talkie days.
Re: How OpenAI delivers low-latency voice AI at scale
#157OpenAI uses Go for the networking implementation for the relays and the services, which makes a ton of sense, instead of something immature as TypeScript / Node or whatever. Yet another reason to not consider anything else like that for low-latency networking. Golang (or even Rust and C++) is unmatched for this use-case.
Can golang do zero copy networking nowadays? In the past golang was terrible at this kind of thing due to allocations and copies of all relayed data.
Even Java together with Netty supports zero copy networking... if Go misses that feature I wouldn't be very hard to implement it yourself
Re: How OpenAI delivers low-latency voice AI at scale
#158Earlier quoted context omitted.
I think he’s saying they are doing an insane level of complexity to shave ~100ms off response times in a scenario where that isn’t important and might even be a negative
When GP mentioned reducing conversational latency as a negative that made sense (and should probably be done IMO), it just wasn't the same category of latency the article talks about reducing. I.e. increasing "network latency" just makes the conversation feel more and more out of sync, it doesn't change the rate at which the AI will interrupt ("turn latency") because the latter is based on the duration of the pause i…
> If you meant there is a case where reducing the network latency […] is actually a negative then I'd love to hear more about it
That is exactly what parent did:
> they are doing an insane level of complexity to shave off ~100 ms
The downside is everything they had to do to achieve it, and maintaining that capability going forward, when the product can tolerate much more. It is the definition of premature optimization.
It just maybe isn’t at a level where it is relevant in your argument/decision space in IT.
Re: How OpenAI delivers low-latency voice AI at scale
#159Earlier quoted context omitted.
I think he’s saying they are doing an insane level of complexity to shave ~100ms off response times in a scenario where that isn’t important and might even be a negative
By you want to be able to interject “hold on…” and have it immediately stop talking, when it goes off the rails. And GP is correctly pointing out that the only negative here (silence waiting latency maybe being too low) is tunable separately from the network latency number.
you actually don’t want it to immediately stop because people say things like “hm” “yeh” during machine output. Maybe you say “no” to someone next to you and don’t want to interrupt output.
To confidently interrupt I would want to assert that the user has been speaking for > N time. You could do other things like parse a streaming transcription for keywords but generally it feels like bad UX to me to cut output the second input is detected. Letting the user talk for 1-2s gives a much stronger signal and it isn’t too weird for someone to keep talking for 1.5s after you start.
Re: How OpenAI delivers low-latency voice AI at scale
#160Earlier quoted context omitted.
I’m not a go fanboy, but I do know from other contexts that so-called “fanboy“ behaviour is frequently associated with level-headed supporters getting defensive in the face of imprecise criticism. There’s an oft-repeated pattern where valid specific criticisms morph into broad criticism, which morphs into judgement, which breeds defensiveness, which feeds the criticism. Once you recognise this pattern, you see it eve…
Sure, and there's the near-identical pattern where valid specific criticisms are taken as broad criticism even though they aren't, etc., etc..
Thats the defensive step outlined above.