Show HN: Open source framework OpenAI uses for Advanced Voice

Post by gastonmorixe » Sat, Oct 05, 2024, 6:59 PM UTC

russ wrote: post #9

gastonmorixe wrote: post #3

Nice they have many partners on this. I see Azure as well. There is a common consensus that the new Realtime API is not actually using the same Advanced Voice model / engine - or however it works - since at least the TTS part doesn’t seem to be as capable as the one shipped with the official OpenAI app. Any idea on this? Source: https://github.com/openai/openai-realtime-api-beta/issues/2

It's using the same model/engine. I don't have knowledge of the internals, but a different subsystem/set of dedicated resources though for API traffic versus first-party apps. One thing to note is there is no separate TTS-phase here, it's happening internally within GPT-4o, in the Realtime API and Advanced Voice.

Post by pj_mukh » Sat, Oct 05, 2024, 7:20 PM UTC

davidz wrote: post #7

pj_mukh wrote: post #5

Super cool! Didn't realize OpenAI is just using LiveKit. Does the pricing breakdown to be the same as having a OpenAI Advanced Voice socket open the whole time? It's like $9/hr! It would be theoretically cheaper to use this without keeping the advanced voice socket open the whole time and just use the GPT4o streaming service [1] for whenever inference is needed (pay per token) and use livekits other components to do…

Currently it does: all audio is sent to the model. However, we are working on turn detection within the framework, so you won't have to send silence to the model when the user isn't talking. It's a fairly straight forward path to cutting down the cost by ~50%.

Post by russ » Sat, Oct 05, 2024, 7:51 PM UTC

gastonmorixe wrote: post #53

russ wrote: post #17

Earlier quoted context omitted.

There’s a ton of complexity under the “relatively simple use case” when you get to a global, 200M+ user scale.

80% of the times I’m experiencing choppy audio on my iPhone 15 Pro Max (18.1b) on Voice Mode (Standard and Advanced). My internet connection is FTTH and WiFi 7 state of the art router. I wonder if this is because bugs or the crazy load livekit may be going through given the popularity in ChatGPT voice modes right now.

Post by fidotron » Sat, Oct 05, 2024, 9:13 PM UTC

russ wrote: post #57

fidotron wrote: post #48

Earlier quoted context omitted.

Essentially I think the Livekit value is a SFU that works, with signalling, and the SDKs exist. My experience is people radically overstate how hard signalling is, and underestimate SFU complexity, especially with fast failover. In terms of being a higher level API arguably it is doomed to failure, thanks to the madness of the domain. (The part that sticks in my mind is audio device switching on Android.) WebRTC prod…

You really don’t need to know about WebRTC at all when you use LiveKit. That’s largely thanks to the SDKs abstracting away all the complexity. Having good SDKs that work across every platform with consistent APIs is more valuable than the SFU imo. There are other options for SFUs and folks like Signal have rolled their own. Try to get WebRTC running on Apple Vision Pro or tvOS and let me know if that’s no big deal.

Post by fosheezy » Sat, Oct 05, 2024, 10:53 PM UTC

racecar789 wrote: post #31

Imagine being able to tell an app to call the IRS during the day, endure the on-hold wait times, then ask the question to the IRS rep and log the answer. Then deliver the answer when you get home. Or, have the app call a pharmacy every month to refill prescriptions. For some drugs, the pharmacy requires a manual phone call to refill which gets very annoying. So many use cases for this.