so the WebRTC helps with the unreliable network between the mobile clients and the server side. if the application is backend only, would it make sense to use WebRTC or should I go directly to realtime api?
Show HN: Open source framework OpenAI uses for Advanced Voice
21–30 of 63 posts
Re: Show HN: Open source framework OpenAI uses for Advanced Voice
#22Earlier quoted context omitted.
Currently it does: all audio is sent to the model. However, we are working on turn detection within the framework, so you won't have to send silence to the model when the user isn't talking. It's a fairly straight forward path to cutting down the cost by ~50%.
Working on this for an internal tool - detecting no speech has been a PITA so far. Interested to see how you go with this.
Use the voice activity detector we wrote for Home Assistant. It works very well: https://github.com/rhasspy/pymicro-vad
Re: Show HN: Open source framework OpenAI uses for Advanced Voice
#23Is there anyone besides OpenAI working on a speech to speech model? I find it incredibly useful and it's the sole reason that I pay for their service but I do find it very limited. I'd be interested to know if any other groups are doing research on voice models.
Re: Show HN: Open source framework OpenAI uses for Advanced Voice
#24Is there anyone besides OpenAI working on a speech to speech model? I find it incredibly useful and it's the sole reason that I pay for their service but I do find it very limited. I'd be interested to know if any other groups are doing research on voice models.
Yes. Kyutai released an opened model called moshi : https://github.com/kyutai-labs/moshi
There's also llama-omni and a few others. None of them are even close to 4o from an LLM standpoint. But moshi is called a "foundational" model and U'm hopeful it will be enhanced. Also there's not yet support for those on most backends like llamacpp / ollama etc. So I'd say we're in a trough but we'll get there.
Re: Show HN: Open source framework OpenAI uses for Advanced Voice
#25This is really helpful, thanks!
OpenAI hired the ex fractional CTO of LiveKit, who created Pion, a popular WebRTC library/tool.
I'd expect OpenAI to migrate off of LiveKit within 6 months. LiveKit is too expensive. Also, WebRTC is hard, and OpenAI now being a less open company will want to keep improvements to itself.
Not affiliated with any competitors, but I did work at a PaaS company similar to LiveKit but used Websockets instead.
Re: Show HN: Open source framework OpenAI uses for Advanced Voice
#26[deleted]
Re: Show HN: Open source framework OpenAI uses for Advanced Voice
#27This suggests that the AI "brain" receives the user input as text prompt (agent relays the speech prompt to GPT-4o) and generates audio as output (GPT-4o streams speech packets back to the agent).
But when I asked advanced voice mode it said the exact opposite. That it receives input as audio and generates text as output.
Re: Show HN: Open source framework OpenAI uses for Advanced Voice
#28Is there anyone besides OpenAI working on a speech to speech model? I find it incredibly useful and it's the sole reason that I pay for their service but I do find it very limited. I'd be interested to know if any other groups are doing research on voice models.
When I asked advanced voice mode it said that it receives input as audio and generates text as output.
Re: Show HN: Open source framework OpenAI uses for Advanced Voice
#29This suggests that the AI "brain" receives the user input as text prompt (agent relays the speech prompt to GPT-4o) and generates audio as output (GPT-4o streams speech packets back to the agent). But when I asked advanced voice mode it said the exact opposite. That it receives input as audio and generates text as output.
Who did you ask? ChatGPT? Not sure if you understand LLMs but its knowledge is based on the training data, it can't reason about itself, it can only hallucinate in this case, sometimes correctly, most times incorrectly.
Re: Show HN: Open source framework OpenAI uses for Advanced Voice
#30This suggests that the AI "brain" receives the user input as text prompt (agent relays the speech prompt to GPT-4o) and generates audio as output (GPT-4o streams speech packets back to the agent). But when I asked advanced voice mode it said the exact opposite. That it receives input as audio and generates text as output.
Who did you ask? ChatGPT? Not sure if you understand LLMs but its knowledge is based on the training data, it can't reason about itself, it can only hallucinate in this case, sometimes correctly, most times incorrectly.
This is also true for petty much all humans and bypassing this limitation is called enlightenment/self realization.
LLMs don't even have a self so it can never be realized. Just the ego alone exists.