Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How OpenAI delivers low-latency voice AI at scale

openai.com

Post reply on HN
Print view View original

71–80 of 172 posts

Re: How OpenAI delivers low-latency voice AI at scale

#71

Post by altmanaltman »

junon wrote: post #63
a456463 wrote: post #48

Earlier quoted context omitted.

This is valid criticism. Go fanbois don't like listening to any go criticism. They were all like who needs templates in go. and now go has templates. To me go code looks like somebody vomitted stuff in the root dir and i have to wade through that every time. No namespacing. nothing

I don't like go as a personal preference but reducing them to "fanboys" is a bit reductive. I'm sure the same could be said about your own favorite language.

Is it reductive when its describing a group of people that like something and refusing to hear any ill of it? The comment wasn't shade at people using the language in general.

And you're right, fanboys are in every language. But resorting to changing the argument by whataboutism is a bit reductive.

Re: How OpenAI delivers low-latency voice AI at scale

#72

Post by haaz »

Sean-Der wrote: post #23

Very grateful that OpenAI published the article/publicized their usage of Pion[0] a library I work on. If you aren't familiar with WebRTC it's a super fun space. I work on a book WebRTC for the Curious [1] that details how it works. [0] https://github.com/pion/webrtc [1] https://webrtcforthecurious.com

Only a software dev would start their referencing at 0 lol

Re: How OpenAI delivers low-latency voice AI at scale

#73

Post by vorticalbox »

haaz wrote: post #72
Sean-Der wrote: post #23

Very grateful that OpenAI published the article/publicized their usage of Pion[0] a library I work on. If you aren't familiar with WebRTC it's a super fun space. I work on a book WebRTC for the Curious [1] that details how it works. [0] https://github.com/pion/webrtc [1] https://webrtcforthecurious.com

Only a software dev would start their referencing at 0 lol

I do this too I never made the connection.

Re: How OpenAI delivers low-latency voice AI at scale

#74

Post by dharma1 »

Lucasoato wrote: post #65

Wait a minute... I’m genuinely happy that they are sharing this, but keep in mind that realtime audio model from OpenAI are still stuck with the 4o family in terms of capabilities, sadly. I still find them so useful, such a pity that there’s no real competitor in this segment, having the experience a real conversation has helped me so much in expressing ideas and concepts. Still, it’s worth to keep in mind that these…

Yes the voice part of OpenAI realtime/voice mode is great but it’s pretty dumb compared to newer models and often gets stuck repeating itself.

Google’s Gemini flash live 3.1 is better, especially used via the API - it can do tool calling (including to other, even smarter LLMs if you set it up yourself), you can set the reasoning level (even high is still close enough to realtime) and it can ground answers in google search. I love bidirectional voice and right now it’s probably the best option. You can try it in AI studio

Re: How OpenAI delivers low-latency voice AI at scale

#75

Post by Lucasoato »

dharma1 wrote: post #74
Lucasoato wrote: post #65

Wait a minute... I’m genuinely happy that they are sharing this, but keep in mind that realtime audio model from OpenAI are still stuck with the 4o family in terms of capabilities, sadly. I still find them so useful, such a pity that there’s no real competitor in this segment, having the experience a real conversation has helped me so much in expressing ideas and concepts. Still, it’s worth to keep in mind that these…

Yes the voice part of OpenAI realtime/voice mode is great but it’s pretty dumb compared to newer models and often gets stuck repeating itself. Google’s Gemini flash live 3.1 is better, especially used via the API - it can do tool calling (including to other, even smarter LLMs if you set it up yourself), you can set the reasoning level (even high is still close enough to realtime) and it can ground answers in google s…

Thanks, I’ll try it, even if my experience wasn’t that great with Google models lately (503s)

Re: How OpenAI delivers low-latency voice AI at scale

#76

Post by dharma1 »

Lucasoato wrote: post #75
dharma1 wrote: post #74

Earlier quoted context omitted.

Yes the voice part of OpenAI realtime/voice mode is great but it’s pretty dumb compared to newer models and often gets stuck repeating itself. Google’s Gemini flash live 3.1 is better, especially used via the API - it can do tool calling (including to other, even smarter LLMs if you set it up yourself), you can set the reasoning level (even high is still close enough to realtime) and it can ground answers in google s…

Thanks, I’ll try it, even if my experience wasn’t that great with Google models lately (503s)

Give it a shot, 3.1 live one in AI studio/API and max out reasoning - not the one in Gemini app it’s an older model.

Another option is to use pipecat with their VAD and separate STT and TTS and any (fast) LLM of your choice - but it’s more plumbing and not a true speech to speech model

Re: How OpenAI delivers low-latency voice AI at scale

#77

Post by doctorpangloss »

Sean-Der wrote: post #28
doctorpangloss wrote: post #13

what i learned from making a webrtc+kubernetes game streaming product: - openai is wrong. almost of the issues they described are issues with libwebrtc, not with webrtc, kubernetes, network architecture, etc. the clue was when they said "the conventional one-port-per-session WebRTC model." - there are no alternatives worth trying. everything else open source in the ecosystem, like pion, coturn, stunner, are too immat…

Did you use libwebrtc on the backend? When you say `libwebrtc` is the only game in town are you talking about clients or servers? Even for clients you have things like libpeer that libwebrtc can't hit.

yes - i used libwebrtc on the backend and, pre-LLM, patched it to work around a lot of the things i discovered that were directly related to low latency AV streaming. pion didn't exist then.

i think the challenge is that pion is an excellent product today. it would benefit me if its innovations were subsumed into libwebrtc, because eventually those innovations will show up in the iOS stack, which is one of the customers that matter to me. it is subjective if it is the MOST important customer, that is my belief and it is probably true of openai, at least until they get their own device out the door.

there can be many, many use cases though! not everything has to be, try to make the thing for 1b people that has to interact with all the most powerful and meanest businesses on the planet.

Re: How OpenAI delivers low-latency voice AI at scale

#78

Post by oezi »

aleda145 wrote: post #55
Sean-Der wrote: post #23

Very grateful that OpenAI published the article/publicized their usage of Pion[0] a library I work on. If you aren't familiar with WebRTC it's a super fun space. I work on a book WebRTC for the Curious [1] that details how it works. [0] https://github.com/pion/webrtc [1] https://webrtcforthecurious.com

Appreciate you putting the entire book online! I read parts of it a while ago when I had an idea on using webRTC data channels to pass data from databases to browser clients via a CLI. Your book made me understand that it's probably not a great fit for my use case. I just used a centralized control plane and websockets instead. I still feel like there is something fun that we can do with webRTC data channels + zero c…

What is preventing the fun is that even though we now have IPv6 widely enough available we still can't have p2p connections in the browser without a cumbersome control plane of servers. If you could join a federation in the browser from some bootstrap IPs then I think we could have some real distributed fun.

Re: How OpenAI delivers low-latency voice AI at scale

#79

Post by jdironman »

saturdaysaint wrote: post #51
legohead wrote: post #22

The low latency is more of a pain point than a good thing, the way they have it implemented. Trying to have a casual conversation with it, as humans we naturally pause, and GPT will take this as you are "done" and start blabbing away. I also suffer from finding the appropriate word I want as I've gotten older and slower, and this fast-voice-gpt just ends up frustrating me more than helping. I have to sit there and th…

In voice conversations I tell it not to reply at all or only say “Understood” until I use some kind of code word. Not perfect, but less intrusive.

Roger that, over.

Re: How OpenAI delivers low-latency voice AI at scale

#80

Post by mschuster91 »

fragmede wrote: post #45
jiggawatts wrote: post #21

Earlier quoted context omitted.

Something I noticed is that companies that are vibe-coding their products miss out on the intelligence that (still) only humans can bring to bear. Just the knowledge cutoff alone puts AI at a serious disadvantage in any rapidly changing field.

GPT 5.5's knowledge cutoff is August 2025. Which aspect of WebRTC has meaningfully changed since then?

The problem is the sheer amount of knowledge out there. Particularly when using niche technologies (which webrtc and web audio still is, when measured by how many people develop using it), it is not surprising that AI doesn't have everything available in its responses, unless you specifically ask it about something you already know it should know.
Post reply on HN