Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s WebRTC problem

moq.dev

Post reply on HN
Print view View original

141–150 of 157 posts

Re: OpenAI’s WebRTC problem

#141

Post by bensyverson »

Kwpolska wrote: post #131
harrison_clarke wrote: post #130

Earlier quoted context omitted.

i haven't used the openai voice thing but, if it's trying to respond in a natural way, with interruptions in both directions, it may still be a good idea. if there's a delay between you stopping and it starting talking, it feels weird (you might be able to fake some of that on the client, but then you need a thicker client)

Which LLM can generate text so quickly a real-time conversation is viable?

There are now realtime “speech-to-speech” models [0]. I believe they skip text to streamline the architecture.

[0]: https://openai.com/index/introducing-gpt-realtime/

Re: OpenAI’s WebRTC problem

#142

Post by modeless »

throwaway89201 wrote: post #140
modeless wrote: post #122

Earlier quoted context omitted.

> Humans will naturally prefer the auditory experience of an occasional dropped packet, vs backed up audio or audio that plays at an uneven rate Yes but the difference here is there is only one human in the conversation. The other side can tolerate a 200ms delay in receiving or sending perfectly fine because it is not constrained to run in exactly real time like a human brain is. I think he is right. This is an inter…

> The reason we skip 200ms instead of pausing for 200ms when we get missed packets in a WebRTC call is because we can't pause the human on the other side of the call. But we can pause AI just fine. This isn't about pausing anyone; it's about doing faster-than-realtime processing after a delay event. Humans can do that to some extent, and this is in fact done with some voice applications like Microsoft Teams, where af…

Yes, it's about pausing. You pause the AI so it doesn't need to perceive the 200ms gap at all, unlike a human who will always perceive the interruption. Yes, then you run faster than real time to catch up.

Yes, humans can listen to audio faster than real time to catch up, but it degrades the experience and there is a fairly low limit to it. When talking to an AI you don't have to skip or speed up at all on the human side, is the point.

Re: OpenAI’s WebRTC problem

#143

Post by thutch76 »

entrope wrote: post #119
thutch76 wrote: post #117

I didn't make it all the way through the post, but I have to say I think he fundamentally understands the purpose of WebRTC. He calls himself an expert, and yeah he's written SFU's in go and rust and different companies ... but his technical credentials do not mean he's correct. Maybe it's a comprehension issue on my end, but he seems to associate things like stun and dtls as related, compounding issues (particularly…

I think you're not really engaging with his point, which is that RTC is a poor fit for communicating with an AI agent. I didn't read the blog as claiming that WebRTC is bad for what it is, only that it's a (very) poor choice for a voice-to-AI application.

That's fair. My attention wanted and I lost the plot.

However, I don't think having an agent on one side necessarily changes anything. Network problems are not predictable, particularly on mobile, so the human is still very likely to experience a poor auditory experience on a tcp connection.

Re: OpenAI’s WebRTC problem

#144

Post by echoangle »

thutch76 wrote: post #143
entrope wrote: post #119

Earlier quoted context omitted.

I think you're not really engaging with his point, which is that RTC is a poor fit for communicating with an AI agent. I didn't read the blog as claiming that WebRTC is bad for what it is, only that it's a (very) poor choice for a voice-to-AI application.

That's fair. My attention wanted and I lost the plot. However, I don't think having an agent on one side necessarily changes anything. Network problems are not predictable, particularly on mobile, so the human is still very likely to experience a poor auditory experience on a tcp connection.

The difference is that the agent doesn’t run in realtime. If 20 packets are lost and resent, the agent can still process them almost instantly and reply, in contrast to a human. Only the direction from the agent to the human needs to be realtime.

Re: OpenAI’s WebRTC problem

#145

Post by kixelated »

modeless wrote: post #122
thutch76 wrote: post #117

I didn't make it all the way through the post, but I have to say I think he fundamentally understands the purpose of WebRTC. He calls himself an expert, and yeah he's written SFU's in go and rust and different companies ... but his technical credentials do not mean he's correct. Maybe it's a comprehension issue on my end, but he seems to associate things like stun and dtls as related, compounding issues (particularly…

> Humans will naturally prefer the auditory experience of an occasional dropped packet, vs backed up audio or audio that plays at an uneven rate Yes but the difference here is there is only one human in the conversation. The other side can tolerate a 200ms delay in receiving or sending perfectly fine because it is not constrained to run in exactly real time like a human brain is. I think he is right. This is an inter…

Yeah that's a really good way of framing the argument, I wish I wrote that. The way robots listen/respond is bounded by compute, not time. Buffering audio isn't a great experience for humans but definitely works for robots.

Re: OpenAI’s WebRTC problem

#146

Post by pedalpete »

I used to work in WebRTC back in it's earlier days and our team developed the open-source rtc.io. (https://github.com/rtc-io)

I never would have imagined that OpenAI is sending the full audio of a request to their servers. I had always assumed the audio was transcribed locally and then sent to the server.

The only reason I can think they'd want the full audio is for later model training, which, ok, fair-enough, but this can still likely be done without the limitations of WebRTC.

Re: OpenAI’s WebRTC problem

#147

Post by coredog64 »

pocksuppet wrote: post #44
kixelated wrote: post #40

Earlier quoted context omitted.

HELLO MR SEAN, 1. Of course users want lower latency, but they also want fewer instances where the LLM "misheard" them. It would be amazing to run A/B experiments on the trade-off between latency vs quality, but WebRTC makes that knob difficult to turn. 2. I'm obviously not an TTS expert, but what benefit is there to trickling out the result? The silicon doesn't care how quickly the time number increments? 3. Yeah, s…

Latency versus reliability is a false dichotomy anyway. The alternative to WebRTC isn't to wait for the user to finish speaking before you send any of the audio. Open a websocket and send the coded audio packets as they're generated. Now you're still sending audio packets immediately, but if one is dropped, TCP retransmits it until it makes it through. If the connection is really slow, packets queue up, and the user…

This is how Nova Sonic works. Having done some implementations it’s trickier than you might like (e.g. the Python library for Sonic had problems with echoes and we had to use the Java library)

Re: OpenAI’s WebRTC problem

#148

Post by ekr____ »

tardedmeme wrote: post #22
whattheheckheck wrote: post #13

Earlier quoted context omitted.

How would ipv6 handle it

You just send packets to the other party's address and they send packets back to yours. Both parties know their address and you don't need a relay in the middle.

This really isn't the case, because people still have firewalls.
Post reply on HN