Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s WebRTC problem

moq.dev

Post reply on HN
Print view View original

131–140 of 157 posts

Re: OpenAI’s WebRTC problem

#131

Post by Kwpolska »

harrison_clarke wrote: post #130
modeless wrote: post #122

Earlier quoted context omitted.

> Humans will naturally prefer the auditory experience of an occasional dropped packet, vs backed up audio or audio that plays at an uneven rate Yes but the difference here is there is only one human in the conversation. The other side can tolerate a 200ms delay in receiving or sending perfectly fine because it is not constrained to run in exactly real time like a human brain is. I think he is right. This is an inter…

i haven't used the openai voice thing but, if it's trying to respond in a natural way, with interruptions in both directions, it may still be a good idea. if there's a delay between you stopping and it starting talking, it feels weird (you might be able to fake some of that on the client, but then you need a thicker client)

Which LLM can generate text so quickly a real-time conversation is viable?

Re: OpenAI’s WebRTC problem

#134

Post by 6510 »

HarHarVeryFunny wrote: post #128
6510 wrote: post #120

Earlier quoted context omitted.

Do speech to text on the client and send the text/subtitles along with the audio. If the connection is truly bad, upload your voice and quantify emotional payload.

I guess different approaches could be applicable for client to server vs server to client. For client to server you want low latency, don't care about pauses introduced by communications (the model doesn't care), and could certainly tolerate a fallback to lower bandwidth text only (local SST) or more heavily compressed voice. For server to client it needs to be high quality voice without pauses, but as the parent was…

HarHar, that makes me think of those people who start each sentence with your name.

Re: OpenAI’s WebRTC problem

#135

Post by kixelated »

vince14 wrote: post #75
kixelated wrote: post #33

Earlier quoted context omitted.

Hello Mr Author here. Apologies that my comment replies aren't as funny. Every low-latency application has to decide the user experience trade-off between quality and latency. Congestion causes queuing (aka latency) and to avoid that, something needs to be skipped (lower quality). The WebRTC latency vs. quality knob is fixed. It's great at minimizing latency, but suffers from a lack of flexibility. We still (try to)…

Isn't the jitterBufferTarget [0] the latency vs. quality knob? [0] https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpRecei...

Close, but that's a minimum latency. We want a maximum latency knob.

Re: OpenAI’s WebRTC problem

#136

Post by regularfry »

6510 wrote: post #120
HarHarVeryFunny wrote: post #98

Earlier quoted context omitted.

So you could just locally generate the "You're absolutely right! ..." prefix without even waiting for the response to stream in!

Do speech to text on the client and send the text/subtitles along with the audio. If the connection is truly bad, upload your voice and quantify emotional payload.

I do wonder if you actually need two models here. Audio-to-audio hindbrain on the client, and a beefy text-mode frontal lobe somewhere in the cloud, with the comms between them explicitly trained in as a potentially low-bandwidth steering connection transferring embeddings, not text.

Re: OpenAI’s WebRTC problem

#137

Post by HarHarVeryFunny »

6510 wrote: post #134
HarHarVeryFunny wrote: post #128

Earlier quoted context omitted.

I guess different approaches could be applicable for client to server vs server to client. For client to server you want low latency, don't care about pauses introduced by communications (the model doesn't care), and could certainly tolerate a fallback to lower bandwidth text only (local SST) or more heavily compressed voice. For server to client it needs to be high quality voice without pauses, but as the parent was…

HarHar, that makes me think of those people who start each sentence with your name.

:) I guess that'd work too if they want to go with the Butt-Head persona!

Re: OpenAI’s WebRTC problem

#138

Post by ricardobeat »

entrope wrote: post #119
thutch76 wrote: post #117

I didn't make it all the way through the post, but I have to say I think he fundamentally understands the purpose of WebRTC. He calls himself an expert, and yeah he's written SFU's in go and rust and different companies ... but his technical credentials do not mean he's correct. Maybe it's a comprehension issue on my end, but he seems to associate things like stun and dtls as related, compounding issues (particularly…

I think you're not really engaging with his point, which is that RTC is a poor fit for communicating with an AI agent. I didn't read the blog as claiming that WebRTC is bad for what it is, only that it's a (very) poor choice for a voice-to-AI application.

Only if you expect to interact with the agent in a turn-taking format, with (possible) pauses between every turn.

ChatGPT’s voice mode is like speaking to someone in real time on a voice call, not input -> output.

Re: OpenAI’s WebRTC problem

#139

Post by jazzyjackson »

Oh is this why 1 800 CHAT GPT is trash now? It worked great when I started using it months ago. Last few times I've called the bot constantly interrupts herself, or stops as if I'm interrupting. I can't get a single full sentence out of her so I stopped calling.

I've experienced super deranged behavior out of 1800CHATGPT too, when I was just bored and called to ask how she's doing, what's her day like, she spiraled into laughing maniacally. It was unsettling, that was just before the service became unreliable, so I'm really curious what changed about the architecture.

Re: OpenAI’s WebRTC problem

#140

Post by throwaway89201 »

modeless wrote: post #122
thutch76 wrote: post #117

I didn't make it all the way through the post, but I have to say I think he fundamentally understands the purpose of WebRTC. He calls himself an expert, and yeah he's written SFU's in go and rust and different companies ... but his technical credentials do not mean he's correct. Maybe it's a comprehension issue on my end, but he seems to associate things like stun and dtls as related, compounding issues (particularly…

> Humans will naturally prefer the auditory experience of an occasional dropped packet, vs backed up audio or audio that plays at an uneven rate Yes but the difference here is there is only one human in the conversation. The other side can tolerate a 200ms delay in receiving or sending perfectly fine because it is not constrained to run in exactly real time like a human brain is. I think he is right. This is an inter…

> The reason we skip 200ms instead of pausing for 200ms when we get missed packets in a WebRTC call is because we can't pause the human on the other side of the call. But we can pause AI just fine.

This isn't about pausing anyone; it's about doing faster-than-realtime processing after a delay event. Humans can do that to some extent, and this is in fact done with some voice applications like Microsoft Teams, where after a network interruption the audio is sometimes played back really fast until the point that it becomes real-time again.

I hope it's an intentional design decision, because it works really well (for me). I can often perfectly keep track of a conversation in spite of the network delay. As much as I hate Teams, its meetings and voice implementation (also noise cancellation) works quite well, especially compared to current open source solutions like Jitsi or BigBlueButton.

Post reply on HN