Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s WebRTC problem

moq.dev

Post reply on HN
Print view View original

111–120 of 157 posts

Re: OpenAI’s WebRTC problem

#111

Post by Aurornis »

jedberg wrote: post #30

I have a lot of experience in this area (and some patent applications). For Alexa, the device established a connection back to the server and then kept that open, sending basically HTTP2/SPDY/Something like it over the wire after it detected the wake word. This allowed the STT start processing before you finish talking, so there is only a small delay in processing the last few chunks of your utterance. The answer cam…

> The user won't even notice a delay until you get over 500ms

I think a lot of comments are getting so laser focused on the transport delays that they’re forgetting that the LLM pipeline isn’t instant.

The transport delays are additive on top of all of the other delays, which are already high.

Which I assume is why they reached for the lowest latency solution they could, because they need every bit of help they can get to start shrinking that end to end delay across the entire pipeline.

Analogies to human voice delay don’t work because in that case we treat the human as having no delay.

Re: OpenAI’s WebRTC problem

#112

Post by sporkland »

lelanthran wrote: post #68
Sean-Der wrote: post #21

Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though. > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the…

> > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate > This is the opposite of the feedback I get. Users want instant responses. I am skeptical that you are getting feedback that users prefer instant wrong results to 200ms-lag correct results. Deeply skeptical!

I think as a user I have 2 modes: 1. Q&A mode where it's basically Google search by voice. 2. I'm trying to process an idea I have with an LLM buddy.

My desires are pretty different in the two scenarios. Q&A mode if it's not quick to respond I'll think something is wrong with my phone.

Deep think mode I'm honestly kind of pissed off at how fast it tries to respond. I want it to slow down and give me a chance to process and use extra compute on its side (including newer models) so it doesn't just spew low thought bullshit at me.

It seems like the system could detect which of these two modes was happening and adapt, including protocol.

I haven't tried the voice mode since the new model updates, maybe it's gotten better.

Counter to everything I just said though and germain to the topic at hand, when I'm in q&a mode that's probably the worst time for it to drop audio as it changes the query significantly. vs when I'm talking at it for 2 minutes it could probably throw half away.

Re: OpenAI’s WebRTC problem

#113

Post by dolmen »

croes wrote: post #72
Sean-Der wrote: post #21

Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though. > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the…

> This is the opposite of the feedback I get. Users want instant responses. Did they really say they prefer fast response over accurate repsonse?

This is assuming the LLM can produce an accurate response.

Re: OpenAI’s WebRTC problem

#117

Post by thutch76 »

I didn't make it all the way through the post, but I have to say I think he fundamentally understands the purpose of WebRTC. He calls himself an expert, and yeah he's written SFU's in go and rust and different companies ... but his technical credentials do not mean he's correct.

Maybe it's a comprehension issue on my end, but he seems to associate things like stun and dtls as related, compounding issues (particularly in round trip time), but they are really orthogonal.

Also, he spends too much time talking about how you can't resend packets, and reiterates that point by stating they tried really hard (at discord?). That's where he lost the plot, imo.

The RTC in WebRTC is about real time communication. Humans will naturally prefer the auditory experience of an occasional dropped packet, vs backed up audio or audio that plays at an uneven rate. To clarify, I'm talking about human speech here.

If you want to tolerate packet loss, use a protocol based on tcp instead of udp. But you know what happens when you send audio over poor network conditions with tcp? There will be pauses on the receiving end as it waits for the next correct packet. Let's say the delay is multiple seconds. What should the receiving end do when packets start flowing again? Plays the clogged audio at a natural clock? Attempt to play the audio back at a higher rate to "catch up" with any other channels? People, humans, do not generally prefer that experience.

Forget about WebRTC for a minute, but instead think about tcp vs udp for voice. Voip has been based on udp since the 90's for a reason.

Re: OpenAI’s WebRTC problem

#119

Post by entrope »

thutch76 wrote: post #117

I didn't make it all the way through the post, but I have to say I think he fundamentally understands the purpose of WebRTC. He calls himself an expert, and yeah he's written SFU's in go and rust and different companies ... but his technical credentials do not mean he's correct. Maybe it's a comprehension issue on my end, but he seems to associate things like stun and dtls as related, compounding issues (particularly…

I think you're not really engaging with his point, which is that RTC is a poor fit for communicating with an AI agent. I didn't read the blog as claiming that WebRTC is bad for what it is, only that it's a (very) poor choice for a voice-to-AI application.

Re: OpenAI’s WebRTC problem

#120

Post by 6510 »

HarHarVeryFunny wrote: post #98
regularfry wrote: post #85

Earlier quoted context omitted.

Delivery of first phoneme and delivery of the important information don't have to be coupled. Politicians on TV get very good at this particular trick, they've got a set of stock phrases which basically fill time while their brain gets in gear. We just need something to fill the gap so our System 1 doesn't lose confidence in the interaction.

So you could just locally generate the "You're absolutely right! ..." prefix without even waiting for the response to stream in!

Do speech to text on the client and send the text/subtitles along with the audio.

If the connection is truly bad, upload your voice and quantify emotional payload.

Post reply on HN