Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s WebRTC problem

moq.dev

Post reply on HN
Print view View original

121–130 of 157 posts

Re: OpenAI’s WebRTC problem

#121

Post by Sean-Der »

kixelated wrote: post #40
Sean-Der wrote: post #21

Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though. > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the…

HELLO MR SEAN, 1. Of course users want lower latency, but they also want fewer instances where the LLM "misheard" them. It would be amazing to run A/B experiments on the trade-off between latency vs quality, but WebRTC makes that knob difficult to turn. 2. I'm obviously not an TTS expert, but what benefit is there to trickling out the result? The silicon doesn't care how quickly the time number increments? 3. Yeah, s…

1.) Latency vs quality doesn't come up enough to make people want to A/B test it unfortunately. At work I would say ~5 people care about WebRTC vs QUIC vs X. All effort is around the models (how can I provide tools to be support those doing that work)

2.) The model isn't processing just text anymore. Also taking into account breathing/emotion etc... not just spitting out big responses anymore. As it generates them it is taking into account the users response.

3.) It works with the LB setup today. Clients are sending ICE traffic, if it roams we lookup the ufrag and route appropriately.

4.) With DTLS 1.3 it is 1 RTT with SNAP[0] for WebRTC session. SCTP info goes in Offer/Answer, DTLS is packed into ICE. You are totally right about signaling though! [1] was my answer for doing WebRTC without signaling, couldn't get anyone to care though.

5.) I don't have anything that I need to tune. If I want to increase (or decrease) latency [3] is something I put into Transceiver. Otherwise I can't think of any 'change this WebRTC behavior' that has been asked by users/developers.

[0] https://datatracker.ietf.org/doc/draft-hancke-tsvwg-snap/

[1] https://github.com/pion/offline-browser-communication

[3] https://webrtc.googlesource.com/src/+/refs/heads/main/docs/n...

Re: OpenAI’s WebRTC problem

#122

Post by modeless »

thutch76 wrote: post #117

I didn't make it all the way through the post, but I have to say I think he fundamentally understands the purpose of WebRTC. He calls himself an expert, and yeah he's written SFU's in go and rust and different companies ... but his technical credentials do not mean he's correct. Maybe it's a comprehension issue on my end, but he seems to associate things like stun and dtls as related, compounding issues (particularly…

> Humans will naturally prefer the auditory experience of an occasional dropped packet, vs backed up audio or audio that plays at an uneven rate

Yes but the difference here is there is only one human in the conversation. The other side can tolerate a 200ms delay in receiving or sending perfectly fine because it is not constrained to run in exactly real time like a human brain is.

I think he is right. This is an interesting point that I haven't considered before. The reason we skip 200ms instead of pausing for 200ms when we get missed packets in a WebRTC call is because we can't pause the human on the other side of the call. But we can pause AI just fine.

Re: OpenAI’s WebRTC problem

#123

Post by jedberg »

Aurornis wrote: post #111
jedberg wrote: post #30

I have a lot of experience in this area (and some patent applications). For Alexa, the device established a connection back to the server and then kept that open, sending basically HTTP2/SPDY/Something like it over the wire after it detected the wake word. This allowed the STT start processing before you finish talking, so there is only a small delay in processing the last few chunks of your utterance. The answer cam…

> The user won't even notice a delay until you get over 500ms I think a lot of comments are getting so laser focused on the transport delays that they’re forgetting that the LLM pipeline isn’t instant. The transport delays are additive on top of all of the other delays, which are already high. Which I assume is why they reached for the lowest latency solution they could, because they need every bit of help they can g…

And that was the entire point of my comment. That your transport layer isn't your bottleneck. You can start processing before they finish speaking. Your bottleneck will always be what happens after that.

Re: OpenAI’s WebRTC problem

#124

Post by croes »

dolmen wrote: post #113
croes wrote: post #72

Earlier quoted context omitted.

> This is the opposite of the feedback I get. Users want instant responses. Did they really say they prefer fast response over accurate repsonse?

This is assuming the LLM can produce an accurate response.

Unlikely if the task gets inaccurately transmitted.

Re: OpenAI’s WebRTC problem

#125

Post by lelanthran »

jasonlotito wrote: post #97
lelanthran wrote: post #68

Earlier quoted context omitted.

> > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate > This is the opposite of the feedback I get. Users want instant responses. I am skeptical that you are getting feedback that users prefer instant wrong results to 200ms-lag correct results. Deeply skeptical!

> I am skeptical that you are getting feedback that users prefer instant wrong results to 200ms-lag correct results. You are skeptical that people would prefer instant responses with 99.99% accuracy to waiting noticeably longer for a higher-accuracy rate? The Internet, over its entire history, suggests otherwise.

Who claimed 99.99% accuracy?

A single dropped or missed word in a sentence can reverse the meaning.

I am skeptical that people would rather have wrong answers than lag. I am not claiming what the percentage is and neither are you, because no one measured it at the low lag.

Re: OpenAI’s WebRTC problem

#126

Post by jedberg »

vasishath4 wrote: post #69
aenis wrote: post #53

Earlier quoted context omitted.

"The author is absolutely right, a real time protocol isn't necessary. It's more important to get all the data. The user won't even notice a delay until you get over 500ms" Not my experience, running around 6,000 conversations per day with voice, with webrtc + cascading (stt/llm/tts) architecture. Maybe I misunderstood your comment, but that 500ms is basically the floor of a stat of the art voice implementation these…

I am myself working on something similar, but i have noticed that if I try to pass on early speech from the user to the LLM to reduce latency, chances of interruptions get even higher. For example, the user may say something like “Yes” followed by a brief pause, leading the speech model to count that as a complete turn, triggering the LLM call. But then the user may add something more, so i have to cancel the previou…

Detecting end of turn is a whole other issue. You can do the easy thing, which is just assign some number of milliseconds of silence as the end, or you can spend a lot of money asking the model to figure it out based on context.

Humans actually do the second thing, where we not only use our "model" to figure out end of turn, we actually predict what they are going to say based on context and will sometimes answer before they even finish.

Re: OpenAI’s WebRTC problem

#127

Post by DaleCurtis »

Sean-Der wrote: post #21

Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though. > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the…

FWIW, the getUserMedia() portion of such a setup remains the same, so you don't lose AEC or anything else coupled there.

Re: OpenAI’s WebRTC problem

#128

Post by HarHarVeryFunny »

6510 wrote: post #120
HarHarVeryFunny wrote: post #98

Earlier quoted context omitted.

So you could just locally generate the "You're absolutely right! ..." prefix without even waiting for the response to stream in!

Do speech to text on the client and send the text/subtitles along with the audio. If the connection is truly bad, upload your voice and quantify emotional payload.

I guess different approaches could be applicable for client to server vs server to client.

For client to server you want low latency, don't care about pauses introduced by communications (the model doesn't care), and could certainly tolerate a fallback to lower bandwidth text only (local SST) or more heavily compressed voice.

For server to client it needs to be high quality voice without pauses, but as the parent was suggesting you could potentially hide response latency (whether due to server or communication degradation) by using a human-like conversational "trick" of at least making some sound before brain is engaged and generating a response. "That's absolutely right! ..." would be a tad annoying, but "Hmm..." might be OK, especially if not done all the time, just as a locally initiated conversational filler when the server is slow to respond.

Re: OpenAI’s WebRTC problem

#130

Post by harrison_clarke »

modeless wrote: post #122
thutch76 wrote: post #117

I didn't make it all the way through the post, but I have to say I think he fundamentally understands the purpose of WebRTC. He calls himself an expert, and yeah he's written SFU's in go and rust and different companies ... but his technical credentials do not mean he's correct. Maybe it's a comprehension issue on my end, but he seems to associate things like stun and dtls as related, compounding issues (particularly…

> Humans will naturally prefer the auditory experience of an occasional dropped packet, vs backed up audio or audio that plays at an uneven rate Yes but the difference here is there is only one human in the conversation. The other side can tolerate a 200ms delay in receiving or sending perfectly fine because it is not constrained to run in exactly real time like a human brain is. I think he is right. This is an inter…

i haven't used the openai voice thing

but, if it's trying to respond in a natural way, with interruptions in both directions, it may still be a good idea. if there's a delay between you stopping and it starting talking, it feels weird

(you might be able to fake some of that on the client, but then you need a thicker client)

Post reply on HN