Most of the problems happen because we want to simulate human conversations. While thats a good goal to have, another approach is to let the user know clearly they are talking to a bot. You will be surprised at how accomodating users can be when they know they are talking to a bot and want their queries resolved.
OpenAI’s WebRTC problem
61–70 of 157 posts
Re: OpenAI’s WebRTC problem
#62My biggest frustration with WebRTC was precisely captured in the article: even if you don't need p2p and your video source is the process on the same host with your browser, you have to dance around connection setup like you're on a different side of a planet
Re: OpenAI’s WebRTC problem
#63Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though. > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the…
HELLO MR SEAN, 1. Of course users want lower latency, but they also want fewer instances where the LLM "misheard" them. It would be amazing to run A/B experiments on the trade-off between latency vs quality, but WebRTC makes that knob difficult to turn. 2. I'm obviously not an TTS expert, but what benefit is there to trickling out the result? The silicon doesn't care how quickly the time number increments? 3. Yeah, s…
[deleted]
Re: OpenAI’s WebRTC problem
#64There're tons of ways to fine-tune WebRTC that it wouldn't corrupt audio in poor network - it has all of the controls to smoothly trade-off latency vs quality. Not just NACKs - FEC, disable PLC/Acceleration/Deceleration, larger JB (tons of parameters) etc.
Most of the glitches I heard with OpenAI's Voice were not WebRTC related - but rather, to my ear, they sounded more like realtime issues with their inference - which is a very different component to optimize.
Re: OpenAI’s WebRTC problem
#65I've been using LiveKit which is also WebRTC based and it is super annoying when speed slows down or speeds up at times when connection is not robust. We were using OpenAI's websocket based RealTime audio which was way too slow. So I don't know which one is better. Generally our users like the LiveKit implementation better so maybe WebRTC with enough clever hacks is the answer.
This blog was super insightful for me to understand what are the root problems in the current implementation though.
Re: OpenAI’s WebRTC problem
#66This is frustratingly one-sided writing. Yeah, WebRTC has limitations, but relying on a standard buys you a lot of correctness and reduces long-term engineering cost. The fact that WebRTC is complicated does not mean it is wrong; it means real-time media over the public internet is complicated. Also, networking is inherently stateful. NAT traversal, jitter buffers, congestion control, packet loss, codec state, encryp…
> This is frustratingly one-sided writing
Tangential, but by being that, it's also refreshingly human writing, vs the both-sidesy bullet listed AI pablum that's all around us these days.
I have zero take on the subject matter, but I like that the article had a detectably human flair.
And if it was AI written, god help us.
Re: OpenAI’s WebRTC problem
#67Earlier quoted context omitted.
You ultimately still need a jitter buffer large enough to absorb retransmisiones. Otherwise you’ve got stuttering audio. And dynamically adjusting this jitter buffer is hard
I'm not an expert. Can't we abuse that LLMs don't need to receive audio as a continuous stream without interruptions? Couldn't we just send data and pipe it into the LLM with deduplication (if resending happens)? x...y...y[dedup]...z
You’re absolutely correct. A jitter buffer is necessary for a human listener, but a LLM isn’t aware of a time lapse, just like it isn’t aware of the time since your last message in the conversion (unless the chat harness explicitly informs it).
Re: OpenAI’s WebRTC problem
#68Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though. > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the…
> > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate
> This is the opposite of the feedback I get. Users want instant responses.
I am skeptical that you are getting feedback that users prefer instant wrong results to 200ms-lag correct results.
Deeply skeptical!
Re: OpenAI’s WebRTC problem
#69I have a lot of experience in this area (and some patent applications). For Alexa, the device established a connection back to the server and then kept that open, sending basically HTTP2/SPDY/Something like it over the wire after it detected the wake word. This allowed the STT start processing before you finish talking, so there is only a small delay in processing the last few chunks of your utterance. The answer cam…
"The author is absolutely right, a real time protocol isn't necessary. It's more important to get all the data. The user won't even notice a delay until you get over 500ms" Not my experience, running around 6,000 conversations per day with voice, with webrtc + cascading (stt/llm/tts) architecture. Maybe I misunderstood your comment, but that 500ms is basically the floor of a stat of the art voice implementation these…
I am myself working on something similar, but i have noticed that if I try to pass on early speech from the user to the LLM to reduce latency, chances of interruptions get even higher. For example, the user may say something like “Yes” followed by a brief pause, leading the speech model to count that as a complete turn, triggering the LLM call. But then the user may add something more, so i have to cancel the previous request so that any irreversible state transitions can be avoided. Now due to the lower latency (due to speculative calls), I get an even smaller window to actually cancel the response or even to stop the model from speaking.
Re: OpenAI’s WebRTC problem
#70This is frustratingly one-sided writing. Yeah, WebRTC has limitations, but relying on a standard buys you a lot of correctness and reduces long-term engineering cost. The fact that WebRTC is complicated does not mean it is wrong; it means real-time media over the public internet is complicated. Also, networking is inherently stateful. NAT traversal, jitter buffers, congestion control, packet loss, codec state, encryp…
You might have noticed that the author started the blog post explaining themselves: Like 6 years ago I wrote a WebRTC SFU at Twitch. Originally we used Pion (Go) just like OpenAI, but forked after benchmarking revealed that it was too slow. I ended up rewriting every protocol, because of course I did! Just a year ago, I was at Discord and I rewrote the WebRTC SFU in Rust. Because of course I did! You’re probably noti…
Right but they also state they have never implemented TURN which IMO is a marker of WebRTC expertness. (I haven't btw, just the WebRTC experts I know absolutely have written or worked on at some point a TURN implementation)