[dead]
OpenAI’s WebRTC problem
91–100 of 157 posts
Re: OpenAI’s WebRTC problem
#92Earlier quoted context omitted.
Latency versus reliability is a false dichotomy anyway. The alternative to WebRTC isn't to wait for the user to finish speaking before you send any of the audio. Open a websocket and send the coded audio packets as they're generated. Now you're still sending audio packets immediately, but if one is dropped, TCP retransmits it until it makes it through. If the connection is really slow, packets queue up, and the user…
You ultimately still need a jitter buffer large enough to absorb retransmisiones. Otherwise you’ve got stuttering audio. And dynamically adjusting this jitter buffer is hard
> And dynamically adjusting this jitter buffer is hard
Unappreciated part of this entire conversation.
Re: OpenAI’s WebRTC problem
#93> But nope, WebRTC has no buffering and renders based on arrival time. Like seriously, timestamps are just suggestions. It’s even more annoying when video enters the picture.
I felt that comment my bones. Why would anyone possibly have the need to know actual presentation timestamp and how that corresponds to actual realtime? Evidently, no one working on WebRTC has had to synchronise data streams from varying sources before with millisecond accuracy.
I was doing a demo for a video stabilisation using a webcam and IMU module in the browser. It turns out the latency between video->rtc->browser and sensor->websocket->browser are wildly different and not constant. The obvious solution would be to send UTC timestamps for the sensors data and synchronise in browser. Not possible, the video has no UTC timestamp reference. When you have control of both sides of the WebRTC pipe, you can do fun things like send the UTC timestamp of the start of the stream, but this won’t solve browser jitter. It worked well enough for a POC but the entire solution had to be reengineered.
Re: OpenAI’s WebRTC problem
#94I have a lot of experience in this area (and some patent applications). For Alexa, the device established a connection back to the server and then kept that open, sending basically HTTP2/SPDY/Something like it over the wire after it detected the wake word. This allowed the STT start processing before you finish talking, so there is only a small delay in processing the last few chunks of your utterance. The answer cam…
"The author is absolutely right, a real time protocol isn't necessary. It's more important to get all the data. The user won't even notice a delay until you get over 500ms" Not my experience, running around 6,000 conversations per day with voice, with webrtc + cascading (stt/llm/tts) architecture. Maybe I misunderstood your comment, but that 500ms is basically the floor of a stat of the art voice implementation these…
This is pretty insightful thank you. Which provider are you guys using? Is it also over the phone or fully web/app based. Do you have any resources you can point me to learn about this?
Re: OpenAI’s WebRTC problem
#95Earlier quoted context omitted.
“How hard can it be?” the strawman asked. It’s 2026 and teleconferencing is still such a shit show. There’s billions of dollars to be had and Zoom is at best mediocre, and it can be as bad as Microsoft Whatchamacallit. I’ve never not seen teleconferencing be a ham handed mess.
Facetime does alright in the consumer segment.
The most frustrating thing about FaceTime is it sometimes appears to significantly duck audio in order to avoid echoes. I can't predict on which devices it will happen, but it often does when I call my parents and it absolutely destroys the conversation. If they're telling me something and I make the slightest "uhuh" acknowledgment sound, their mic input gets effectively muted for a second or so and I miss what they say.
Re: OpenAI’s WebRTC problem
#96Earlier quoted context omitted.
> Instant but confident sounding (and confident sounding because it's instant) will beat slower every time. Sure, but I am skeptical that users are actually saying "I prefer wrong answers over lag", which is what the post I responded to implied. This is different to user's saying "I prefer quick answers to laggy answers", which is what I presume they may have said. To actually settle this, the feedback must answer th…
Dunno. Feels like stated vs revealed preferences to me. Of course everyone will _say_ they want the wrong answers, but I can totally see users getting annoyed at slow responses, thinking that the developers should've traded accuracy for quicker responses. (or not thinking that at all, just demanding quicker responses unconditionally)
No I think they are saying no one would say they want wrong answers. People say they want fast answers and they are implying they should also be correct.
Re: OpenAI’s WebRTC problem
#97Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though. > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the…
> > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate > This is the opposite of the feedback I get. Users want instant responses. I am skeptical that you are getting feedback that users prefer instant wrong results to 200ms-lag correct results. Deeply skeptical!
> I am skeptical that you are getting feedback that users prefer instant wrong results to 200ms-lag correct results.
You are skeptical that people would prefer instant responses with 99.99% accuracy to waiting noticeably longer for a higher-accuracy rate?
The Internet, over its entire history, suggests otherwise.
Re: OpenAI’s WebRTC problem
#98Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though. > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the…
Delivery of first phoneme and delivery of the important information don't have to be coupled. Politicians on TV get very good at this particular trick, they've got a set of stock phrases which basically fill time while their brain gets in gear. We just need something to fill the gap so our System 1 doesn't lose confidence in the interaction.
So you could just locally generate the "You're absolutely right! ..." prefix without even waiting for the response to stream in!
Re: OpenAI’s WebRTC problem
#99Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though. > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the…
HELLO MR SEAN, 1. Of course users want lower latency, but they also want fewer instances where the LLM "misheard" them. It would be amazing to run A/B experiments on the trade-off between latency vs quality, but WebRTC makes that knob difficult to turn. 2. I'm obviously not an TTS expert, but what benefit is there to trickling out the result? The silicon doesn't care how quickly the time number increments? 3. Yeah, s…
The misunderstanding the user comes down to understanding how the user prompts and what type of responses the user gets in return. I’m wondering if for anything code the llm could have an interrupter that it would first read what the user wrote and translate it to proper sentence structure and the return a truer value. I think the llm is having an understanding issue because everyone has a unique signature in how they explain something. That signature operates like a personal language of the user as to which most of us will run through different scenarios to come up with a conclusion from our personal signature/language in which we conduct ourselves. And since llms are gamed to get to the answer faster using less tokens it probably picks the average high level signature that can be used for multiple users.
Re: OpenAI’s WebRTC problem
#100[dead]