Just give me mpegts in element, I'm dying.
OpenAI’s WebRTC problem
101–110 of 157 posts
Re: OpenAI’s WebRTC problem
#102Earlier quoted context omitted.
HELLO MR SEAN, 1. Of course users want lower latency, but they also want fewer instances where the LLM "misheard" them. It would be amazing to run A/B experiments on the trade-off between latency vs quality, but WebRTC makes that knob difficult to turn. 2. I'm obviously not an TTS expert, but what benefit is there to trickling out the result? The silicon doesn't care how quickly the time number increments? 3. Yeah, s…
Human spoken conversation doesn’t really work like file buffering. People can tolerate missing words surprisingly well. If a phrase is slightly clipped, masked by noise, or dropped, the listener can often infer it from context. That happens constantly in real speech. But pauses and stalls are much more damaging. A sudden freeze in the middle of speech breaks turn-taking, timing, and attention. It feels like the speak…
I think this is mixing domains quite a bit;
If I'm talking to a friend or peer and I'm on a crappy link, we can probably work it out. If I'm calling my lawyer from prison with my "one call" I really want my lawyer to get my instructions clearly and correctly, ideally the first time without a lot of coaching.
Where on this scale does "person talking to LLM" fit?
I believe there's a ton of research into the shannon limit and human speech. You can trivially observe how much redundancy there is by listening to a podcast at 1x, 1.2x, 1.5x, 2x, etc, and when you can't follow what's going on, you've found the "redundancy" built into that language. This number falls way off when you're listening to a person with an accent or when the recording is noisy or whatever.
You'll also find that your tolerance for lossy media is radically different based on latency and echos and jitter in the audio (which I believe is the point of the original "don't use webrtc" article...)
Finally, people may tolerate this, but the "phonem to token" thinger may be less tolerant, and will certainly not be able to magic correct meaning from lost packets, and if the resulting exchange is extremely expensive or important (from the lawyer and the "I'm in jail in poughkeepsie; I need bail!" exchange) you really want to take the time to get it right, not make things guess.
Re: OpenAI’s WebRTC problem
#103Earlier quoted context omitted.
"The author is absolutely right, a real time protocol isn't necessary. It's more important to get all the data. The user won't even notice a delay until you get over 500ms" Not my experience, running around 6,000 conversations per day with voice, with webrtc + cascading (stt/llm/tts) architecture. Maybe I misunderstood your comment, but that 500ms is basically the floor of a stat of the art voice implementation these…
This is pretty insightful thank you. Which provider are you guys using? Is it also over the phone or fully web/app based. Do you have any resources you can point me to learn about this?
We use a bunch, at the moment we mainly self host (and use pipecat) use Daily, and a few niche boutique suppliers who built things for us.
There is a great resource for learning this stuff - the CEO of Daily, Kwindla Kramer, hosted a series of 1hr sessions on low latency voice ai. Here:
https://youtube.com/playlist?list=PLzU2zoMTQIHjMPZ-OnpC3ozZs...
Some of this is a bit outdated but most of it is very valuable.
Kwindla posts a lot of extremely useful stuff on x and linkedin, incl. working, easily replicable sub 500ms setups.
Re: OpenAI’s WebRTC problem
#104Earlier quoted context omitted.
"The author is absolutely right, a real time protocol isn't necessary. It's more important to get all the data. The user won't even notice a delay until you get over 500ms" Not my experience, running around 6,000 conversations per day with voice, with webrtc + cascading (stt/llm/tts) architecture. Maybe I misunderstood your comment, but that 500ms is basically the floor of a stat of the art voice implementation these…
I agree with everything you've said, I must have written it wrong. What I was saying is the same as you -- the user will tolerate a total delay of 500ms, and then happiness starts to fall off. We had some Alexa utterances at 500ms, the most basic ones, but most took longer. However, even with http2 and the like, we could get in that range because of the fact that it was sending data right away, so we were mostly done…
Sorry, I misunderstood your comment.
As for webrtc - it was mainly for decent support in browsers and built in AEC. I think we will take another look at this design choice if we run out of ways to further optimize.
Re: OpenAI’s WebRTC problem
#105Earlier quoted context omitted.
Oh, I can absolutely believe it. Humans are deeply irrational, especially about things that mess about in time frames too short for our conscious thought processes to kick in. Instant but confident sounding (and confident sounding because it's instant) will beat slower every time. You don't know which is correct until a long time after you've made a decision to trust it, or whether you like it.
> Instant but confident sounding (and confident sounding because it's instant) will beat slower every time. Sure, but I am skeptical that users are actually saying "I prefer wrong answers over lag", which is what the post I responded to implied. This is different to user's saying "I prefer quick answers to laggy answers", which is what I presume they may have said. To actually settle this, the feedback must answer th…
> actually saying
Yeah, I don't think that's the form of the feedback here.
Re: OpenAI’s WebRTC problem
#106Earlier quoted context omitted.
HELLO MR SEAN, 1. Of course users want lower latency, but they also want fewer instances where the LLM "misheard" them. It would be amazing to run A/B experiments on the trade-off between latency vs quality, but WebRTC makes that knob difficult to turn. 2. I'm obviously not an TTS expert, but what benefit is there to trickling out the result? The silicon doesn't care how quickly the time number increments? 3. Yeah, s…
Human spoken conversation doesn’t really work like file buffering. People can tolerate missing words surprisingly well. If a phrase is slightly clipped, masked by noise, or dropped, the listener can often infer it from context. That happens constantly in real speech. But pauses and stalls are much more damaging. A sudden freeze in the middle of speech breaks turn-taking, timing, and attention. It feels like the speak…
> People can tolerate missing words surprisingly well. If a phrase is slightly clipped, masked by noise, or dropped, the listener can often infer it from context. That happens constantly in real speech.
LLMs are surprisingly good at this, too.
This entire blog post is based on assumptions that
1) WebRTC garbling is common
2) LLMs fall apart if there are any audio glitches
I would bet money that OpenAI explored and has statistics on both of those and how it impacts service. More than this blogger heaping snark upon snark to avoid having a realistic conversation about pros and cons
Re: OpenAI’s WebRTC problem
#107Earlier quoted context omitted.
You ultimately still need a jitter buffer large enough to absorb retransmisiones. Otherwise you’ve got stuttering audio. And dynamically adjusting this jitter buffer is hard
I'm not an expert. Can't we abuse that LLMs don't need to receive audio as a continuous stream without interruptions? Couldn't we just send data and pipe it into the LLM with deduplication (if resending happens)? x...y...y[dedup]...z
Audio -> ASR - no jitter buffer
TTS -> human - jitter buffer
Re: OpenAI’s WebRTC problem
#108Earlier quoted context omitted.
This is pretty insightful thank you. Which provider are you guys using? Is it also over the phone or fully web/app based. Do you have any resources you can point me to learn about this?
We use a bunch, at the moment we mainly self host (and use pipecat) use Daily, and a few niche boutique suppliers who built things for us. There is a great resource for learning this stuff - the CEO of Daily, Kwindla Kramer, hosted a series of 1hr sessions on low latency voice ai. Here: https://youtube.com/playlist?list=PLzU2zoMTQIHjMPZ-OnpC3ozZs... Some of this is a bit outdated but most of it is very valuable. Kwin…
Beautiful thanks. We are also looking at this and another complication is transcripts can get pretty messy updates, corrections etc.
Re: OpenAI’s WebRTC problem
#109Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though. > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the…
> > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate > This is the opposite of the feedback I get. Users want instant responses. I am skeptical that you are getting feedback that users prefer instant wrong results to 200ms-lag correct results. Deeply skeptical!
Deeply false dichotomy!
The blog post glosses over the details and implies that 200ms of latency would be a magic solution. They do admit that WebRTC already has provisions for up to 200ms, so I guess they’re really implying that 400ms would be the happy case path for their alternative buffering, which is starting to get in the range where users would probably be annoyed.
Have you tried having conversational speech over a link with almost half a second of delay? It’s bad. You have to work hard to establish a turn taking routine with the other party and do extra mental work to identify your slot to talk.
The other half of this problem requires acknowledging that LLMs are actually pretty decent at interpreting input with gaps. You can drop words or even letters from LLM input and still get surprisingly decent results back. This post acts like a dropped packet means your response is going to send the LLM off on a wrong response or something.
Re: OpenAI’s WebRTC problem
#110Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though. > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the…
> . You also push complexity on clients. The simplicity of WebRTC (createOffer -> setRemoteDescription) is what lets people onboard easily. WebRTC is complex, even if it's a library (even if it's a library built into the browser they're already using). For a client/server voice interaction, I don't see why you would willingly use it. Ship voice samples over something else; maybe borrow some jitter buffer logic for pl…
> I could write an enormous rant about TURN [1]. But all of the webrtc protocol suite is designed for an internet that doesn't exist.
This is closer to being the real problem with WebRTC than the whole "it's making decisions about latency that I disagree with".
If you had a way to setup the tracks/channels over UDP connections that didn't involve P2P/STUN/TURN etc. but got to keep all the codec negotiation and things like AEC that would be awesome. MoQ isn't that though, because it's by people that don't actually see the whole problem end-to-end; just their little piece of it in the middle.