I didn't understand - why is WebRTC good for Google Meet and not good for all other conferencing apps?
OpenAI’s WebRTC problem
71–80 of 157 posts
Re: OpenAI’s WebRTC problem
#72Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though. > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the…
> This is the opposite of the feedback I get. Users want instant responses.
Did they really say they prefer fast response over accurate repsonse?
Re: OpenAI’s WebRTC problem
#73Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though. > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the…
> > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate > This is the opposite of the feedback I get. Users want instant responses. I am skeptical that you are getting feedback that users prefer instant wrong results to 200ms-lag correct results. Deeply skeptical!
I would be punching my phone if the stupid network causing a wrong prompt and the LLM sends me unrelated answers. Correctness should be foundational no matter what, then improve the latency as best as possible. We all understand that if the network is bad then the latency can not be guaranteed but correctness should be.
Re: OpenAI’s WebRTC problem
#74Why does the voice need to be sent to the server? Why not perform speech-to-text on-device? Is the p10 phone/laptop not capable of this yet, despite every "dictation" feature I see in every modern OS?
Re: OpenAI’s WebRTC problem
#75> WebRTC is designed to degrade and drop my prompt during poor network conditions You want real time that's what you are going to deal with. If you don't want real time and instead imagine everything as STT -> Prompt -> TTS then maybe you shouldn't even be sending audio on the wire at all.
Hello Mr Author here. Apologies that my comment replies aren't as funny. Every low-latency application has to decide the user experience trade-off between quality and latency. Congestion causes queuing (aka latency) and to avoid that, something needs to be skipped (lower quality). The WebRTC latency vs. quality knob is fixed. It's great at minimizing latency, but suffers from a lack of flexibility. We still (try to)…
Isn't the jitterBufferTarget [0] the latency vs. quality knob?
[0] https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpRecei...
Re: OpenAI’s WebRTC problem
#76Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though. > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the…
> > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate > This is the opposite of the feedback I get. Users want instant responses. I am skeptical that you are getting feedback that users prefer instant wrong results to 200ms-lag correct results. Deeply skeptical!
100% agree. Sounds like they're either asking the wrong questions, or quoting answers selectively to suit this argument.
Re: OpenAI’s WebRTC problem
#77Amazing read. Blog posts rarely keep my attention like this one.
Re: OpenAI’s WebRTC problem
#78Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though. > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the…
> > …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate > This is the opposite of the feedback I get. Users want instant responses. I am skeptical that you are getting feedback that users prefer instant wrong results to 200ms-lag correct results. Deeply skeptical!
Especially when 200ms is the rule of thumb for things still feeling "instant" to users in terms of UX, this is like a rounding error in terms of latency when I regularly wait for actual minutes for an LLM to finish its bloody thinking and have to refresh through several "we're experiencing heavy load" errors.
Re: OpenAI’s WebRTC problem
#79This poor soul. There are few protocols I hate implementing more than WebRTC. Getting a simple client going means you need to quickly acclimate to SDP, TURN/STUN, ice-candidates, offers, peer-to-peer protocols, and the complex handshake that is implemented from scratch each time. I can't imagine re-writing the whole trenchcoat of protocols and unintended "best-practices".
The first time I was able to get a working webrtc datachannel setup with aiortc was when LLMs became a thing, before that it it was pretty much impossible full stop. Nobody knows what or how, there are no examples. It's a horrible protocol that just needs to die.
Re: OpenAI’s WebRTC problem
#80Why does the voice need to be sent to the server? Why not perform speech-to-text on-device? Is the p10 phone/laptop not capable of this yet, despite every "dictation" feature I see in every modern OS?
An eventual goal is likely to allow interacting with the LLM directly via audio tokens in input/output skipping tts and stt completely.