Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s WebRTC problem

moq.dev

Post reply on HN
Print view View original

11–20 of 157 posts

Re: OpenAI’s WebRTC problem

#11

Post by telman17 »

fidotron wrote: post #9

> WebRTC is designed to degrade and drop my prompt during poor network conditions You want real time that's what you are going to deal with. If you don't want real time and instead imagine everything as STT -> Prompt -> TTS then maybe you shouldn't even be sending audio on the wire at all.

Yep. Maybe there's some additional configuration I'm missing to mitigate the delay but clients don't seem to want to deal with the delay with STT -> Prompt -> TTS. They'll happily suffer occasional quality issues if the conversation feels "real".

Re: OpenAI’s WebRTC problem

#12

Post by charcircuit »

r2vcap wrote: post #3

This is frustratingly one-sided writing. Yeah, WebRTC has limitations, but relying on a standard buys you a lot of correctness and reduces long-term engineering cost. The fact that WebRTC is complicated does not mean it is wrong; it means real-time media over the public internet is complicated. Also, networking is inherently stateful. NAT traversal, jitter buffers, congestion control, packet loss, codec state, encryp…

QUIC is also a standard.

Re: OpenAI’s WebRTC problem

#14

Post by tekacs »

r2vcap wrote: post #3

This is frustratingly one-sided writing. Yeah, WebRTC has limitations, but relying on a standard buys you a lot of correctness and reduces long-term engineering cost. The fact that WebRTC is complicated does not mean it is wrong; it means real-time media over the public internet is complicated. Also, networking is inherently stateful. NAT traversal, jitter buffers, congestion control, packet loss, codec state, encryp…

You might have noticed that the author started the blog post explaining themselves:

  Like 6 years ago I wrote a WebRTC SFU at Twitch.
  Originally we used Pion (Go) just like OpenAI,
  but forked after benchmarking revealed that it was too slow.
  I ended up rewriting every protocol, because of course I did!

  Just a year ago, I was at Discord and I rewrote the WebRTC SFU in Rust.
  Because of course I did! You’re probably noticing a trend.

  Fun Fact: WebRTC consists of ~45 RFCs dating back to the early 2000s.
  And some de-facto standards that are technically drafts (ex. TWCC, REMB).
  Not a fun fact when you have to implement them all.

  You should consider me a Certified WebRTC Expert.
  Which is why I never, never want to use WebRTC again.
I think that they've done more than enough of 'trying the normal way' to be warranted in having an opinion the other way, don't you think?

Re: OpenAI’s WebRTC problem

#15

Post by lpln3452 »

I haven't really experienced disconnections while using ChatGPT. Gemini is the frustrating part. Simply backgrounding the app (and the web version too) and resuming it causes the response or the conversation with an assigned ID to disappear. Haha.

Re: OpenAI’s WebRTC problem

#16

Post by Sean-Der »

lpln3452 wrote: post #15

I haven't really experienced disconnections while using ChatGPT. Gemini is the frustrating part. Simply backgrounding the app (and the web version too) and resuming it causes the response or the conversation with an assigned ID to disappear. Haha.

I believe Gemini is Websockets? I have the same experience with heavy/custom applications that try to roll their own media stuff.

You run into issues around AudioContext and resumption etc... it's a PITA to have to handle all those corner cases :(

Re: OpenAI’s WebRTC problem

#17

Post by moomoo11 »

awkii wrote: post #4

This poor soul. There are few protocols I hate implementing more than WebRTC. Getting a simple client going means you need to quickly acclimate to SDP, TURN/STUN, ice-candidates, offers, peer-to-peer protocols, and the complex handshake that is implemented from scratch each time. I can't imagine re-writing the whole trenchcoat of protocols and unintended "best-practices".

i like livekit for this reason and their ceo is cool

Re: OpenAI’s WebRTC problem

#18

Post by edoceo »

jgalt212 wrote: post #6
awkii wrote: post #4

This poor soul. There are few protocols I hate implementing more than WebRTC. Getting a simple client going means you need to quickly acclimate to SDP, TURN/STUN, ice-candidates, offers, peer-to-peer protocols, and the complex handshake that is implemented from scratch each time. I can't imagine re-writing the whole trenchcoat of protocols and unintended "best-practices".

Have you attempted to use the Microsoft Graph API to interact with email?

Ugh. Who's decided to Graph all the things.

Re: OpenAI’s WebRTC problem

#19

Post by spongebobstoes »

this misses a few key things but hits on many others

webrtc is a bad protocol, without a doubt. I do like websockets as an easy alternative, but you do need to reinvent decent portions of webrtc as a result

I like the idea of MoQ but it's not widely used. probably worth experimenting with, especially as video enters the chat

> and then a GPU pretends to talk to you via text-to-speech

OpenAI is speech-to-speech, there is no TTS in voice mode

> It takes a minimum of 8* round trips (RTT) to establish a WebRTC connection

signalling can be done long ahead of time, though I don't see this mentioned in the OpenAI blog. I also saw some new webrtc extensions that should reduce setup time further

ultimately though, it comes down to

> It’s not like LLMs are particularly responsive anyway

I expect to see a shift in how S2S models work to be lower latency like the new voice API models that OpenAI announced

to be fair, the new models were released the day after this MoQ blog was published

Post reply on HN