Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s WebRTC problem

moq.dev

Post reply on HN
Print view View original

21–30 of 157 posts

Re: OpenAI’s WebRTC problem

#21

Post by Sean-Der »

Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though.

> …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate

This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the magic. You also don't want to send faster than real-time. If the user interrupts the model you just wasted a bunch of bandwidth sending 3 minutes of audio (but only played 10 seconds)

> TTS is faster than real-time

https://research.nvidia.com/labs/adlr/personaplex/ Voice AI for the latest/aspirational is moving away from what the author describes. It is trickled in/out at 20ms

> We really hope the user’s source IP/port never changes, because we broke that functionality.

That is supported. When new IP for ufrag comes in its supported

> It takes a minimum of 8* round trips (RTT)

That's wrong. https://datatracker.ietf.org/doc/draft-hancke-webrtc-sped/

> I’d just stream audio over WebSockets

You lose stuff like AEC. You also push complexity on clients. The simplicity of WebRTC (createOffer -> setRemoteDescription) is what lets people onboard easily. Lots of developers struggled with Realtime API + web sockets (lots of code and having to do stuff by hand)

----

I think if I had my choice I would pick Offer/Answer model and then doing QUIC instead of DTLS+SCTP. Maybe do RTP over QUIC? I personally don't feel strongly about the protocol itself. I don't know how to ship code to multiple clients (and customers clients) with a much large code footprint.

Re: OpenAI’s WebRTC problem

#22

Post by tardedmeme »

whattheheckheck wrote: post #13
Giefo6ah wrote: post #7

Yet another victim of IPv4, and you still find countless detractors of IPv6 on every thread where it's mentioned.

How would ipv6 handle it

You just send packets to the other party's address and they send packets back to yours. Both parties know their address and you don't need a relay in the middle.

Re: OpenAI’s WebRTC problem

#23

Post by cowsandmilk »

fidotron wrote: post #9

> WebRTC is designed to degrade and drop my prompt during poor network conditions You want real time that's what you are going to deal with. If you don't want real time and instead imagine everything as STT -> Prompt -> TTS then maybe you shouldn't even be sending audio on the wire at all.

> You want real time

Isn’t the point that OpenAI’s use case does not require realtime?

When OpenAI responds, it has most of the audio in advance of when the user needs to hear it. It produces audio faster than real time, so a real time protocol is a bad fit.

Re: OpenAI’s WebRTC problem

#24

Post by Sean-Der »

cowsandmilk wrote: post #23
fidotron wrote: post #9

> WebRTC is designed to degrade and drop my prompt during poor network conditions You want real time that's what you are going to deal with. If you don't want real time and instead imagine everything as STT -> Prompt -> TTS then maybe you shouldn't even be sending audio on the wire at all.

> You want real time Isn’t the point that OpenAI’s use case does not require realtime? When OpenAI responds, it has most of the audio in advance of when the user needs to hear it. It produces audio faster than real time, so a real time protocol is a bad fit.

That is not the case. See get-realtime-translate[0 that's doing it as a trickle instead (not turn based).

[0] https://developers.openai.com/api/docs/models/gpt-realtime-t...

Re: OpenAI’s WebRTC problem

#27

Post by fragmede »

Waterluvian wrote: post #10
r2vcap wrote: post #3

This is frustratingly one-sided writing. Yeah, WebRTC has limitations, but relying on a standard buys you a lot of correctness and reduces long-term engineering cost. The fact that WebRTC is complicated does not mean it is wrong; it means real-time media over the public internet is complicated. Also, networking is inherently stateful. NAT traversal, jitter buffers, congestion control, packet loss, codec state, encryp…

“How hard can it be?” the strawman asked. It’s 2026 and teleconferencing is still such a shit show. There’s billions of dollars to be had and Zoom is at best mediocre, and it can be as bad as Microsoft Whatchamacallit. I’ve never not seen teleconferencing be a ham handed mess.

Facetime does alright in the consumer segment.

Re: OpenAI’s WebRTC problem

#28

Post by keizo »

interesting read albeit over my head, but i spent half of yesterday comparing Gemini Live (websockets) vs gpt-realtime-2 and while gpt is super good, seemingly more robust. Gemini connects faster.

Re: OpenAI’s WebRTC problem

#29

Post by tempaccount5050 »

jgalt212 wrote: post #6
awkii wrote: post #4

This poor soul. There are few protocols I hate implementing more than WebRTC. Getting a simple client going means you need to quickly acclimate to SDP, TURN/STUN, ice-candidates, offers, peer-to-peer protocols, and the complex handshake that is implemented from scratch each time. I can't imagine re-writing the whole trenchcoat of protocols and unintended "best-practices".

Have you attempted to use the Microsoft Graph API to interact with email?

It's way better than the old powershell modules imo. What don't you like?

Re: OpenAI’s WebRTC problem

#30

Post by jedberg »

I have a lot of experience in this area (and some patent applications). For Alexa, the device established a connection back to the server and then kept that open, sending basically HTTP2/SPDY/Something like it over the wire after it detected the wake word. This allowed the STT start processing before you finish talking, so there is only a small delay in processing the last few chunks of your utterance.

The answer came back over the same connection.

In the case of OpenAI, they can't exactly keep a persistent connection open like Alexa does, but they can use HTTP2 from the phone and both iOS and Android will pretty much take care of that connection magically.

The author is absolutely right, a real time protocol isn't necessary. It's more important to get all the data. The user won't even notice a delay until you get over 500ms. Especially in the age of mobile phones, where most people are used to their real time human to human communications to have a delay.

(If you work at OpenAI or Anthropic, give me a shout, I'm happy to get into more details with you)

Post reply on HN