What would actually be really interesting, text to speech on the device, you could easily stream text to the client which could generate the voice in realtime, far less bandwidth, latency is not really an issue.
OpenAI’s WebRTC problem
151–157 of 157 posts
Re: OpenAI’s WebRTC problem
#152Earlier quoted context omitted.
Facetime does alright in the consumer segment.
The most frustrating thing about FaceTime is it sometimes appears to significantly duck audio in order to avoid echoes. I can't predict on which devices it will happen, but it often does when I call my parents and it absolutely destroys the conversation. If they're telling me something and I make the slightest "uhuh" acknowledgment sound, their mic input gets effectively muted for a second or so and I miss what they…
I’ve found that if the recipient is wearing headphones/earphones, you can freely interrupt them without getting ducked (and vice-versa). Doesn’t help much when you’re calling multiple people / have to be on speaker, but makes it predictable at least.
Re: OpenAI’s WebRTC problem
#153> But nope, WebRTC has no buffering and renders based on arrival time. Like seriously, timestamps are just suggestions. It’s even more annoying when video enters the picture. I felt that comment my bones. Why would anyone possibly have the need to know actual presentation timestamp and how that corresponds to actual realtime? Evidently, no one working on WebRTC has had to synchronise data streams from varying sources…
At least the WebRTC library (not sure about browser integration) can do some a/v sync. RTP audio and video both have timestamps; but of course they have different frequencies and epochs. RTCP sender reports include an RTP time and an NTP time, so you can correlate them.
Personally, I'm not thrilled with how webrtc modulates playback to try to synchronize the two streams, so the SFU I work with doesn't send NTP timestamps in the sender reports or we just don't send sender reports; I can't recall the details atm. Part of the problem may be that our SFU always send audio immediately, but video gets buffered and paced.
For 1:1 calls not using the SFU, a/v sync seems to work and was not controversial when we enabled it.
Re: OpenAI’s WebRTC problem
#154> You speak into the microphone, it gets sent to one of OpenAI’s billion servers, and then a GPU pretends to talk to you via text-to-speech. Neato.
People (including this article) keep talking about OpenAI realtime like it’s a STT - LLM - TTS pipeline but I think this is a fundamental misunderstanding of how the model works. My understanding is that it accepts (and outputs) actual raw audio waveforms. Which, for me, is the sheer joy and wonder of the thing.
Re: OpenAI’s WebRTC problem
#155Earlier quoted context omitted.
The most frustrating thing about FaceTime is it sometimes appears to significantly duck audio in order to avoid echoes. I can't predict on which devices it will happen, but it often does when I call my parents and it absolutely destroys the conversation. If they're telling me something and I make the slightest "uhuh" acknowledgment sound, their mic input gets effectively muted for a second or so and I miss what they…
I’ve found that if the recipient is wearing headphones/earphones, you can freely interrupt them without getting ducked (and vice-versa). Doesn’t help much when you’re calling multiple people / have to be on speaker, but makes it predictable at least.
Well sure, that's the easy case for echo cancellation (remove the source of echoes entirely).
But we've had solutions to this for decades and it doesn't involve ducking the recipients audio. Apple and their billions of dollars should have had this completely solved by now.
Re: OpenAI’s WebRTC problem
#156Just use UDP
This feels like an underrated comment. Anyone here got the technical chops to address it?
Re: OpenAI’s WebRTC problem
#157Earlier quoted context omitted.
Do speech to text on the client and send the text/subtitles along with the audio. If the connection is truly bad, upload your voice and quantify emotional payload.
I do wonder if you actually need two models here. Audio-to-audio hindbrain on the client, and a beefy text-mode frontal lobe somewhere in the cloud, with the comms between them explicitly trained in as a potentially low-bandwidth steering connection transferring embeddings, not text.