Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Audio Models

openai.fm

Post reply on HN
Print view View original

211–220 of 317 posts

Re: OpenAI Audio Models

#211

Post by jeffharris »

taf2 wrote: post #181
ekzy wrote: post #178

Earlier quoted context omitted.

Do you know when we can expect an update on the realtime API? It’s still in beta and there are many issues (e.g voice randomly cutting off, VAD issues, especially with mulaw etc…) which makes it impossible to use in production, but there’s not much communication from OpenAI. It’s difficult to know what to bet on. Pushing for stt->llm->tts makes you wonder if we should carry on building with the realtime API.

Agreed- really not liking how they are neglecting it… I hope they are just hard at work behind the scenes and will release something soon

S2S is where we're investing the most effort on audio ... sorry it's been slow but we are working hard on it

Top priorities at the moment 1) Better function calling performance 2) Improved perception accuracy (not mishearing) 3) More reliable instruction following 4) Bug fixes (cutoffs, run ons, modality steering)

Re: OpenAI Audio Models

#212

Post by jeffharris »

minimaxir wrote: post #71
benjismith wrote: post #61

Is there way to get "speech marks" alongside the generated audio? FYI, Speech marks provide millisecond timestamp for each word in a generated audio file/stream (and a start/end index into your original source string), as a stream of JSONL objects, like this: {"time":6,"type":"word","start":0,"end":5,"value":"Hello"} {"time":732,"type":"word","start":7,"end":11,"value":"it's"} {"time":932,"type":"word","start":12,"en…

Passing the generated audio back to GPT-4o to ask for the structured annotations would be a fun test case.

this is a good solve. we don't support word time stamps natively yet, but are working on teaching GPT-4o that skill

Re: OpenAI Audio Models

#213

Post by jeffharris »

forgotpasagain wrote: post #68

It sounds very expressive but weirdly "fake" as if it's targeting to be similar to some NPC character, dataset issue?

really depends on the voice. but in general, no we want to sound as realistic as possible and I expect future voices to keep improving on this front

Re: OpenAI Audio Models

#216

Post by goshx »

oidar wrote: post #202
benjismith wrote: post #49

If I'm reading the pricing correctly, these models are SIGNIFICANTLY cheaper than ElevenLabs. https://platform.openai.com/docs/pricing If these are the "gpt-4o-mini-tts" models, and if the pricing estimate of "$0.015 per minute" of audio is correct, then these prices 85% cheaper than those of ElevenLabs. https://elevenlabs.io/pricing With ElevenLabs, if I choose their most cost-effectuve "Business" plan for $1100 per…

ElevenLabs is the only one offering speech to speech generation where the intonation, prosody, and timing is kept intact. This allows for one expressive voice actor to slip into many other voices.

OpenAI’s Realtime speech to speech is far superior than ElevenLabs.

Re: OpenAI Audio Models

#218

Post by jeffharris »

saint_yossarian wrote: post #205

The site just crashes with service workers disabled. First time I ran into a problem with that setting, which I set over two years ago.

oh doh. thanks ... we just pushed a fix for the crash. Unfortunately our currently implementation needs service works for streaming audio, so the "fix" was to disable the feature if the worker isn't available

Re: OpenAI Audio Models

#219

Post by urbandw311er »

jeffharris wrote: post #18

Hey, I'm Jeff and I was PM for these models at OpenAI. Today we launched three new state-of-the-art audio models. Two speech-to-text models—outperforming Whisper. A new TTS model—you can instruct it how to speak (try it on openai.fm!). And our Agents SDK now supports audio, making it easy to turn text agents into voice agents. We think you'll really like these models. Let me know if you have any questions here!

Hey Jeff, this is awesome! I’m actually building a S2S application right now for a startup with the Realtime API and keen to know when these new voices/expressive prompting will be coming to it?

Also, any word on when there might be a way to move the prompting to the server side (of a full stack web app)? At the moment we have no way to protect our prompts from being inspected in the browser dev tools — even the initial instructions when the session is initiated on the server end up being spat back out to the browser client when the WebRTC connection is first made! It’s damaging to any viable business model.

Some sort of tri-party WebRTC session maybe?

Re: OpenAI Audio Models

#220

Post by stavros »

lherron wrote: post #118
lukebuehler wrote: post #56

Earlier quoted context omitted.

yes, I think you are right. When I did the math on 11labs million chars I got the same numbers (Pro plan). I'm super happy about this, since I took a bet that exactly this would happen. I've just been building a consumer TTS app that could only work with significant cheaper TTS prices per million character (or self-hosted models)

Kokoro TTS is pretty good for open source. Worth checking out.

Oh man, they have the "Sky" voice, and it seems to be the same one that OpenAI had but then removed? Not sure how that's possible, but I'm very happy about it.
Post reply on HN