OpenAI Audio Models • Hacker News Board

Re: OpenAI Audio Models

#51

Post by jeffharris » Thu, Mar 20, 2025, 6:22 PM UTC

staticautomatic wrote: post #27

jeffharris wrote: post #18

Hey, I'm Jeff and I was PM for these models at OpenAI. Today we launched three new state-of-the-art audio models. Two speech-to-text models—outperforming Whisper. A new TTS model—you can instruct it how to speak (try it on openai.fm!). And our Agents SDK now supports audio, making it easy to turn text agents into voice agents. We think you'll really like these models. Let me know if you have any questions here!

Any plans to directly support diarization or voiceprinting?

We're thinking about diarization (adding time awareness to GPT models) but no firm plans to share just yet

Re: OpenAI Audio Models

#52

Post by jeffharris » Thu, Mar 20, 2025, 6:22 PM UTC

TheAceOfHearts wrote: post #25

jeffharris wrote: post #18

Hey, I'm Jeff and I was PM for these models at OpenAI. Today we launched three new state-of-the-art audio models. Two speech-to-text models—outperforming Whisper. A new TTS model—you can instruct it how to speak (try it on openai.fm!). And our Agents SDK now supports audio, making it easy to turn text agents into voice agents. We think you'll really like these models. Let me know if you have any questions here!

Is it against the TOS to use it for sexually explicit content?

Yes, from our terms: "Don’t build tools that may be inappropriate for minors, including: Sexually explicit or suggestive content. This does not include content created for scientific or educational purposes." https://openai.com/policies/usage-policies/

Re: OpenAI Audio Models

#53

Post by tosh » Thu, Mar 20, 2025, 6:23 PM UTC

Are these models only available via the API right now or also available as open weights?

Re: OpenAI Audio Models

#54

Post by fixprix » Thu, Mar 20, 2025, 6:25 PM UTC

benjismith wrote: post #49

If I'm reading the pricing correctly, these models are SIGNIFICANTLY cheaper than ElevenLabs. https://platform.openai.com/docs/pricing If these are the "gpt-4o-mini-tts" models, and if the pricing estimate of "$0.015 per minute" of audio is correct, then these prices 85% cheaper than those of ElevenLabs. https://elevenlabs.io/pricing With ElevenLabs, if I choose their most cost-effectuve "Business" plan for $1100 per…

It looks like they are targeting Google's TTS price point which is $16 per million characters which comes out to $0.015/minute.

Re: OpenAI Audio Models

#55

Post by pzo » Thu, Mar 20, 2025, 6:26 PM UTC

knicholes wrote: post #39

TheAceOfHearts wrote: post #25

Earlier quoted context omitted.

Is it against the TOS to use it for sexually explicit content?

I don't have your answer, but as far as innuendo goes, it's definitely capable!

[flagged]

Re: OpenAI Audio Models

#56

Post by lukebuehler » Thu, Mar 20, 2025, 6:27 PM UTC

benjismith wrote: post #49

If I'm reading the pricing correctly, these models are SIGNIFICANTLY cheaper than ElevenLabs. https://platform.openai.com/docs/pricing If these are the "gpt-4o-mini-tts" models, and if the pricing estimate of "$0.015 per minute" of audio is correct, then these prices 85% cheaper than those of ElevenLabs. https://elevenlabs.io/pricing With ElevenLabs, if I choose their most cost-effectuve "Business" plan for $1100 per…

yes, I think you are right. When I did the math on 11labs million chars I got the same numbers (Pro plan).

I'm super happy about this, since I took a bet that exactly this would happen. I've just been building a consumer TTS app that could only work with significant cheaper TTS prices per million character (or self-hosted models)

Re: OpenAI Audio Models

#57

Post by evalstate » Thu, Mar 20, 2025, 6:28 PM UTC

Really looking forward to integrating with these models.

The next version of Model Context Protocol will have native audio support (https://github.com/modelcontextprotocol/specification/pull/9...), which will open up plenty of opportunities for interop.

Re: OpenAI Audio Models

#58

Post by atlasunshrugged » Thu, Mar 20, 2025, 6:30 PM UTC

prdonahue wrote: post #38

theoryofx wrote: post #34

Still seems like Elevenlabs is crushing them on realtime audio, or does this change things?

Do you have any affiliation with Elevenlabs?

FWIW I have no affiliation with any of these companies but I have a book coming out soon and have been researching AI audiobook tools and Elevenlabs seems to be far and away the consensus for that at least

Re: OpenAI Audio Models

#59

Post by progbits » Thu, Mar 20, 2025, 6:31 PM UTC

jeffharris wrote: post #18

Hey, I'm Jeff and I was PM for these models at OpenAI. Today we launched three new state-of-the-art audio models. Two speech-to-text models—outperforming Whisper. A new TTS model—you can instruct it how to speak (try it on openai.fm!). And our Agents SDK now supports audio, making it easy to turn text agents into voice agents. We think you'll really like these models. Let me know if you have any questions here!

> Two speech-to-text models—outperforming Whisper

On what metric? Also Whisper is no longer state of the art in accuracy, how does it compare to the others in this benchmark?

https://artificialanalysis.ai/speech-to-text

Re: OpenAI Audio Models

#60

Post by pklimk » Thu, Mar 20, 2025, 6:32 PM UTC

Interestingly "replaces every second word with potato" and "speaks in Spanish instead of English" both (kind of) work as a style, so it's clear there's significant flexibility and probably some form of LLM-like thing under the hood.