Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Audio Models

openai.fm

Post reply on HN
Print view View original

191–200 of 317 posts

Re: OpenAI Audio Models

#191

Post by alach11 »

zhyder wrote: post #188
alach11 wrote: post #168

It's interesting that they pitch this for agent development. The realtime API provides a much simpler architecture for developing agents. Why would you want to string together STT -> LLM -> TTS when you could have a consolidated model doing all three steps? They alluded to there being some quality/intelligence benefits to the multi-step approach, but in the long-run I'd expect them to improve the realtime API to make…

Text allows developers lots for flexibility to do other processing, including RAG, calling APIs yourself and multiple chained LLM invocations. The low latency of realtime API means relying fully on one invocation of their model to do everything.

The realtime API can be used to call tools [0], but I agree with your general point on the flexibility of working directly with text.

[0] https://github.com/openai/openai-realtime-agents

Re: OpenAI Audio Models

#192

Post by zhyder »

jeffharris wrote: post #18

Hey, I'm Jeff and I was PM for these models at OpenAI. Today we launched three new state-of-the-art audio models. Two speech-to-text models—outperforming Whisper. A new TTS model—you can instruct it how to speak (try it on openai.fm!). And our Agents SDK now supports audio, making it easy to turn text agents into voice agents. We think you'll really like these models. Let me know if you have any questions here!

How is the latency (Time To First Byte of audio, when streaming) and throughput (non-vibe characters input per second) compared to the existing 'tts-1' non-HD that's the same price? TTFB in particular is important and needs to be much better than 'tts-1'.

Re: OpenAI Audio Models

#193

Post by ZeroTalent »

kibbi wrote: post #156
ZeroTalent wrote: post #144

Earlier quoted context omitted.

Look into https://superwhisper.com and their local models. Pretty decent.

Thank you, but they say "Offline models only run really well on Apple Silicon macs."

Many SOTA apps are, unfortunately, only for Apple M Macs.

Re: OpenAI Audio Models

#194

Post by lukebuehler »

setsewerd wrote: post #170
lukebuehler wrote: post #145

Earlier quoted context omitted.

Convert any file (pdf, epub, txt) to an audoibook, downloadable as mp3, or directly listenable via RSS feed in, say, Apple Potcasts app. Basically make one-off audiobooks for yourself or a few friends.

Any plans to make a Chrome extension variant? Been looking for a high quality and cheap TTS extension for ages (like ElevenLabs Human Reader, except with less absurd pricing)

I din't think of that, interesting idea. What I'm focusing right now is long-form content for more offline-ish listening, but maybe a plugin could work to load longer texts, but I'm not working on a screen reader atm.

Re: OpenAI Audio Models

#195

Post by lukebuehler »

lherron wrote: post #118
lukebuehler wrote: post #56

Earlier quoted context omitted.

yes, I think you are right. When I did the math on 11labs million chars I got the same numbers (Pro plan). I'm super happy about this, since I took a bet that exactly this would happen. I've just been building a consumer TTS app that could only work with significant cheaper TTS prices per million character (or self-hosted models)

Kokoro TTS is pretty good for open source. Worth checking out.

Yes, kokoro is great, and the language flexibility is a huge plus too. And the best prices per character is for sure if you self-host.

Re: OpenAI Audio Models

#196

Post by buybackoff »

I was experimenting recently with voiceover TTS generation. Did run Kokoro TTS locally and it's magical for how few resources it takes (runs fine in a browser), but only the default female voices (Heart/Bella) are usable, and very good. Then I found that Clipchamp has it built-in and several voices from a big selection there are very good, and free. I've listened to this OpenAI TTS and I could not like them at all even compared to Kokoro.

Re: OpenAI Audio Models

#197

Post by Etheryte »

jeffharris wrote: post #18

Hey, I'm Jeff and I was PM for these models at OpenAI. Today we launched three new state-of-the-art audio models. Two speech-to-text models—outperforming Whisper. A new TTS model—you can instruct it how to speak (try it on openai.fm!). And our Agents SDK now supports audio, making it easy to turn text agents into voice agents. We think you'll really like these models. Let me know if you have any questions here!

After toying around with the TTS model it seems incredibly nondeterministic. Running the same input with the same parameters can have widely different results, some really good, others downright bad. The tone, intonation and character all vary widely. While some of the outputs are great, this inconsistency makes it a really tough sell. Imagine if Siri responded to you with a different voice every time, as an example. Is this something you're looking to address somewhere down the line or do you consider that working as intended?

Re: OpenAI Audio Models

#198

Post by sandspar »

Foreignborn wrote: post #183
hnhn34 wrote: post #159

Earlier quoted context omitted.

The good news is Orpheus-3B just made Sesame essentially obsolete.

thanks for this, it sounds pretty good. link for anyone else: https://canopylabs.ai/model-releases

These voices are all annoying, though. The thing about Sesame's Miles is that he's cool.

Re: OpenAI Audio Models

#199

Post by wholinator2 »

lukebuehler wrote: post #194
setsewerd wrote: post #170

Earlier quoted context omitted.

Any plans to make a Chrome extension variant? Been looking for a high quality and cheap TTS extension for ages (like ElevenLabs Human Reader, except with less absurd pricing)

I din't think of that, interesting idea. What I'm focusing right now is long-form content for more offline-ish listening, but maybe a plugin could work to load longer texts, but I'm not working on a screen reader atm.

Do you know if there's any offerings today that can read math? Like speak an equation the way a human would? It's something I've been thinking about a long time and would be an essential feature for me (the only things i read are physics)
Post reply on HN