Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Audio Models

openai.fm

Post reply on HN
Print view View original

181–190 of 317 posts

Re: OpenAI Audio Models

#181

Post by taf2 »

ekzy wrote: post #178
jeffharris wrote: post #18

Hey, I'm Jeff and I was PM for these models at OpenAI. Today we launched three new state-of-the-art audio models. Two speech-to-text models—outperforming Whisper. A new TTS model—you can instruct it how to speak (try it on openai.fm!). And our Agents SDK now supports audio, making it easy to turn text agents into voice agents. We think you'll really like these models. Let me know if you have any questions here!

Do you know when we can expect an update on the realtime API? It’s still in beta and there are many issues (e.g voice randomly cutting off, VAD issues, especially with mulaw etc…) which makes it impossible to use in production, but there’s not much communication from OpenAI. It’s difficult to know what to bet on. Pushing for stt->llm->tts makes you wonder if we should carry on building with the realtime API.

Agreed- really not liking how they are neglecting it… I hope they are just hard at work behind the scenes and will release something soon

Re: OpenAI Audio Models

#182

Post by wewewedxfgdf »

jeffharris wrote: post #88
dandiep wrote: post #80

Earlier quoted context omitted.

1) Previous TTS models had problems with major problems accents. E.g. a Spanish sentence could drift from a Spain accent to Mexican to American all within one sentence. Has this been improved and/or is it still a WIP? 2) What is the latency? 3) Your STT API/Whisper had MAJOR problems with hallucinating things the user didn't say. Is this fixed? 4) Whisper and your audio models often auto corrected speech, e.g. if som…

1/ we've been working a lot on accents, so expect improvements with these models... though we're not done. Would be curious how you find them. And try giving specific detailed instructions + examples for the accents you want 2/ We're doing everything we can to make it fast. Very critical that it can stream audio meaningfully faster than realtime 3+4/ I wouldn't call hallucinations "solved", but it's been the central…

As mentioned in another comment, the British accents are very far from being authentic.

Re: OpenAI Audio Models

#183

Post by Foreignborn »

hnhn34 wrote: post #159
kuprel wrote: post #152

Earlier quoted context omitted.

They released a crippled model: https://github.com/SesameAILabs/csm/issues/63

The good news is Orpheus-3B just made Sesame essentially obsolete.

thanks for this, it sounds pretty good.

link for anyone else: https://canopylabs.ai/model-releases

Re: OpenAI Audio Models

#184

Post by a-r-t »

ekzy wrote: post #173
a-r-t wrote: post #95

Earlier quoted context omitted.

Hi Jeff, are there any plans to support dual-channel audio recordings (e.g., Twilio phone call audio) for speech-to-text models? Currently, we have to either process each channel separately and lose conversational context, or merge channels and lose speaker identification.

I’m not entirely sure what you mean but twilio recordings supports dual channels already

Transcribing Twilio's dual-channel recordings using OpenAI's speech-to-text while preserving channel identification.

Re: OpenAI Audio Models

#185

Post by com2kid »

benjismith wrote: post #49

If I'm reading the pricing correctly, these models are SIGNIFICANTLY cheaper than ElevenLabs. https://platform.openai.com/docs/pricing If these are the "gpt-4o-mini-tts" models, and if the pricing estimate of "$0.015 per minute" of audio is correct, then these prices 85% cheaper than those of ElevenLabs. https://elevenlabs.io/pricing With ElevenLabs, if I choose their most cost-effectuve "Business" plan for $1100 per…

Elevenlabs is an ecosystem play. They have hundreds of different voices, legally licensed from real people who chose to upload their voice. It is a marketplace of voices.

None of the other major players is trying to do that, not sure why.

Re: OpenAI Audio Models

#186

Post by clbrmbr »

crazygringo wrote: post #75

This is astonishing. I can type anything I want into the "vibe" box and it does it for the given text. Accents, attitudes, personality types... I'm amazed. The level of intelligent "prosody" here -- the rhythm and intonation, the pauses and personality -- I wasn't expecting anything like this so soon. This is truly remarkable. It understands both the text and the prompt for how the speaker should sound. Like, we're g…

I got one German “w” when using the following prompt, but most of the “w” were still pronounced as liquids rather than labial fricatives.

> Speak with an exaggerated German accent, pronouncing all “w” as “v”

Re: OpenAI Audio Models

#187

Post by ekzy »

a-r-t wrote: post #184
ekzy wrote: post #173

Earlier quoted context omitted.

I’m not entirely sure what you mean but twilio recordings supports dual channels already

Transcribing Twilio's dual-channel recordings using OpenAI's speech-to-text while preserving channel identification.

Oh I see what you mean that would be a neat feature. Assuming you can get timestamps though it should be trivial to work around the issue?

Re: OpenAI Audio Models

#188

Post by zhyder »

alach11 wrote: post #168

It's interesting that they pitch this for agent development. The realtime API provides a much simpler architecture for developing agents. Why would you want to string together STT -> LLM -> TTS when you could have a consolidated model doing all three steps? They alluded to there being some quality/intelligence benefits to the multi-step approach, but in the long-run I'd expect them to improve the realtime API to make…

Text allows developers lots for flexibility to do other processing, including RAG, calling APIs yourself and multiple chained LLM invocations. The low latency of realtime API means relying fully on one invocation of their model to do everything.

Re: OpenAI Audio Models

#189

Post by dietr1ch »

taf2 wrote: post #180
jeffharris wrote: post #18

Hey, I'm Jeff and I was PM for these models at OpenAI. Today we launched three new state-of-the-art audio models. Two speech-to-text models—outperforming Whisper. A new TTS model—you can instruct it how to speak (try it on openai.fm!). And our Agents SDK now supports audio, making it easy to turn text agents into voice agents. We think you'll really like these models. Let me know if you have any questions here!

Please release a stable realtime speech to speech model. The current version constantly thinks it’s a young teen heading to college and sad but then suddenly so excited about it

can't wait for scam calls after this gets perfected

Re: OpenAI Audio Models

#190

Post by kibbi »

dharmab wrote: post #169
kibbi wrote: post #141

Large text-to-speech and speech-to-text models have been greatly improving recently. But I wish there were an offline , on-device, multilingual text-to-speech solution with good voices for a standard PC — one that doesn't require a GPU, tons of RAM, or max out the CPU. In my research, I didn't find anything that fits the bill. People often mention Tortoise TTS, but I think it garbles words too often. The only plug-in…

I use Piper for one of my apps. It runs on CPU and doesn't require a GPU. It will run well on a raspberry pi. I found a couple of permissively licensed voices that could handle technical terms without garbling them. However, it is unmaintained and the Apple Silicon build is broken. My app also uses whisper.cpp. It runs in real time on Apple Sillicon or on modern fast CPUs like AMD's gaming CPUs.

I had already suspected that I hadn't found all the possibilities regarding Tortoise TTS, Coqui, Piper, etc. It is sometimes difficult to determine how good a TTS framework really is.

Do you possibly have links to the voices you found?

Post reply on HN