Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Audio Models

openai.fm

Post reply on HN
Print view View original

221–230 of 317 posts

Re: OpenAI Audio Models

#221

Post by edwinarbus »

mazd wrote: post #175
jeffharris wrote: post #18

Hey, I'm Jeff and I was PM for these models at OpenAI. Today we launched three new state-of-the-art audio models. Two speech-to-text models—outperforming Whisper. A new TTS model—you can instruct it how to speak (try it on openai.fm!). And our Agents SDK now supports audio, making it easy to turn text agents into voice agents. We think you'll really like these models. Let me know if you have any questions here!

The Realtime API via WebRTC sample code for transcription is erroring. Could you take a look into this?

Just to confirm, is it this sample under Connection details? https://platform.openai.com/docs/guides/realtime?text-genera...

Re: OpenAI Audio Models

#222

Post by maest »

MrAssisted wrote: post #143
eob wrote: post #128

Earlier quoted context omitted.

Are you comfortable sharing the video & lip-sync stack you use? I don't know anything about the space but am curious to check out what's possible these days.

For my last video I used https://github.com/warmshao/FasterLivePortrait with a png of the character on my RTX 3090 desktop and recorded the output of that real-time but in the next video I'm going to spin up a runpod instance and do the FasterLivePortrait in the cloud after the fact because then I can get a smooth 60fps which looks better. I think the only real-time cloud way to do AI vtubing in the cloud is my own G…

Do you mind sharing your yt account? If you are okay with linking it to your hn account. I'd quite like to see the results.

Re: OpenAI Audio Models

#223

Post by claiir »

jeffharris wrote: post #18

Hey, I'm Jeff and I was PM for these models at OpenAI. Today we launched three new state-of-the-art audio models. Two speech-to-text models—outperforming Whisper. A new TTS model—you can instruct it how to speak (try it on openai.fm!). And our Agents SDK now supports audio, making it easy to turn text agents into voice agents. We think you'll really like these models. Let me know if you have any questions here!

Hi Jeff. This is awesome. Any plans to add word timestamps to the new speech-to-text models, though?

> Other parameters, such as timestamp_granularities, require verbose_json output and are therefore only available when using whisper-1.

Word timestamps are insanely useful for large calls with interruptions (e.g. multi-party debate/Twitter spaces), allowing transcript lines to be further split post-transcription on semantic boundaries rather than crude VAD-detected silence. Without timestamps it’s near-impossible to make intelligible two paragraphs from Speaker 1 and Speaker 2 with both interrupting each other without aggressively partitioning source audio pre-transcription—which severely degrades transcript quality, increases hallucination frequency and still doesn’t get the same quality as word timestamps. :)

Re: OpenAI Audio Models

#224

Post by noosphr »

jeffharris wrote: post #18

Hey, I'm Jeff and I was PM for these models at OpenAI. Today we launched three new state-of-the-art audio models. Two speech-to-text models—outperforming Whisper. A new TTS model—you can instruct it how to speak (try it on openai.fm!). And our Agents SDK now supports audio, making it easy to turn text agents into voice agents. We think you'll really like these models. Let me know if you have any questions here!

Having read the docs - used chat gpt to summarize them - there is no mention of speaker diarization for these models.

This is a _very_ low hanging fruit anyone with a couple of dgx h100 servers can solve in a month and is a real world problem that needs solving.

Right now _no_ tools on the market - paid or otherwise - can solve this with better than 60% accuracy. One killer feature for decision makers is the ability to chat with meetings to figure out who promised what, when and why. Without speaker diarization this only reliably works for remote meetings where you assume each audio stream is a separate person.

In short: please give us a diarization model. It's not that hard - I've done it one for a board of 5, with a 4090 over a weekend.

Re: OpenAI Audio Models

#225

Post by stavros »

jeffharris wrote: post #18

Hey, I'm Jeff and I was PM for these models at OpenAI. Today we launched three new state-of-the-art audio models. Two speech-to-text models—outperforming Whisper. A new TTS model—you can instruct it how to speak (try it on openai.fm!). And our Agents SDK now supports audio, making it easy to turn text agents into voice agents. We think you'll really like these models. Let me know if you have any questions here!

Hey Jeff! Are there any plans to offer the Sky voice again?

Re: OpenAI Audio Models

#227

Post by kibbi »

wingworks wrote: post #207
kibbi wrote: post #141

Large text-to-speech and speech-to-text models have been greatly improving recently. But I wish there were an offline , on-device, multilingual text-to-speech solution with good voices for a standard PC — one that doesn't require a GPU, tons of RAM, or max out the CPU. In my research, I didn't find anything that fits the bill. People often mention Tortoise TTS, but I think it garbles words too often. The only plug-in…

Did you try Kokoro? You can self host that. https://huggingface.co/spaces/hexgrad/Kokoro-TTS

Thanks! But I get the impression that with Kokoro, a strong CPU still requires about two seconds to generate one sentence, which is too much of a delay for a TTS voice in an AAC app.

I'd rather accept a little compromise regarding the voice and intonation quality, as long as the TTS system doesn't frequently garble words. The AAC app is used on tablet PCs running from battery, so the lower the CPU usage and energy draw, the better.

Re: OpenAI Audio Models

#228

Post by [deleted] »

com2kid wrote: post #185
benjismith wrote: post #49

If I'm reading the pricing correctly, these models are SIGNIFICANTLY cheaper than ElevenLabs. https://platform.openai.com/docs/pricing If these are the "gpt-4o-mini-tts" models, and if the pricing estimate of "$0.015 per minute" of audio is correct, then these prices 85% cheaper than those of ElevenLabs. https://elevenlabs.io/pricing With ElevenLabs, if I choose their most cost-effectuve "Business" plan for $1100 per…

Elevenlabs is an ecosystem play. They have hundreds of different voices, legally licensed from real people who chose to upload their voice. It is a marketplace of voices. None of the other major players is trying to do that, not sure why.

[deleted]

Re: OpenAI Audio Models

#229

Post by stavros »

pier25 wrote: post #45
jtbayly wrote: post #40

I don’t get it. These voices all have a not-so-subtle vibration in them that makes them feel worse than Siri to me. I was expecting a lot better.

yeah the voices sound terrible I'm guessing their spectral generator is super low res to save on resources

Is there a way to pay for higher quality? I don't see a way to pay at all, this just works without an API key, even with the generated code. I agree though, these voices sound like their buffer is always underrunning.
Post reply on HN