Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Audio Models

openai.fm

Post reply on HN
Print view View original

231–240 of 317 posts

Re: OpenAI Audio Models

#231

Post by jbaudanza »

dandiep wrote: post #80
jeffharris wrote: post #18

Hey, I'm Jeff and I was PM for these models at OpenAI. Today we launched three new state-of-the-art audio models. Two speech-to-text models—outperforming Whisper. A new TTS model—you can instruct it how to speak (try it on openai.fm!). And our Agents SDK now supports audio, making it easy to turn text agents into voice agents. We think you'll really like these models. Let me know if you have any questions here!

1) Previous TTS models had problems with major problems accents. E.g. a Spanish sentence could drift from a Spain accent to Mexican to American all within one sentence. Has this been improved and/or is it still a WIP? 2) What is the latency? 3) Your STT API/Whisper had MAJOR problems with hallucinating things the user didn't say. Is this fixed? 4) Whisper and your audio models often auto corrected speech, e.g. if som…

3) Whisper really needs to be paired with Silero VAD, otherwise the hallucination problem makes it almost unusable.

Re: OpenAI Audio Models

#232

Post by anigbrowl »

crazygringo wrote: post #75

This is astonishing. I can type anything I want into the "vibe" box and it does it for the given text. Accents, attitudes, personality types... I'm amazed. The level of intelligent "prosody" here -- the rhythm and intonation, the pauses and personality -- I wasn't expecting anything like this so soon. This is truly remarkable. It understands both the text and the prompt for how the speaker should sound. Like, we're g…

Can't say I'm enthused about another novel technological way to destroy the living of people who work in the arts.

Re: OpenAI Audio Models

#233

Post by diggan »

stavros wrote: post #220
lherron wrote: post #118

Earlier quoted context omitted.

Kokoro TTS is pretty good for open source. Worth checking out.

Oh man, they have the "Sky" voice, and it seems to be the same one that OpenAI had but then removed? Not sure how that's possible, but I'm very happy about it.

> Not sure how that's possible

Download bunch of movies Scarlet Johansen been in, segment into audio clips where she talks and train the model :)

Re: OpenAI Audio Models

#234

Post by dharmab »

kibbi wrote: post #190
dharmab wrote: post #169

Earlier quoted context omitted.

I use Piper for one of my apps. It runs on CPU and doesn't require a GPU. It will run well on a raspberry pi. I found a couple of permissively licensed voices that could handle technical terms without garbling them. However, it is unmaintained and the Apple Silicon build is broken. My app also uses whisper.cpp. It runs in real time on Apple Sillicon or on modern fast CPUs like AMD's gaming CPUs.

I had already suspected that I hadn't found all the possibilities regarding Tortoise TTS, Coqui, Piper, etc. It is sometimes difficult to determine how good a TTS framework really is. Do you possibly have links to the voices you found?

Here's my code! https://github.com/dharmab/skyeye/tree/main/pkg/synthesizer

Re: OpenAI Audio Models

#235

Post by anigbrowl »

lukeinator42 wrote: post #217
forgotpasagain wrote: post #68

It sounds very expressive but weirdly "fake" as if it's targeting to be similar to some NPC character, dataset issue?

yeah it's almost like an uncanny valley where it sometimes feels like the voice trying to be an actor and play a character or something

I mean that's literally the service it's providing. If you asked humans to do the same thing it would sound equally forced. All acting sounds cringe out of context.

Re: OpenAI Audio Models

#236

Post by anigbrowl »

kgeist wrote: post #174

In Russian, OpenAI audio models usually have a slight American (?) accent. The intonation and the phonetics fall into the uncanney valley. Does the same happen in other languages?

The Japanese sounds OK to me. Not 100% but better than most human speakers. I understand Japanese well enough to be able to pick up a few different foreign accents in that language.

Re: OpenAI Audio Models

#237

Post by youssefabdelm »

youssefabdelm wrote: post #138
jeffharris wrote: post #51

Earlier quoted context omitted.

We're thinking about diarization (adding time awareness to GPT models) but no firm plans to share just yet

Jeff you know what would be magical? Not just vanilla diarization "Speaker 1" and "2" but if the model can know from the conversation this speaker was referred to as "Jeff Harris" or "Jeff" so it uses that instead.

Or if we could even provide samples of what an example speaker sounds like in general so that it would always classify them the way we want.

Re: OpenAI Audio Models

#238

Post by anigbrowl »

RobinL wrote: post #105

I'm surprised at how poor this is at following a detailed prompt. It seems capable of generating a consistent style, and so in that sense quite useful. But if you want (say) a regional UK accent it's not even close. I also find it confusing you have to choose a voice. Surely that's what the prompt should be for, especially when the voices have such abstract names. I mean, it's still very impressive when you stand bac…

That seems way overwritten. Try something like 'Jolly old-fashioned rural farmer, Somerset.'

Re: OpenAI Audio Models

#239

Post by anigbrowl »

smokeydoe wrote: post #161

Does anyone know of any decent newer open source models for generating sound effects?

Just use a synthesizer. Writing textual prompts is about the most inefficient way of getting what you want. When I was working in film I'd tell directors to stop describing what they had in mind (unless they were referencing something very specific) and try just making some funny mouth noises.
Post reply on HN