Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Audio Models

openai.fm

Post reply on HN
Print view View original

141–150 of 317 posts

Re: OpenAI Audio Models

#141

Post by kibbi »

Large text-to-speech and speech-to-text models have been greatly improving recently.

But I wish there were an offline, on-device, multilingual text-to-speech solution with good voices for a standard PC — one that doesn't require a GPU, tons of RAM, or max out the CPU.

In my research, I didn't find anything that fits the bill. People often mention Tortoise TTS, but I think it garbles words too often. The only plug-in solution for desktop apps I know of is the commercial and rather pricey Acapela SDK.

I hope someone can shrink those new neural network–based models to run efficiently on a typical computer. Ideally, it should run at under 50% CPU load on an average Windows laptop that’s several years old, and start speaking almost immediately (less than 400ms delay).

The same goes for speech-to-text. Whisper.cpp is fine, but last time I looked, it wasn't able to transcribe audio at real-time speed on a standard laptop.

I'd pay for something like this as long as it's less expensive than Acapela.

(My use case is an AAC app.)

Re: OpenAI Audio Models

#142

Post by GordonS »

wewewedxfgdf wrote: post #111
jeffharris wrote: post #93

Earlier quoted context omitted.

try the ballad or fable voices

Doesn't really sound very British to be honest. Sounds kinda international/like an American trying to do a British accent. I've been looking for real TTS British accents so this product doesn't meet my goals.

Azure TTS has some great British accents - I used a British female voice for a demo video voice over, and the quality was great. Not as good as ElevenLabs, but I was still really impressed with the final result.

Re: OpenAI Audio Models

#143

Post by MrAssisted »

eob wrote: post #128
MrAssisted wrote: post #81

Earlier quoted context omitted.

I hope they find a more unique product offering that takes hold. Everybody thinks of them as text-to-speech but I use ElevenLabs exclusively for speech-to-speech for vtubing as my AI character. They're kind of the only game in town for doing super high quality speech-to-speech (unless someone here has an alternative which I'd LOVE to know about). I've tried https://github.com/w-okada/voice-changer which is great beca…

Are you comfortable sharing the video & lip-sync stack you use? I don't know anything about the space but am curious to check out what's possible these days.

For my last video I used https://github.com/warmshao/FasterLivePortrait with a png of the character on my RTX 3090 desktop and recorded the output of that real-time but in the next video I'm going to spin up a runpod instance and do the FasterLivePortrait in the cloud after the fact because then I can get a smooth 60fps which looks better. I think the only real-time cloud way to do AI vtubing in the cloud is my own GenDJ project (fork of https://github.com/kylemcdonald/i2i-realtime but tweaked for cloud real-time) but that just doesn't look remotely as good as LivePortrait. Somebody needs to rip out and replace insightface in FasterLivePortait (it's prohibited for commercial use) and fork https://github.com/GenDJ to have the runpod it spins up run the de-insightfaced LivePortrait instead of i2i-realtime. I'll probably get around to doing that in the next few months if nobody else does and nothing else comes along and makes LivePortrait obsolete (both are big ifs).

AIWarper recently released a simpler way to run FasterLivePortrait for vtubing purposes https://huggingface.co/AIWarper/WarpTuber but I haven't tried it yet because I already have my own working setup and as I mentioned I'm shifting my workload for that to the cloud anyways

Re: OpenAI Audio Models

#144

Post by ZeroTalent »

kibbi wrote: post #141

Large text-to-speech and speech-to-text models have been greatly improving recently. But I wish there were an offline , on-device, multilingual text-to-speech solution with good voices for a standard PC — one that doesn't require a GPU, tons of RAM, or max out the CPU. In my research, I didn't find anything that fits the bill. People often mention Tortoise TTS, but I think it garbles words too often. The only plug-in…

Look into https://superwhisper.com and their local models. Pretty decent.

Re: OpenAI Audio Models

#145

Post by lukebuehler »

zacmps wrote: post #84
lukebuehler wrote: post #56

Earlier quoted context omitted.

yes, I think you are right. When I did the math on 11labs million chars I got the same numbers (Pro plan). I'm super happy about this, since I took a bet that exactly this would happen. I've just been building a consumer TTS app that could only work with significant cheaper TTS prices per million character (or self-hosted models)

What does it do?

Convert any file (pdf, epub, txt) to an audoibook, downloadable as mp3, or directly listenable via RSS feed in, say, Apple Potcasts app.

Basically make one-off audiobooks for yourself or a few friends.

Re: OpenAI Audio Models

#146

Post by 5kg »

kibbi wrote: post #141

Large text-to-speech and speech-to-text models have been greatly improving recently. But I wish there were an offline , on-device, multilingual text-to-speech solution with good voices for a standard PC — one that doesn't require a GPU, tons of RAM, or max out the CPU. In my research, I didn't find anything that fits the bill. People often mention Tortoise TTS, but I think it garbles words too often. The only plug-in…

May I introduce to you

https://huggingface.co/canopylabs/orpheus-3b-0.1-ft

(no affiliation)

it's English only afaics.

Re: OpenAI Audio Models

#147

Post by Sohcahtoa82 »

jeffharris wrote: post #23
Etheryte wrote: post #7

Recommended input for anyone trying this out: Voice: Onyx Vibe: Heavy german accent, doing an Arnold Schwarzenegger impression, way over the top for comedic effect. Deep booming voice, uses pauses for dramatic effect.

so good! https://www.openai.fm/#28540f27-5b51-445a-b1d6-1c89711a2c4f

I hit Play 3 times and got 3 very different results.

One merely sounded like it had a slight German accent, once just sounded kind of raspy, and the third sound like a normal American English speaker.

Re: OpenAI Audio Models

#148

Post by modeless »

jeffharris wrote: post #18

Hey, I'm Jeff and I was PM for these models at OpenAI. Today we launched three new state-of-the-art audio models. Two speech-to-text models—outperforming Whisper. A new TTS model—you can instruct it how to speak (try it on openai.fm!). And our Agents SDK now supports audio, making it easy to turn text agents into voice agents. We think you'll really like these models. Let me know if you have any questions here!

Whisper's major problem was hallucinations, how are the new models doing there? The performance of ChatGPT advanced voice in recognizing speech is, frankly, terrible. Are these models better than what's used there?

Re: OpenAI Audio Models

#149

Post by infecto »

simonw wrote: post #121
jeffharris wrote: post #51

Earlier quoted context omitted.

We're thinking about diarization (adding time awareness to GPT models) but no firm plans to share just yet

The feature I want is speaker differentiation - I want to feed in an audio file and get back a transcript with "Speaker 1: ..., Speaker 2: ..." indications. That plus timestamps would be incredible. The Google Gemini 2.0 models are showing some promise with this, I can't speak to their reliability just yet though.

I thought Deepgram already did speaker diarization (which is differentiation) pretty well. That and it can include timestamps plus other metadata.

Re: OpenAI Audio Models

#150

Post by dougiejones »

Etheryte wrote: post #7

Recommended input for anyone trying this out: Voice: Onyx Vibe: Heavy german accent, doing an Arnold Schwarzenegger impression, way over the top for comedic effect. Deep booming voice, uses pauses for dramatic effect.

My recommendation for Ash:

Delivery: Cow noises. You are actually a cow. You can only moo and grunt. No human noises. Only moo. No words.

Pauses: Moo and grunt between sentences. Some burps and farts.

Tone: Cow.

Post reply on HN