Live data from Hacker News

OpenAI Audio Models

openai.fm

241–250 of 317 posts

Re: OpenAI Audio Models

#241
I just tested the "gpt-4o-mini-tts" model on several texts in Japanese, a particularly challenging language for TTS because many character combinations are read differently depending on the context. The produced speech was quite good, with natural intonation and pronunciation. There were, however, occasional glitches, such the word 現在 genzai “now, present” read with a pause between the syllables (gen ... zai) and the conjunction 而も read nadamo instead of the correct shikamo. There were also several places where the model skipped a word or two.

However, unlike some other TTS models offering Japanese support that have been discussed here recently [1], I think this new offering from OpenAI is good enough for language users. I certainly could have put it to good use when I was studying Japanese many years ago. But it’s not quite ready for public-facing applications such as commercial audiobooks.

That said, I really like the ability to instruct the model on how to read the text. In that regard, my tests in both English and Japanese went well.

[1] https://news.ycombinator.com/item?id=42968893

Re: OpenAI Audio Models

#242
post #80

Earlier quoted context omitted.

1) Previous TTS models had problems with major problems accents. E.g. a Spanish sentence could drift from a Spain accent to Mexican to American all within one sentence. Has this been improved and/or is it still a WIP? 2) What is the latency? 3) Your STT API/Whisper had MAJOR problems with hallucinating things the user didn't say. Is this fixed? 4) Whisper and your audio models often auto corrected speech, e.g. if som…

3) Whisper really needs to be paired with Silero VAD, otherwise the hallucination problem makes it almost unusable.

100% and I’ve done this, but it’s still there.

Re: OpenAI Audio Models

#243

Hey, I'm Jeff and I was PM for these models at OpenAI. Today we launched three new state-of-the-art audio models. Two speech-to-text models—outperforming Whisper. A new TTS model—you can instruct it how to speak (try it on openai.fm!). And our Agents SDK now supports audio, making it easy to turn text agents into voice agents. We think you'll really like these models. Let me know if you have any questions here!

[dead]

Re: OpenAI Audio Models

#244
post #181

Earlier quoted context omitted.

Agreed- really not liking how they are neglecting it… I hope they are just hard at work behind the scenes and will release something soon

S2S is where we're investing the most effort on audio ... sorry it's been slow but we are working hard on it Top priorities at the moment 1) Better function calling performance 2) Improved perception accuracy (not mishearing) 3) More reliable instruction following 4) Bug fixes (cutoffs, run ons, modality steering)

Appreciate the efforts. It’s not there yet, but when it gets there it will open up a lot of use cases.

Any fine tuning for s2s in the horizon?

Re: OpenAI Audio Models

#245
post #37

Earlier quoted context omitted.

Are these models downloadable, like whisper? What’s the minimum hardware for running them? Would they run on a raspberry pi? Or a smartphone?

not open source at this time. unfortunately they're much to large to run on normal consumer hardware

I guess you are aware of this, but just in case: some of us rely on dictation in our daily computer usage (think people with disabilities or pain problems). A MacBook Pro with M4 Max and 64GB of RAM could easily run something much larger than Whisper Large (around 3GB).

I would love a larger, better Whisper for use in the MacWhisper dictation app.

Re: OpenAI Audio Models

#246
post #187
post #184

Earlier quoted context omitted.

Transcribing Twilio's dual-channel recordings using OpenAI's speech-to-text while preserving channel identification.

Oh I see what you mean that would be a neat feature. Assuming you can get timestamps though it should be trivial to work around the issue?

There are two options that I know of:

1. Merge both channels into one (this is what Whisper does with dual-channel recordings), then map transcription timestamps back to the original channels. This works only when speakers don't talk over each other, which is often not the case.

2. Transcribe each channel separately, then merge the transcripts. This preserves perfect channel identification but removes valuable conversational context (e.g., Speaker A asks a question, Speaker B answers incomprehensively) that helps model's accuracy.

So yes, there are two technically trivial solutions, but you either get somewhat inaccurate channel identification or degraded transcription quality. A better solution would be a model trained to accept an additional token indicating the channel ID, preserving it in the output while benefiting from the context of both channels.

Re: OpenAI Audio Models

#247
post #95

Earlier quoted context omitted.

Hi Jeff, are there any plans to support dual-channel audio recordings (e.g., Twilio phone call audio) for speech-to-text models? Currently, we have to either process each channel separately and lose conversational context, or merge channels and lose speaker identification.

this has been coming up often recently. nothing to announce yet, but when enough developers ask for it, we'll build it into the model's training diarization is also a feature we plan to add

Glad to hear it's on your radar. I'd imagine phone call transcription is a significant use case.

Re: OpenAI Audio Models

#248

Hey, I'm Jeff and I was PM for these models at OpenAI. Today we launched three new state-of-the-art audio models. Two speech-to-text models—outperforming Whisper. A new TTS model—you can instruct it how to speak (try it on openai.fm!). And our Agents SDK now supports audio, making it easy to turn text agents into voice agents. We think you'll really like these models. Let me know if you have any questions here!

Hi Jeff, I have an app that already supports the Whisper API, so I added the GPT4o models as options. I noticed that the GPT4o models don't support prompting, and as a result my app had a higher error rate in practice when using GPT4o compared to Whisper. Is prompting on the roadmap?

Re: OpenAI Audio Models

#249
post #84

Earlier quoted context omitted.

What does it do?

Convert any file (pdf, epub, txt) to an audoibook, downloadable as mp3, or directly listenable via RSS feed in, say, Apple Potcasts app. Basically make one-off audiobooks for yourself or a few friends.

For anyone else reading this, librera reader + sherpaTTS are both FOSS android apps and can read anything librera can open on an ad-hoc basis, with no need to futz with files, just load your ebook bookmark and hit play.

SherpaTTS has a bunch of different models (piper/coqui) with a ton of voices/languages. There's a slight but tolerable delay with piper high models but low is realtime.

Re: OpenAI Audio Models

#250
Hi! Can you add prefix support? This would be very valuable in being able to support overlapping windows. The only other way would be to use another ai to determine the overlap
Post reply on HN