Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Whisper – open source speech recognition by OpenAI

openai.com

Post reply on HN
Print view View original

141–150 of 508 posts

Re: Whisper – open source speech recognition by OpenAI

#141

Post by selfmodruntime »

thfuran wrote: post #85
anigbrowl wrote: post #74

Earlier quoted context omitted.

It was already better. I edit a podcast and have > a decade of pro audio editing experience in the film industry, and I was already using a commercial AI transcription service to render the content to text and sometimes edit it as such (outputting edited audio). Existing (and affordable) offerings are so good that they can cope with shitty recordings off a phone speaker and maintain ~97% accuracy over hour-long conve…

>~97% accuracy over hour-long conversations. I'm sure it's been an absolute godsend for law enforcement 97% accuracy means roughly three or four errors per minute of speech. That seems potentially extremely problematic for something like law enforcement use where decisions with significant impact on people's day and/or life might be made on the basis of "evidence".

I've worked with similar technology in the law enforcement space and the software is never used to make decisions. You can make out critical timestamps in conversations and a law enforcement officer will always manually confirm the software's assessments.

Re: Whisper – open source speech recognition by OpenAI

#142

Post by neongreen »

Kuinox wrote: post #96
The5thElephant wrote: post #26

How is it Apple, Google, or Microsoft are not further ahead of the game on speech recognition like this? They have the resources to hire the best ML researchers and throw tons of computing hours at it, yet Siri, Google, and Cortana continue to struggle to get anywhere near this level of comprehension.

OpenAI is owned by Microsoft FYI.

Is it? Googling suggests that Microsoft invested in OpenAI but doesn’t actually own it.

Re: Whisper – open source speech recognition by OpenAI

#143

Post by lunixbochs »

mmh0000 wrote: post #131

Okay this is super impressive. I just downloaded Whisper and fed it a random flac file I had handy and it did a really good job. Also impressive that it works on my weak CPU: A 3m07s flac took 5m to transcribe: $ whisper --device cpu 'BLACKPINK - BORN PINK/01 Pink Venom.flac' Detecting language using up to the first 30 seconds. Use `--language` to specify the language Detected language: korean [00:00.000 --> 00:10.00…

Looks like it defaults to the model called "small".

I just ran some benchmarks - M1 Max, pytorch, with a 1.29 second flac (looks like the matrix math was running on a single thread):

    tiny
    146.522ms detect_lang
    549.131ms decode_one
    0.057ms tokenizer

    base
    354.885ms detect_lang
    1046.679ms decode_one
    0.011ms tokenizer

    small
    803.892ms detect_lang
    3194.503ms decode_one
    0.017ms tokenizer

    medium
    2279.689ms detect_lang
    10128.255ms decode_one
    0.023ms tokenizer

    large
    3656.478ms detect_lang
    17249.024ms decode_one
    0.016ms tokenizer

Re: Whisper – open source speech recognition by OpenAI

#145

Post by seszett »

nik_s wrote: post #34

I just tested the model [1] using an RTX3090, trying to translate a french text I found here [2]. Some observations: - The full translation of the 6:22 minute video takes about 22 seconds (17x real time) - It recognizes the language by default (and did a good job to recognize it was french audio) - MIT License [3]! - The quality of the transcription is good, but not perfect. - The quality of the translation (if you d…

> dans son ensemble

> in sound together

That's hilarious and honestly, incredibly bad. "Dans son ensemble" is a very common idiom (meaning "as a whole") while "in sound together" has to be pretty rare. "Son" means "his/hers/its" as well as "sound", and the former meaning is probably more common in general so I have no idea how this result could arise.

"Termo" also doesn't exist in French, it's "thermo", so the transcript even makes orthographic errors.

And I forgot about "couplisser" which is also a hilarious made-up word that sounds like it could mean something, but doesn't! Edit Google finds exactly one reference of this, in a patent with a typo on the word "coulisser".

I'm still impressed by the transcript quality since it covers many languages, but the translation part is quite poor.

Re: Whisper – open source speech recognition by OpenAI

#146

Post by nshm »

goffi wrote: post #128

Really interesting, I can see ton of potential uses. 2 questions: 1) how does it compare to state of the art FOSS solutions? I'm seeking about DeepSpeech or Vosk 2) would it be somehow possible to associate timestamp to the words recognized? That would be amazing for things such as audio editing or skipping to a particular location on a video

You properly mentioned timestamps. There are many other important properties of good ASR system like vocabulary adaptability (if you can introduce new words) or streaming. Or confidences. Or latency of the output. Compared to Vosk models this model can not work in streaming manner, so not very suitable for real-time applications.

But in general the model is robust and accurate and trained on the amount of speech we never dreamed about in Vosk. We will certainly benefit from this model as a teacher (together with others like gigaspeech models). I recently wrote about it https://alphacephei.com/nsh/2022/06/14/voting.html

Re: Whisper – open source speech recognition by OpenAI

#147

Post by deegles »

anigbrowl wrote: post #74
pen2l wrote: post #5

Neat, https://github.com/openai/whisper - they have open-sourced it, even the model weights, so they are living up to their name in this instance. The 4 examples are stunningly good (the examples have speakers with heavy accents, speaking in foreign language, speaking with dynamic background noise, etc.), this is far and away better than anything else I've seen. Will be super curious to see other folks trying it out…

It was already better. I edit a podcast and have > a decade of pro audio editing experience in the film industry, and I was already using a commercial AI transcription service to render the content to text and sometimes edit it as such (outputting edited audio). Existing (and affordable) offerings are so good that they can cope with shitty recordings off a phone speaker and maintain ~97% accuracy over hour-long conve…

There's already software that can imitate a person's voice, so we have all the pieces already to do speech-to-text, clean up with GPT-3, and back to text-to-speech in the original person's voice. Maybe with a style transfer to keep the person's inflections etc the same?

Re: Whisper – open source speech recognition by OpenAI

#148

Post by petercooper »

Just tested this on some developer podcasts which usually fail hard given they're full of technical jargon, brand names, etc. Whisper is a revolution! It's picking up terms like Heroku, DigitalOcean, GitHub, ECS, AWS, etc. and capitalizing properly - something nothing else did unless you provided a whole pile of guiding vocabulary.

Re: Whisper – open source speech recognition by OpenAI

#149

Post by coder543 »

fxtentacle wrote: post #114
The5thElephant wrote: post #26

How is it Apple, Google, or Microsoft are not further ahead of the game on speech recognition like this? They have the resources to hire the best ML researchers and throw tons of computing hours at it, yet Siri, Google, and Cortana continue to struggle to get anywhere near this level of comprehension.

This AI has a 30 second delay on the audio processing because it needs to be able to "look into the future" to get these good results. That 30s delay would be unacceptable for Siri/Google/Cortana.

A lot of models we currently use seem to do the same thing. The model will transcribe a "best effort" interpretation in real time, then as you can continue speaking, you'll see it go back and make corrections. I'm sure you can feed the first X seconds you have into the model, followed by (30-X) seconds of silence, and it will do real time transcription just fine... it would be weird if this broke anything. Then, as you get more speech, you continue getting better transcription of the first 30 seconds, then you switch to a 30 second sliding window.

Maybe I'm missing something, but I don't see the problem here.

Post reply on HN