Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Whisper – open source speech recognition by OpenAI

openai.com

Post reply on HN
Print view View original

31–40 of 508 posts

Re: Whisper – open source speech recognition by OpenAI

#31

Post by TaylorAlexander »

Hey this looks great! I like to record audio notes while driving in my car after work, to kind of decompress my thoughts from the day. But I never go back and listen as they can be long and meandering. Sometimes in the audio log I will sum up my thoughts, but this might be 20 minutes in and hard to find. I really wish I had transcriptions so I could easily scan the full contents. I have tried Mozilla Deepspeech (I don't want a cloud solution) and I was surprised to find that I could not get Deepspeech to reliably transcribe them. There is a bit of road noise, though I think for a human listener they are easy to understand. It looks like this one might actually do the trick!

EDIT: Tried it and it worked great! It is very easy to use. I just did the pip install line in the readme and was ready to go. You literally just run the one pip install line, and then you run the program in the format "whisper my_audio.wav" and it goes. Really nice job OpenAI!

Re: Whisper – open source speech recognition by OpenAI

#32

Post by pen2l »

bambax wrote: post #29
pen2l wrote: post #5

Neat, https://github.com/openai/whisper - they have open-sourced it, even the model weights, so they are living up to their name in this instance. The 4 examples are stunningly good (the examples have speakers with heavy accents, speaking in foreign language, speaking with dynamic background noise, etc.), this is far and away better than anything else I've seen. Will be super curious to see other folks trying it out…

The French version is a little contrived. The speaker is a native speaker, but the text is obviously the result of a translation from English to French, not idiomatic French. I will try to put the code to the test, see how it goes.

Interesting, I'm a non-native French speaker, the original French piece struck me as being entirely normal (but maybe it was just the perfect French accent that swayed me). Can you please point out what he said which wasn't idiomatic or naturally-worded French?

Re: Whisper – open source speech recognition by OpenAI

#33

Post by StevenWaterman »

gok wrote: post #19

Comparing this model's word error rates to the state of the art [1] on a few common test sets: Whisper SoTA LibriSpeech test-clean 2.7% 1.8% LibriSpeech test-other 5.6% 2.9% Switchboard 13.1% 4.9% CallHome 15.8% 9.5% The authors do explicitly state that they're trying to do a lot of fancy new stuff here, like be multilingual, rather than pursuing just accuracy. [1] https://github.com/syhw/wer_are_we

One of the things they point out is that the SoTA on e.g. LibriSpeech is only good at LibriSpeech, and doesn't generalise as well.

> Because Whisper was trained on a large and diverse dataset and was not fine-tuned to any specific one, it does not beat models that specialize in LibriSpeech performance, a famously competitive benchmark in speech recognition. However, when we measure Whisper’s zero-shot performance across many diverse datasets we find it is much more robust and makes 50% fewer errors than those models.

Re: Whisper – open source speech recognition by OpenAI

#34

Post by nik_s »

I just tested the model [1] using an RTX3090, trying to translate a french text I found here [2].

Some observations:

- The full translation of the 6:22 minute video takes about 22 seconds (17x real time)

- It recognizes the language by default (and did a good job to recognize it was french audio)

- MIT License [3]!

- The quality of the transcription is good, but not perfect.

- The quality of the translation (if you don't consider transcription errors as a translation error) is generally very good.

---

The transcription:

> Bonjour à tous, j'suis espère que vous allez bien, c''est ENTI. Et aujourd', aujourd', on se retrouve un peu physique pour parler de la termo dynamique. Vous ne vous inquiétez pas, ça va bien se passer. On va y aller ensemble, être à par exemple, je vous accompagne à travers une série de vidéos pour vous expliquer les principes de base en termo dynamique. Et bah, c''est parti, on va y aller tranquillement. Lidée, c''est vous puissiez comprendre la termo dynamique dans son ensemble. Donc, je vais vraiment prendre mon temps pour couplisser bien comprendre les notions,

The translation:

> Hello everyone, I hope you're doing well, it's NT and today we find ourselves a little physical to talk about the thermo dynamic. Don't worry, it's going well, we're going to go together and be the same. I'm going to accompany you through a series of videos to explain the basic principles in thermo dynamic. Well, let's go, we're going to go quietly. The idea is that you can understand the thermo dynamic in sound together. So I'm really going to take my time to understand the notions,

---

All in all very happy that OpenAI is publishing their models. If Stable Diffusion is any guide, people will hack some crazy things with this.

[1] https://github.com/openai/whisper [2] https://www.youtube.com/watch?v=OFLt-KL0K7Y [3] https://github.com/openai/whisper/blob/main/LICENSE

Re: Whisper – open source speech recognition by OpenAI

#35

Post by emcq »

Be wary of using this model - the licensing of this model seems sketchy. Several of the datasets used for training like WSJ and TED-LIUM have clear non-commercial clauses. I'm not a lawyer but releasing a model as "MIT" seems dubious, and hopefully OpenAI has paid for the appropriate licenses during training as they are no longer a research-only non profit.

Re: Whisper – open source speech recognition by OpenAI

#36

Post by solarmist »

nik_s wrote: post #34

I just tested the model [1] using an RTX3090, trying to translate a french text I found here [2]. Some observations: - The full translation of the 6:22 minute video takes about 22 seconds (17x real time) - It recognizes the language by default (and did a good job to recognize it was french audio) - MIT License [3]! - The quality of the transcription is good, but not perfect. - The quality of the translation (if you d…

Is it translation or transcription? Or both?

Both, wow. This is really interesting.

Re: Whisper – open source speech recognition by OpenAI

#37

Post by TaylorAlexander »

andy_xor_andrew wrote: post #27

Hold on, it does not only speech recognition, but also language translation, in the same model? What an interesting approach. What benefits does this have over having two dedicated models, one for speech-to-text, and another for translation? It just seems so odd, given the problems of speech-to-text and Spanish-to-English seems so different from one another (in terms of the problem domain). Seems so unusual to have b…

It seems these days that language-oriented models are commonly becoming multilingual by default. There are a lot of common threads when understanding sentence construction between different languages. French and English have different rules but they will still have things like nouns, adjectives, subjects, prepositions, etc. It seems that by training models on many languages you get both a more robust understanding of language, and it saves you the trouble of having to make many more localized models for every language. I also believe that the other languages help the models construct sentences in languages which have very small training sets. If it has a few examples in a rare language as well as good translations to a better-known language, then it can provide good support for the rare language.

We also see in image generation models that multi-modal networks are more powerful than single purpose networks. As we move towards more advanced AI systems I suspect we will see more and more generalizable networks with distinct advantages over separate networks that get plugged together.

Re: Whisper – open source speech recognition by OpenAI

#38

Post by zhynn »

TaylorAlexander wrote: post #31

Hey this looks great! I like to record audio notes while driving in my car after work, to kind of decompress my thoughts from the day. But I never go back and listen as they can be long and meandering. Sometimes in the audio log I will sum up my thoughts, but this might be 20 minutes in and hard to find. I really wish I had transcriptions so I could easily scan the full contents. I have tried Mozilla Deepspeech (I do…

I do this too! I have been doing it for about a year now, and haven't ever run into someone else that does this kind of audio-journaling. Would you be up for comparing notes sometime about how it is working out for you? I am finding that it is extremely effective form of self-care, but with lots of personal caveats. I would be so interested to hear your experience.

Re: Whisper – open source speech recognition by OpenAI

#39

Post by StevenWaterman »

nik_s wrote: post #34

I just tested the model [1] using an RTX3090, trying to translate a french text I found here [2]. Some observations: - The full translation of the 6:22 minute video takes about 22 seconds (17x real time) - It recognizes the language by default (and did a good job to recognize it was french audio) - MIT License [3]! - The quality of the transcription is good, but not perfect. - The quality of the translation (if you d…

Was this with the `base` model? `large` is running ok on a P100 in colab, but is about 4% the speed of `base.en`. Certainly seems like some of these models will be fast enough for real-time.

Re: Whisper – open source speech recognition by OpenAI

#40

Post by StevenWaterman »

solarmist wrote: post #36
nik_s wrote: post #34

I just tested the model [1] using an RTX3090, trying to translate a french text I found here [2]. Some observations: - The full translation of the 6:22 minute video takes about 22 seconds (17x real time) - It recognizes the language by default (and did a good job to recognize it was french audio) - MIT License [3]! - The quality of the transcription is good, but not perfect. - The quality of the translation (if you d…

Is it translation or transcription? Or both? Both, wow. This is really interesting.

Both, the blog covers it in detail. Pass in audio in any language, and get an English transcription out.
Post reply on HN