Great to see OpenAI finally being open :)
Whisper – open source speech recognition by OpenAI
81–90 of 508 posts
Re: Whisper – open source speech recognition by OpenAI
#82Earlier quoted context omitted.
This kind of model is harder to abuse, so I guess it passed their internal checks much more easily. I can understand not releasing GPT-3, even if I disagree with the decision.
> I can understand not releasing GPT-3, even if I disagree with the decision. Why do you disagree?
I don’t see how GPT-3 is any more dangerous than Stable Diffusion, Photoshop, that fake news website the crazy person you’re friends with on Facebook really likes, or any of the number of other tools and services that can be used to generate or spread fake information.
Re: Whisper – open source speech recognition by OpenAI
#83I really wish I had this about half a year ago when I was building a tool to automatically turn online school lectures into searchable, clickable transcripts (kind of like YouTube or EdX transcripts). I was originally using Adobe Premiere Pro's speech to text to do it, and wrote Python to convert its output to the Hyperaudio format on GitHub. With this, I can totally skip all of that step and this is fully open sourc…
You could already do the speech recognition in a fully open source way with vosk easily, although Whisper may be more accurate
Re: Whisper – open source speech recognition by OpenAI
#84Be wary of using this model - the licensing of this model seems sketchy. Several of the datasets used for training like WSJ and TED-LIUM have clear non-commercial clauses. I'm not a lawyer but releasing a model as "MIT" seems dubious, and hopefully OpenAI has paid for the appropriate licenses during training as they are no longer a research-only non profit.
This is a big dispute right now: OpenAI and other AI companies generally take the position that models learning from data does not make the output of the models a derivative work of that data. For example, GitHub Co-pilot uses all publicly available GitHub code regardless of license, and DALLE-2/StableDiffusion/etc use lots of non-free images. I don't think this has been challenged in court yet, and I'm very curious…
This is even slightly more direct: access to WSJ data requires paying LDC for the download, and the pricing varies depending on what institution / license you're from. The cost may be a drop in the bucket compared to compute, but I don't know that these licenses are transferable to the end product. We might be a couple court cases away from finding out but I wouldn't want to be inviting one of those cases :)
Re: Whisper – open source speech recognition by OpenAI
#85Neat, https://github.com/openai/whisper - they have open-sourced it, even the model weights, so they are living up to their name in this instance. The 4 examples are stunningly good (the examples have speakers with heavy accents, speaking in foreign language, speaking with dynamic background noise, etc.), this is far and away better than anything else I've seen. Will be super curious to see other folks trying it out…
It was already better. I edit a podcast and have > a decade of pro audio editing experience in the film industry, and I was already using a commercial AI transcription service to render the content to text and sometimes edit it as such (outputting edited audio). Existing (and affordable) offerings are so good that they can cope with shitty recordings off a phone speaker and maintain ~97% accuracy over hour-long conve…
>~97% accuracy over hour-long conversations. I'm sure it's been an absolute godsend for law enforcement
97% accuracy means roughly three or four errors per minute of speech. That seems potentially extremely problematic for something like law enforcement use where decisions with significant impact on people's day and/or life might be made on the basis of "evidence".
Re: Whisper – open source speech recognition by OpenAI
#86Neat, https://github.com/openai/whisper - they have open-sourced it, even the model weights, so they are living up to their name in this instance. The 4 examples are stunningly good (the examples have speakers with heavy accents, speaking in foreign language, speaking with dynamic background noise, etc.), this is far and away better than anything else I've seen. Will be super curious to see other folks trying it out…
Quoted post unavailable.
This seems to not be true for McDonald:
https://www.snopes.com/fact-check/mcdonalds-100-beef/
Re: Whisper – open source speech recognition by OpenAI
#87Earlier quoted context omitted.
This kind of model is harder to abuse, so I guess it passed their internal checks much more easily. I can understand not releasing GPT-3, even if I disagree with the decision.
> I can understand not releasing GPT-3, even if I disagree with the decision. Why do you disagree?
Because why should the wealthy and connected be the only ones -allowed- have access to such life improving technology?
Re: Whisper – open source speech recognition by OpenAI
#88Anyone get it running on m1 mac?
I keep getting `ModuleNotFoundError: No module named 'setuptools.command.build'`
Re: Whisper – open source speech recognition by OpenAI
#89Comparing this model's word error rates to the state of the art [1] on a few common test sets: Whisper SoTA LibriSpeech test-clean 2.7% 1.8% LibriSpeech test-other 5.6% 2.9% Switchboard 13.1% 4.9% CallHome 15.8% 9.5% The authors do explicitly state that they're trying to do a lot of fancy new stuff here, like be multilingual, rather than pursuing just accuracy. [1] https://github.com/syhw/wer_are_we
One of the things they point out is that the SoTA on e.g. LibriSpeech is only good at LibriSpeech, and doesn't generalise as well. > Because Whisper was trained on a large and diverse dataset and was not fine-tuned to any specific one, it does not beat models that specialize in LibriSpeech performance, a famously competitive benchmark in speech recognition. However, when we measure Whisper’s zero-shot performance acr…
My own experience agrees: the generally available "SOTA" models are not especially robust, and can be _extremely_ bad (>50% absolute error rate) at some tasks. I'll post some preliminary numbers in a sibling comment and look into running my full set of tests on Whisper.
It looks like Whisper is probably leaving a lot of accuracy on the table, but initially it does seem to be a lot more robust than general "SOTA" models.
For a quick comparison, Silero's accuracy charts are kind of nice because they post results for a large variety of datasets. Scroll down to the EN V6 xlarge EE model (not the xlarge CE) [1]
[1] https://github.com/snakers4/silero-models/wiki/Quality-Bench...
Re: Whisper – open source speech recognition by OpenAI
#90Hoping to see this out to use in open source voice assistants, eg. mycroft