Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Whisper – open source speech recognition by OpenAI

openai.com

Post reply on HN
Print view View original

411–420 of 508 posts

Re: Whisper – open source speech recognition by OpenAI

#411

Post by Tistron »

It understands my Swedish attempts at English really well with the medium.en model. (Although, it gives me a funny warning: `UserWarning: medium.en is an English-only model but receipted 'English'; using English instead.`. I guess it doesn't want to be told to use English when that's all it can do.)

However, it runs very slowly. It uses the CPU on my macbook, presumably because it hasn't got a NVidia card.

Googling about that I found [plaidML](https://github.com/plaidml/plaidml) which is a project promising to run ML on many different gpu architectures. Does anyone know whether it is possible to plug them together somehow? I am not an ML researcher, and don't quite understand anything about the technical details of the domain, but I can understand and write python code in domains that I do understand, so I could do some glue work if required.

Re: Whisper – open source speech recognition by OpenAI

#412

Post by nl »

shpx wrote: post #259

We shouldn't call this open source. The model definition + the data is the source code. The model weights are a compilation artifact. > The source code must be the preferred form in which a programmer would modify the program. [...] Intermediate forms such as the output of a preprocessor or translator are not allowed. > https://opensource.org/osd If I asked a programmer from OpenAI to modify the model to better suppo…

This isn't really true.

You can do a lot with weights and no training data - for example you can pull the end layer off it and use it as a feature extractor.

And to modify it for Japanese speakers you'd fine train the existing model on additional data. If you wanted to modify the model you can (sometimes, depending on what you want to do) modify an existing architecture by removing layers, adding replacements and fine tuning.

I don't quite know what the right analogy of trained data is. In many ways it is more valuable than the training data because the compute needed to generate it is significant. In other ways it is nice to be able to inspect the data.

> The source code must be the preferred form in which a programmer would modify the program.

As a machine learning programmer I'd much prefer the weights than the raw data. It's no realistic for me to use that training data in any way with any compute I have access to.

Re: Whisper – open source speech recognition by OpenAI

#413

Post by MayeulC »

dom96 wrote: post #231
solarkraft wrote: post #216

Earlier quoted context omitted.

Are you thinking about reimplementing Mycroft? The Mycroft has done a lot of cool and important work in the field to ship an actual personal assistant product (stuff like wake word detection).

hah, of course someone had the idea already and executed on it. But yeah, basically that but without the screen (probably would go a long way to decrease the cost, $299 is pretty steep for such a device)

Well, you can always install Mycroft on a Pi, or on your computer.

Almond is also interesting as a voice assistant, though I think it doesn't perform speech recognition itself.

Re: Whisper – open source speech recognition by OpenAI

#414

Post by capableweb »

bruckie wrote: post #384
capableweb wrote: post #262

Earlier quoted context omitted.

Is that application actually doing on-device transcription? Under "Data safety" on the Google Play page it says "This app may share these data types with third parties: Audio" which doesn't exactly instill confidence that my audio will 100% always stay on my device. It also says "Data is encrypted in transit" but if data stays on the device, why it has to be "encrypted in transit"? There should be no transit at all.

Yes, it works completely offline, including transcription and recognition of music. There's an optional cloud sync feature, which I assume is the reason for the notice on Google Play. (Work for Google, don't speak for them.)

Thanks. Whose the third party that might get access to the audio? First party would be me, second party would be Google and then the third?

Re: Whisper – open source speech recognition by OpenAI

#415

Post by MayeulC »

forgingahead wrote: post #374
NaturalPhallacy wrote: post #348

Earlier quoted context omitted.

It was doing it slowly , but hadn't got to the insane bit when I killed it to try and get it working with CUDA, so I had to do some digging and it turns out I need a version of pytorch with CUDA enabled, and so I had to go and install Anaconda, and now now conda is stuck trying to "solve" my environment to install pytorch with CUDA. So...probably? Pre-post edit: I can't get it to work. I've installed pytorch with cud…

Welcome to every single Python ML project - dependency hell will quickly kill any enthusiasm one may have for trying out projects. It really feels archaic to have these issues with such cutting edge technology.

You can blame CUDA quite a bit for that. Proprietary, you need to sort out which driver you need, plus an nvidia GPU...

I tried compiling pytorch with vulkan support, but there are a few LDFLAGS that are wrong. I'll try to solve that some time later.

One piece of advice: use distribution packages! Arch provides pytorch-cuda, and has PKGBUILDS as well.

For reproductibility, I wish we were all on Nix/Guix, but that's not the case (and CUDA+HW dependency would make it complicated).

Re: Whisper – open source speech recognition by OpenAI

#417

Post by danso »

This is an astonishing package. Every AI voice-to-text model I've tried on "The Wire's" famous "fuck" scene [0] usually fails, because the youtube clip's audio quality is bad and it's a scene with virtually no dialogue except breathing and "Fuck". But Whisper returned impressive results [1]

[0] https://www.youtube.com/watch?v=DS6pE88Xg3s

[1]

    $ yt-dlp --extract-audio --audio-format mp3 -o wire-fuck.mp3 https://www.youtube.com/watch?v=DS6pE88Xg3s

    $ whisper --language en wire-fuck.mp3
    [00:00.000 --> 00:02.000]  Oh
    [00:13.260 --> 00:15.260]  Fuck
    [00:15.260 --> 00:31.260]  Motherfucker
    [00:50.700 --> 00:52.700]  Fuck
    [00:52.700 --> 00:58.700]  Oh
    [00:58.700 --> 01:10.700]  Fuck
    [01:28.700 --> 01:55.900]  Fuck
    [02:02.340 --> 02:03.700]  Motherfuck.
    [02:10.220 --> 02:11.220]  Oh, fuck.
    [02:11.780 --> 02:12.780]  Oh, fuck.
    [02:25.900 --> 02:27.900]  Fuck, fuck, fuck, fuck, fuck, fuck.
    [02:27.900 --> 02:28.900]  Motherfucker.
    [02:32.900 --> 02:33.900]  Oh, fuck.
    [02:34.900 --> 02:35.900]  Fuck.
    [02:35.900 --> 02:36.900]  Oh, fuck.
    [02:36.900 --> 02:37.900]  Oh, fuck.
    [02:37.900 --> 02:38.900]  Oh, fuck.
    [02:48.900 --> 02:49.900]  Motherfucker.
    [02:53.900 --> 02:54.900]  Fucking A.
    [02:54.900 --> 02:56.900]  Mm hmm.
    [02:56.900 --> 03:12.900]  Fuck.
    [03:26.900 --> 03:28.900]  Motherfucker.
    [03:28.900 --> 03:32.900]  Fuck me.
    [03:58.900 --> 04:01.900]  Oh.
    [04:28.900 --> 04:34.900]  Fuck.

Re: Whisper – open source speech recognition by OpenAI

#418

Post by adeptima »

NaturalPhallacy wrote: post #317
gzer0 wrote: post #271

Earlier quoted context omitted.

Shocked at how good the results are, and how easy of an installation it is. Here are the exact steps to follow to get it running on Ubuntu 22.04 via WSL and yt-dlp: 1. pip install git+https://github.com/openai/whisper.git 2. yt-dlp -f 'ba' -x --audio-format mp3 https://www.youtube.com/watch/?v\=bZkNIzeRBk4 3. renamed the file to test.mp3 4. whisper test.mp3 --language Japanese --task translate --model large Note: the…

I did something similar (my ytdl is ytdlp too). You don't even have to grab just the audio, it'll take a webm: https://i.imgur.com/03UFGc8.gif Amazing work.

cause ffmpeg inside

https://github.com/openai/whisper/blob/main/requirements.txt

should process most formats

Re: Whisper – open source speech recognition by OpenAI

#419

Post by adeptima »

knaik94 wrote: post #255

I got a super weird results with the 'medium' and language Japanese (with a --task translate). The song is False Sympathy by Mondo Grosso. "[01:17.000 --> 01:32.000] Translated by Releska" when using the translate to english. That entire part of the song is instrumental. This line does not appear at all in the original transcribe only in the opus format rip. It shows up in the yt rip in format 251 (opus), but not in…

Japanese output will produce lot of tiny mistakes. However the whole output is still good enough. Like 95% plus good enough.

Found lot mistakes in 3-4 characters kanji ... and I guess most native Japanese will do mistakes time to time too, and this is why they pop up lot of buzzwords on screen with all kind of highlighting to avoid double guessing.

Post reply on HN