Live data from Hacker News

Whisper – open source speech recognition by OpenAI

openai.com

271–280 of 508 posts

Re: Whisper – open source speech recognition by OpenAI

#271

Japanese results looks pretty impressive! Took マッコウクジラ14頭が海岸に打ち上げられる オーストラリア(2022年9月21日) https://www.youtube.com/watch?v=bZkNIzeRBk4 Extracted audio with youtube-dl -f bestaudio https://www.youtube.com/watch\?v\=bZkNIzeRBk4 Converted into [00:00.000 --> 00:13.000] オーストラリア南部の島で、真っ向くじら14棟が海岸に打ち上げられて死んでいるのが見つかり、専門家が調査のため原地入りしました。 [00:13.000 --> 00:25.000] 原地メディアによりますと、オーストラリア南部のキング棟で、19日、少なくとも14棟の真っ向くじらが海岸に打ち上げられて死んでいるの…

Shocked at how good the results are, and how easy of an installation it is.

Here are the exact steps to follow to get it running on Ubuntu 22.04 via WSL and yt-dlp:

  1. pip install git+https://github.com/openai/whisper.git

  2. yt-dlp -f 'ba' -x --audio-format mp3 https://www.youtube.com/watch/?v\=bZkNIzeRBk4

  3. renamed the file to test.mp3

  4. whisper test.mp3 --language Japanese --task translate --model large
Note: the large model will download a ~3Gb file

Re: Whisper – open source speech recognition by OpenAI

#275

Does this work with multiple speakers? I want to build a tool that takes a video and generates subtitles for it, then I want to index the subtitles and let people search for a specific quote to scrub to that part of the video using automatically generated urls. This is for a specific fandom of a ton of content, lots of dirty audio mostly recorded in a gym setting with multiple people speaking.

pretty sure such a tool made HN front page a few months ago

Re: Whisper – open source speech recognition by OpenAI

#276
post #5

Neat, https://github.com/openai/whisper - they have open-sourced it, even the model weights, so they are living up to their name in this instance. The 4 examples are stunningly good (the examples have speakers with heavy accents, speaking in foreign language, speaking with dynamic background noise, etc.), this is far and away better than anything else I've seen. Will be super curious to see other folks trying it out…

It was already better. I edit a podcast and have > a decade of pro audio editing experience in the film industry, and I was already using a commercial AI transcription service to render the content to text and sometimes edit it as such (outputting edited audio). Existing (and affordable) offerings are so good that they can cope with shitty recordings off a phone speaker and maintain ~97% accuracy over hour-long conve…

Since you work on podcasts, do any open source transcription tools currently identity the speaker in the output? This would be particularly helpful for interviews.

Re: Whisper – open source speech recognition by OpenAI

#277
post #127

Earlier quoted context omitted.

No it isn't. That just means 2-3% of your content needs to be double-checked by a person at the audio level, saving huge amounts of time - equally true of human transcription, in which individual words are often [UNINTELLIGEBLE]. Would you want to review this fully before going into court, absolutely - because you'd want to play the recording to a jury for emotional impact. Can you rely on it when you want to quickly…

Doesn't it mean 100% of your content needs to be double-checked? You can't easily identify which 2-3% of your content has errors. I'm aware that errors are more likely when the model is less confident of its predictions, but that shouldn't be enough. (edit for clarification: errors are not always something like "[UNINTELLIGIBLE]", where the system knows it doesn't know; they can also be misrecognitions that the syste…

Presumably you can use the 97% that is correctly transcribed to rapidly filter out the relevant content. This is likely to be only a small portion of the total content. Then you check 100% of that.

Re: Whisper – open source speech recognition by OpenAI

#278
post #233

Super impressive. I tested it on a Japanese streamer whose enunciation isn't exactly perfect and it did a decent job: https://www.youtube.com/watch?v=ROiOU1scaNA [00:00.000 --> 00:06.500] Since the last one started, the number of times I've eaten has decreased. [00:06.500 --> 00:11.000] If I get too carried away with the last one, I'll get hungry and do it. [00:11.000 --> 00:14.500] I don't have time to eat. [00:15.5…

It's struggling with Norwegian. Which I guess isn't shocking. The large model performs a fair bit better than the small, though neither is "good".

Though I assume the amount of Norwegian it has been exposed to is fairly limited, so in that light I'm actually impressed as well.

I tried it on a news segment from the radio[1], this is the large model output:

    [00:14.000 --> 00:17.200]  En skamløs krenking av FN pakten.
    [00:17.200 --> 00:24.000]  USAs president og verdensledere svarer på den russiske presidentens atomtrusler og krigsmobilisering.
    [00:25.500 --> 00:29.400]  Arbeidsklær som er ment til å være til begge kjønn, har det med å være tilpasset.
    [00:29.400 --> 00:33.400]  Men hvordan ville det gått, om det var motsatt?
    [00:34.100 --> 00:38.900]  Dyrevernsorganisasjon vil ha digital merking av regnstyr,
    [00:38.900 --> 00:44.900]  men næringen selv insisterer på den gamle tradisjonsrike måten med rissing av kniv.
    [00:45.600 --> 00:51.400]  Mange strømselskaper er positive til å tilby kundene fastpris på strøm, og det årevis.
    [00:51.400 --> 00:59.900]  Da risikerer de å måtte betale mye i nettopp åretsvis, sier aktører som aldri tilbyr fastpris.
    [00:59.900 --> 01:21.900]  Dette er onsdagens Dagsnytten. Jeg heter Espen Ås.
For reference, here's what he actually said, from the source[1] itself:

    * En skamløs krenking av FN-pakten. USAs president og verdensledere svarer på den russiske presidentens atomtrusler og krigsmobilisering.
    * Arbeidsklær som er ment å være til begge kjønn, er som regel tilpasset ... menn. Hvordan hadde det gått om det var motsatt?
    * Dyrevernsoganisasjon vil ha digital merking av reinsdyr, men næringen selv insisterer på den gamle tradisjonsrike måten med rissing av kniv.
    * Mange strømselskaper er positive til å tilby kundene fastpris på strøm - og det i årevis.
    - Da risikerer de å måtte betale mye i nettopp; årevis, sier aktør som aldri tilbyr fastpris
    Dette er onsdagens Dagsnytt 18 - jeg heter Espen Aas.
The translation didn't fare that well though:

    [00:14.000 --> 00:17.000]  A shameless violation of the UN treaty.
    [00:17.000 --> 00:24.000]  The US president and world leaders respond to the Russian president's nuclear threats and war mobilization.
    [00:24.000 --> 00:33.000]  Work clothes that are meant to be for both genders have to be suitable, but how would it be if it was the other way around?
    [00:34.000 --> 00:44.000]  The animal welfare organization will have a digital marking of reindeer, but the industry itself insists on the old traditional way of tearing a knife.
    [00:45.000 --> 00:51.000]  Many electricity companies are positive in offering customers fixed electricity prices, and that is annual.
    [00:51.000 --> 00:58.000]  Then they risk having to pay a lot in just a year, says an actor who has never offered fixed prices.
    [00:58.000 --> 01:20.000]  This is Wednesday's Dagsnytt 18. My name is Espen Ås.
For reference, here's Google Translate's attempt, which is pretty good:

    * A shameless violation of the UN Charter. The US president and world leaders respond to the Russian president's nuclear threats and war mobilization.
    * Work clothes intended for both sexes are usually adapted to ... men. How would it have gone if it had been the other way around?
    * Animal welfare organizations want digital marking of reindeer, but the industry itself insists on the old, traditional way of marking with a knife.
    * Many electricity companies are positive about offering customers a fixed price for electricity - and for years.
    - Then they risk having to pay a lot in precisely; for years, says a player who never offers a fixed price
    This is Wednesday's Dagsnytt 18 - my name is Espen Aas.

[1]: https://radio.nrk.no/podkast/dagsnytt_atten/l_5ce3e323-97a3-... (not sure if it's available outside of Norway)

Re: Whisper – open source speech recognition by OpenAI

#279

For those on NixOS, here's a quick and dirty flake.nix that will let you make a venv in which to "pip install"' Just put it in a flake.nix, and "nix develop" followed by "virtualenv ./venv; . ./venv/bin/activate; pip install git+ https://github.com/openai/whisper.git " { description = "Python 3.9 development environment"; outputs = { self, nixpkgs }: let system = "x86_64-linux"; pkgs = import nixpkgs { inherit system…

This should, in theory, work with CUDA; my GPU doesn't have enough RAM to do it (it runs out at 2.9GiB allocated, I have 4GiB, but am running a compositing desktop, which chews up about 600MiB; not sure where the other ~400MiB went) [edit] I confirmed CUDA worked with the "small" model, which used 3.3GB of GPU ram, and resulted in much poorer recognition than the "medium" model on my CPU (but it ran at least two orde…

CUDA worked fine with large on my 2080Ti FWIW. The speedup is ridiculous, as expected. My Ryzen 3800X used almost an hour transcribing a minute worth of speech, while the 2080Ti does it in like 10-20 seconds.

Re: Whisper – open source speech recognition by OpenAI

#280
I tried running it in realtime with live audio input (kind of).

If you want to give it a shot, you can find the python script in this repo: https://github.com/tobiashuttinger/openai-whisper-realtime

A bit more context on how it works: The systems default audio input is captured with python, split into small chunks and is then fed to OpenAI's original transcription function. It tries (currently rather poorly) to detect word breaks and doesn't split the audio buffer in those cases. With how the model is designed, it doesn't make the most sense to do this, but i found it would be worth trying. It works acceptably well.

Post reply on HN