Live data from Hacker News

Whisper – open source speech recognition by OpenAI

openai.com

201–210 of 508 posts

Re: Whisper – open source speech recognition by OpenAI

#201

Just tested this on some developer podcasts which usually fail hard given they're full of technical jargon, brand names, etc. Whisper is a revolution! It's picking up terms like Heroku, DigitalOcean, GitHub, ECS, AWS, etc. and capitalizing properly - something nothing else did unless you provided a whole pile of guiding vocabulary.

Did these podcasts have transcripts? You might be inadvertently evaluating it on data that it was trained on, which is basically cheating. Even if not, it might be trained on similar podcasts. Judging how good these kinds of models are is really hard.

Re: Whisper – open source speech recognition by OpenAI

#202
This is absolute garbage python as I am neither a python developer, nor a good developer. I was trying to play around with real time transcriptions. However, it does work!

> * recording * done recording Recording saved to file.wav Press enter to transcribe

/Users/laptop/Development/Personal/Public/pythonProject1/venv/lib/python3.9/site-packages/whisper/transcribe.py:70: UserWarning: FP16 is not supported on CPU; using FP32 instead warnings.warn("FP16 is not supported on CPU; using FP32 instead") Detected language: english Goodbye, I need to go pick up my wife. Press enter to start recording

Any improvements welcome here.

``` # This is a sample Python script.

# Press ⌃R to execute it or replace it with your code. # Press Double ⇧ to search everywhere for classes, files, tool windows, actions, and settings.

def print_hi(name): # Use a breakpoint in the code line below to debug your script. print(f'Hi, {name}') # Press ⌘F8 to toggle the breakpoint.

def record_microphone(seconds): import pyaudio import wave

    CHUNK = 1024
    FORMAT = pyaudio.paInt16
    CHANNELS = 1
    RATE = 44100
    RECORD_SECONDS = seconds
    WAVE_OUTPUT_FILENAME = "file.wav"

    p = pyaudio.PyAudio()

    stream = p.open(format=FORMAT,
                    channels=CHANNELS,
                    rate=RATE,
                    input=True,
                    frames_per_buffer=CHUNK)

    print("* recording")

    frames = []

    for i in range(0, int(RATE / CHUNK * RECORD_SECONDS)):
        data = stream.read(CHUNK)
        frames.append(data)

    print("* done recording")

    stream.stop_stream()
    stream.close()
    p.terminate()

    wf = wave.open(WAVE_OUTPUT_FILENAME, 'wb')
    wf.setnchannels(CHANNELS)
    wf.setsampwidth(p.get_sample_size(FORMAT))
    wf.setframerate(RATE)
    wf.writeframes(b''.join(frames))
    wf.close()

    return WAVE_OUTPUT_FILENAME



if __name__ == '__main__': seconds = 5 while True: print("Press enter to start recording") input() filename = record_microphone(seconds) print("Recording saved to " + filename) print("Press enter to transcribe") input() import whisper model = whisper.load_model("base")

        result = model.transcribe(filename)
        print(result["text"])

```

Re: Whisper – open source speech recognition by OpenAI

#203

Earlier quoted context omitted.

17x realtime on a 3090 I did some basic tests on CPU, the "small" Whisper model is in the ballpark of 0.5x realtime, which is probably not great for interactive use. My models in Talon run closer to 100x realtime on CPU.

“CPU” isn’t necessarily the benchmark, though. Most smartphones going back years have ML inference accelerators built in, and both Intel and AMD are starting to build in instructions to accelerate inference. Apple’s M1 and M2 have the same inference accelerator hardware as their phones and tablets. The question is whether this model is a good fit for those inference accelerators, and how well it works there, or how w…

See https://news.ycombinator.com/item?id=32929029 re accuracy, I'm working on a wider comparison. My models are generally more robust than open-source models such as Vosk and Silero, but I'm definitely interested in how my stuff compares to Whisper on difficult held-out data.

> Brute forcing the model with just traditional CPU instructions is fine, but… obviously going to be pretty slow.

It's not that simple. Many of the mobile ML accelerators are more targeted for conv net image workloads, and current-gen Intel and Apple CPUs have dedicated hardware to accelerate matrix math (which helps quite a bit here, and these instructions were in use in my tests).

Also, not sure which model they were using at 17x realtime on the 3090. (If it's one of the smaller models, that bodes even worse for non-3090 performance.) The 3090 is one of the fastest ML inference chips in the world, so it doesn't necessarily set realistic expectations.

There are also plenty of optimizations that aren't applied to the code we're testing, but I think it's fairly safe to say the Large model is likely to be slow on anything but a desktop-gpu-class accelerator just due to the sheer parameter size.

Re: Whisper – open source speech recognition by OpenAI

#204

Earlier quoted context omitted.

No it isn't. That just means 2-3% of your content needs to be double-checked by a person at the audio level, saving huge amounts of time - equally true of human transcription, in which individual words are often [UNINTELLIGEBLE]. Would you want to review this fully before going into court, absolutely - because you'd want to play the recording to a jury for emotional impact. Can you rely on it when you want to quickly…

I think an [UNINTELLIGIBLE] indication would be a great addition to automatic transcription systems.

It'd [UNINTELLIGIBLE score="92%" alternatives="pro-rabble; pourable"]probably[/UNINTELLIGIBLE] be useful to make a markup-based output... though you'd probably find it gave you more info than you wanted.

Re: Whisper – open source speech recognition by OpenAI

#205
post #10

It seems like OpenAI are finally living up to their name for once with this release? Anything I'm missing? From what I can gather: 1. Includes model weights. I can't find the URL, but they reference them enough and have a CLI tool, so I presume I just haven't found them yet. 2. Includes code: https://github.com/openai/whisper 3. Released under MIT License: https://github.com/openai/whisper/blob/main/LICENSE

It's one model and in a non-strategic area where there are existing open source projects (Kaldi, DeepSpeech, ...).

For a company that raised $1B, that's not exactly living up to their name and original mission.

Re: Whisper – open source speech recognition by OpenAI

#206
post #169
post #163

This really makes me want to build a Amazon Echo/Google Nest/etc replacement that's open hardware, open source and most importantly recognises voice completely offline. I find that I don't use these smart devices for much more than setting timers anyway so this seems like an easy project. I just wonder what system requirements Whisper has and whether there are open source voice recognition models that are specificall…

This is only one side of the coin, you still need really good models for Speech Synthesis and then be able to have it all working in almost real time, ideally locally on device.

As far as TTS goes, Mycroft.ai[0] has released a decent offline one.

[0]https://mycroft.ai/

Re: Whisper – open source speech recognition by OpenAI

#208
post #7

AI speech recognition FN scares the heck out of me... for so many reasons. But one that really pisses me off is not being able to turn it off on the iphone, and the fact that aside from "hidden cameras in my airBnB" -- soon we will have to worry about secret listening machines EVERYWHERE

We will see an explosion of AI capabilities in the next couple of years. This will have a huge impact on our lives, much of it good but some of it also bad.

Re: Whisper – open source speech recognition by OpenAI

#209
Japanese results looks pretty impressive!

Took マッコウクジラ14頭が海岸に打ち上げられる オーストラリア(2022年9月21日) https://www.youtube.com/watch?v=bZkNIzeRBk4

Extracted audio with youtube-dl -f bestaudio https://www.youtube.com/watch\?v\=bZkNIzeRBk4

Converted into [00:00.000 --> 00:13.000] オーストラリア南部の島で、真っ向くじら14棟が海岸に打ち上げられて死んでいるのが見つかり、専門家が調査のため原地入りしました。 [00:13.000 --> 00:25.000] 原地メディアによりますと、オーストラリア南部のキング棟で、19日、少なくとも14棟の真っ向くじらが海岸に打ち上げられて死んでいるのが見つかりました。 [00:25.000 --> 00:31.000] ほとんどが若いオーストを見られ、専門家が現場に重むき調査に当たっています。 [00:31.000 --> 00:41.000] くじらの死害は大きく運んだり埋めたりすることが難しいため、自然に分解されるのを待つ方針が検討されています。 [00:41.000 --> 00:52.000] また、死害を狙い、サメが海に集まる可能性があるとして、原地東局はサーファーなどに周囲に近づかないように呼びかけています。 [00:52.000 --> 01:02.000] 一方、21日にはタスマニア棟でおよそ230棟のくじらが浜辺に打ち上げられた状態で見つかりました。 [01:02.000 --> 01:07.000] およそ半数がまだ生きている模様で急助活動が進められています。 [01:07.000 --> 01:23.000] 見つかったのは、ゴンドーくじらの仲間と見られています。

Re: Whisper – open source speech recognition by OpenAI

#210
post #127

Earlier quoted context omitted.

No it isn't. That just means 2-3% of your content needs to be double-checked by a person at the audio level, saving huge amounts of time - equally true of human transcription, in which individual words are often [UNINTELLIGEBLE]. Would you want to review this fully before going into court, absolutely - because you'd want to play the recording to a jury for emotional impact. Can you rely on it when you want to quickly…

Doesn't it mean 100% of your content needs to be double-checked? You can't easily identify which 2-3% of your content has errors. I'm aware that errors are more likely when the model is less confident of its predictions, but that shouldn't be enough. (edit for clarification: errors are not always something like "[UNINTELLIGIBLE]", where the system knows it doesn't know; they can also be misrecognitions that the syste…

You double check things that you think are important, in this case, passages that will be used as evidence in court.
Post reply on HN