Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FFmpeg 8.0 adds Whisper support

code.ffmpeg.org

Post reply on HN
Print view View original

301–310 of 341 posts

Re: FFmpeg 8.0 adds Whisper support

#301

Post by forgingahead »

jokethrowaway wrote: post #159
kmfrk wrote: post #93

Whisper is genuinely amazing - with the right nudging. It's the one AI thing that has genuinely turned my life upside-down in an unambiguously good way. People should check out Subtitle Edit (and throw the dev some money) which is a great interface for experimenting with Whisper transcription. It's basically Aegisub 2.0, if you're old, like me. HOWTO: Drop a video or audio file to the right window, then go to Video >…

whisper is definitely nice, but it's a bit too slow. Having subtitles and transcription for everything is great - but Nemo Parakeet (pretty much whisper by nvidia) completely changed how I interact with the computer. It enables dictation that actually works and it's as fast as you can think. I also have a set of scripts which just wait for voice commands and do things. I can pipe the results to an LLM, run commands,…

Yeah, mind sharing any of the scripts? I looked at the docs briefly, looks like we need to install ALL of nemo to get access to Parakeet? Seems ultra heavy.

Re: FFmpeg 8.0 adds Whisper support

#302

Post by spauldo »

gregoryl wrote: post #290
spauldo wrote: post #232

Earlier quoted context omitted.

I have that issue as well - I can hear faint noises OK but if there's background noise I can't understand what people say. But I'm pretty sure there's a physical issue at the root of it in my case. The problem showed up after several practice sessions with a band whose guitarist insisted on always playing at full volume.

I'd love your thoughts on why it might be hardware. I reason that my hearing is generally fine - there's no issue picking apart loud complex music (I love breakcore!). But play two songs at the same time, or try talking to me with significant background noise, and I seem to be distinctly impaired vs. most others. If I concentrate, I can sometimes work through it. My uninformed model is a pipeline of sorts, and some s…

I don't have much beyond what I said. It happened to me after repeated exposure to dangerously loud sounds in a small room. I can hear faint sounds, but I have trouble with strong accents and I can't understand words if there's a lot of background noise. I noticed it shortly after I left that band, and I left because the last practice was so loud it felt like a drill boring into my ears.

I don't think I have any harder time appreciating complex music than I did before, but I'm more of a 60s-70s rock kinda guy and a former bass player, so I tend to focus more on the low end. Bass tends to be less complex because you can't fit as much signal into the waveform without getting unpleasant muddling.

And of course, just because we have similar symptoms doesn't mean the underlying causes are the same. My grandfather was hard of hearing so for all I know it's genetic and the timing was a coincidence. Who knows?

Re: FFmpeg 8.0 adds Whisper support

#303

Post by adi_kurian »

lawik wrote: post #7

I wonder if they'll be satisfied there or add a chunk of others now that they've started. Parakeet is supposed to be good? Should they add Voice Activity Detection? Are these separate filters or just making the whisper filter more fancy?

Parakeet is indeed really awesome.

Re: FFmpeg 8.0 adds Whisper support

#304

Post by kwar13 »

dotancohen wrote: post #185
kwar13 wrote: post #5

Fantastic! I am working on a speech-to-text GNOME extension that would immensely benefit from this. https://github.com/kavehtehrani/gnome-speech2text

Why is this a Gnome extension? I would love to use this in KDE.

I use Ubuntu 24.04 and comes with GNOME Shell.

Re: FFmpeg 8.0 adds Whisper support

#305

Post by sam_lowry_ »

dewey wrote: post #115
dzhiurgis wrote: post #92

Earlier quoted context omitted.

It's just so annyoing how someone like Netflix offers like 3-4 languages for most of its content when you can basically get it for free via browser extensions (if you watch on browser). Must be union thing.

That Netflix who would need to pay more to license more subtitles can't compete with pirated or unlicensed auto-generated subtitles shouldn't really be a surprise. It's also annoying that you have to pay for Netflix when you can get the same movies for free with less restrictions on a pirate site.

You mean, a sharing site? That is a site where someone benevolently shared a movie with me?

Re: FFmpeg 8.0 adds Whisper support

#306

Post by anchpop »

ambicapter wrote: post #87
Lio wrote: post #45

Once local transcription is in more places hopefully we can persuade content creator not to burn bouncing sub-titles into their videos. I've seen professionally produced recordings on dry and technical subjects with good sound quality where they've decided to use distracting sub-titles with no way to disable them. It seems so unnecessary if you're not making novelty videos about cats. Also local transcription allows…

They do that because it increases “engagement”, not because they care about the user’s experience with the subtitles.

I did that (distracting subtitles) on one of my videos and it had a very negative response. I won't do it again, but I was puzzled because I find it much nicer than the traditional subtitle format personally. It's easier for my brain to focus on. (And no one in my test audience minded.)

Re: FFmpeg 8.0 adds Whisper support

#307

Post by rhdunn »

forgingahead wrote: post #301
jokethrowaway wrote: post #159

Earlier quoted context omitted.

whisper is definitely nice, but it's a bit too slow. Having subtitles and transcription for everything is great - but Nemo Parakeet (pretty much whisper by nvidia) completely changed how I interact with the computer. It enables dictation that actually works and it's as fast as you can think. I also have a set of scripts which just wait for voice commands and do things. I can pipe the results to an LLM, run commands,…

Yeah, mind sharing any of the scripts? I looked at the docs briefly, looks like we need to install ALL of nemo to get access to Parakeet? Seems ultra heavy.

You only need the ASR bits -- this is where I got to when I previously looked into running Parakeet:

    # NeMo does not run on 3.13+
    python3.12 -m venv .venv
    source .venv/bin/activate

    git clone https://github.com/NVIDIA/NeMo.git nemo
    cd nemo

    pip install torch torchaudio torchvision --index-url https://download.pytorch.org/whl/cu128
    pip install .[asr]

    deactivate
Then run a transcribe.py script in that venv:

    import os
    import sys
    import nemo.collections.asr as nemo_asr

    model_path = sys.argv[1]
    audio_path = sys.argv[2]

    # Load from a local path...
    asr_model = nemo_asr.models.EncDecRNNTBPEModel.restore_from(restore_path=model_path)

    # Or download from huggingface ('org/model')...
    asr_model = nemo_asr.models.EncDecRNNTBPEModel.from_pretrained(model_name=model_path)

    output = asr_moel.transcribe([audio_path])
    print(output[0])
With that I was able to run the model, but I ran out of memory on my lower-spec laptop. I haven't yet got around to running it on my workstation.

You'll need to modify the python script to process the response and output it in a format you can use.

Re: FFmpeg 8.0 adds Whisper support

#309

Post by radiator »

May I ask, if there is a movie where English people speak English, French people speak French, and German people speak German, is there a software that can generate subtitles in English, French and German without translating anything? I mean, just record what it hears.
Post reply on HN