Live data from Hacker News

FFmpeg 8.0 adds Whisper support

code.ffmpeg.org

241–250 of 341 posts

Re: FFmpeg 8.0 adds Whisper support

#241

Earlier quoted context omitted.

A good subtitle isn't a perfect copy of what was said.

Hard disagree. When I'm reading a transcript, I want word-for-word what the people said, not a creative edit. I want the speakers' voice, not the transcriptionist's. And when I'm watching subtitles in my own language (say because I want the volume low so I'm not disturbing others), I hate when the words I see don't match the words I hear. It's the quickest way I can imagine to get sucked out of the content and into a…

> When I'm reading a transcript

That's the thing though, subtitles aren't intended as full transcripts. They are intended to allow a wide variety of people to follow the content.

A lot of people read slower than they would hear speech. So subtitles often need to condense or rephrase speech to keep pace with the video. The goal is usually to convey meaning clearly within the time available on screen. Not to capture every single word.

If they tried to be fully verbatim, you'd either have subtitles disappearing before most viewers could finish reading them or large blocks of text covering the screen. Subtitlers also have to account for things like overlapping dialogue, filler words, and false starts, which can make exact transcriptions harder to read and more distracting in a visual medium.

I mean, yeah in your own native language I agree it sort of sucks if you can still hear the spoken words as well. But, to be frank, you are also the minority group here as far as subtitle target audiences go.

And to be honest, if they were fully verbatim, I'd wager you quickly would be annoyed as well. Simply because you will notice how much attention they then draw, making you less able to actually view the content.

Re: FFmpeg 8.0 adds Whisper support

#242
post #224

Why would one use FFmpeg with Whisper support, instead of using Whisper directly?

At least whisper.cpp only supports a few input formats like WAV and MP3. To get subtitles for videos I always have to first run ffmpeg to get an audio file and then run whisper.cpp. Guess this new feature may mean that I can do it in just one step, so slightly more convenient?

I see, thanks. I actually do almost all my Whisper work with ogg files, and got into a snag recently with m4a files. Transcoding to an equivalent size ogg or mp3 killed the quality, and wav is too big. Maybe FFmpeg could be of service here.

Re: FFmpeg 8.0 adds Whisper support

#243

Does this have the ability to edit historic words as more info becomes available? Eg. If I say "I scream", it sounds phonetically identical to "Ice cream". Yet the transcription of "I scream is the best dessert" makes a lot less sense than "Ice cream is the best dessert". Doing this seems necessary to have both low latency and high accuracy, and things like transcription on android do that and you can see the adjusti…

A good opportunity to point people to the paper with my favorite title of all time: "How to wreck a nice beach you sing calm incense" https://dl.acm.org/doi/10.1145/1040830.1040898

Also relevant: The Two Ronnies - "Four Candles"

https://www.youtube.com/watch?v=gi_6SaqVQSw

Re: FFmpeg 8.0 adds Whisper support

#244
post #4

Shut off the broken bot filter so we can read it please

They don't need to shut off Anubis, they just need to configure it beyond the defaults. If they turned on the meta-refresh based challenge then all browsers could access it while still keeping most of the bots away. But few people ever configure these things and just accept the broken defaults. With the current broken default config my browser can't even run the JS challenge due to it using unsupported bleeding edge…

Hi, can you please paste the error message you get? This should be using features that are supported widely as of 2022 and I regularly test on Firefox LTS.

Re: FFmpeg 8.0 adds Whisper support

#245
I hope this is the start of more ML filters in ffmpeg. They added the sr (super resolution) filter years ago, but it's old and it's difficult to get the weights so you can run it, since they're not included. They have added support for multiple inference libraries like libtorch, but again, it's difficult to even get started. Hopefully they can get behind a consistent ML strategy, ideally with a "models" directory with ready to use models for upscaling, temporal upscaling, noise cancelling, etc. A lot of audio and video filter research use ML now, new codecs will probably also use it soon.

Re: FFmpeg 8.0 adds Whisper support

#246
post #176

Earlier quoted context omitted.

Whisper is excellent, but not perfect. I used Whisper last week to transcribe a phone call. In the transcript, the name of the person I was speaking with (Gem) was alternately transcribed as either "Jim" or "Jem", but never "Gem."

That's at least as good as a human, though. Getting to "better-than-human" in that situation would probably require lots of potentially-invasive integration to allow the software to make correct inferences about who the speakers are in order to spell their names correctly, or manually supplying context as another respondent mentioned.

When she told me her name, I didn't ask her to repeat it, and I got it right through the rest of the call. Whisper didn't, so how is this "at least s good as a human?"

Re: FFmpeg 8.0 adds Whisper support

#247

Earlier quoted context omitted.

I had similar thoughts when reading Huck Finn. It's not just phonetically spelled, it's much different. Almost like Twain came up with a list of words, and then had a bunch of 2nd graders tell him the spelling of words they had seen. I guess at some point, you just get good at bad spelling?

Writing in the vernacular, I believe it's called. I do something like that if I'm texting. The book "Feersum Endjinn" by Iain M. Banks uses something like this for one of its characters to quite good effect.

Except it forces me to slow down to "decypher" the text and makes the reading labored. I understand the point as it is part of the character, but it is easier to understand someone speaking in that vernacular vs reading the forced misspellings. I definitely don't want to get to the point of being good at reading it though. I wonder if this is how second grade teachers feel reading the class' schoolwork?

Re: FFmpeg 8.0 adds Whisper support

#248
post #176

Earlier quoted context omitted.

That's at least as good as a human, though. Getting to "better-than-human" in that situation would probably require lots of potentially-invasive integration to allow the software to make correct inferences about who the speakers are in order to spell their names correctly, or manually supplying context as another respondent mentioned.

When she told me her name, I didn't ask her to repeat it, and I got it right through the rest of the call. Whisper didn't, so how is this "at least s good as a human?"

I wouldn't expect any transcriber to know that the correct spelling in your case used a G rather than a J - the J is far more common in my experience. "Jim" would be an aberration that could be improved, but substitution "Jem" for "Gem" without any context to suggest the latter would be just fine IMO.

Re: FFmpeg 8.0 adds Whisper support

#249
The only problem with this PR/diff is that it creates just a avfilter wrapper around whisper.cpp library and requires the user to manage the dependencies on their own. This is not helpful for novice users who will first need to:

1. git clone whisper.cpp

2. Make sure they have all dependencies for `that` library

3. Hope the build passes

4. Download the actual model

AND only then be able to use `-af "whisper=model...` filter.

If they try to use the filter without all the prereqs they'll fail and it'll create frustration.

It'd be better to natively create a Whisper avfilter and only require the user to download the model -- I feel like this would streamline the whole process and actually make people use it much more.

Re: FFmpeg 8.0 adds Whisper support

#250
post #93

Whisper is genuinely amazing - with the right nudging. It's the one AI thing that has genuinely turned my life upside-down in an unambiguously good way. People should check out Subtitle Edit (and throw the dev some money) which is a great interface for experimenting with Whisper transcription. It's basically Aegisub 2.0, if you're old, like me. HOWTO: Drop a video or audio file to the right window, then go to Video >…

you can install suing winget or chocolately

    winget install --id=Nikse.SubtitleEdit  -e
Post reply on HN