Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FFmpeg 8.0 adds Whisper support

code.ffmpeg.org

Post reply on HN
Print view View original

111–120 of 341 posts

Re: FFmpeg 8.0 adds Whisper support

#111

Post by ethan_smith »

donatj wrote: post #69

I know nothing about Whisper, is this usable for automated translation? I own a couple very old and as far as I'm aware never translated Japanese movies. I don't speak Japanese but I'd love to watch them. A couple years ago I had been negotiating with a guy on Fiver to translate them. At his usual rate-per-minute of footage it would have cost thousands of dollars but I'd negotiated him down to a couple hundred before…

Whisper can indeed transcribe Japanese and translate it to English, though quality varies by dialect and audio clarity. You'll need the "large-v3" model for best results, and you can use ffmpeg's new integration with a command like `ffmpeg -i movie.mp4 -af whisper=model=large-v3:task=translate output.srt`.

Re: FFmpeg 8.0 adds Whisper support

#112

Post by webinar »

Xunjin wrote: post #107
webinar wrote: post #104

I've been using FFmpeg and Whisper to record and transcribe live police scanner audio for my city, and update it in real-time to a live website. It works great, with the expected transcription errors and hallucinations.

Is this website open? Would love to see your work :P

somerville.votolab.com

Re: FFmpeg 8.0 adds Whisper support

#113

Post by porridgeraisin »

I had a small bash pipeline for doing this until now.

  ffmpeg -f pulse -i "$(pactl get-default-source)" -t 5 -f wav -ar 16000 -ac 1 -c:a pcm_s16le - \
  | ./main - \
  | head -2 \
  | tail -1 \
  | cut -d] -f2 \
  | awk '{$1=$1};1'
The reading from mic part (-f pulse, pactl...) is linux-specific rest of it should be cross platform. The `main` executable is the whisper.cpp executable (see whisper.cpp github readme, it's just the output of `make base.en` from that).

Edit: -t 5 controls recording duration.

Oh and add 2>/dev/null to silence the debug output. I copied this from a pipe that further sends it into an LLM that then looks at the meaning and turns it into a variety of structured data (reminders, todo items, etc) which I then....

Re: FFmpeg 8.0 adds Whisper support

#114

Post by MaxikCZ »

I tried to use whisper to generate non-english subs from english audio, but wasnt able to figure out. I know it can do english subs from non-english audio, and that earlier (less precise) versions could do any language audio -> any language subs, but latest whisper only to english subs.

Anyone found a way?

Re: FFmpeg 8.0 adds Whisper support

#115

Post by dewey »

dzhiurgis wrote: post #92
Lio wrote: post #45

Once local transcription is in more places hopefully we can persuade content creator not to burn bouncing sub-titles into their videos. I've seen professionally produced recordings on dry and technical subjects with good sound quality where they've decided to use distracting sub-titles with no way to disable them. It seems so unnecessary if you're not making novelty videos about cats. Also local transcription allows…

It's just so annyoing how someone like Netflix offers like 3-4 languages for most of its content when you can basically get it for free via browser extensions (if you watch on browser). Must be union thing.

That Netflix who would need to pay more to license more subtitles can't compete with pirated or unlicensed auto-generated subtitles shouldn't really be a surprise.

It's also annoying that you have to pay for Netflix when you can get the same movies for free with less restrictions on a pirate site.

Re: FFmpeg 8.0 adds Whisper support

#116

Post by MaKey »

kwar13 wrote: post #53
bondarchuk wrote: post #31

Can whisper do multilingual yet? Last time I tried it on some mixed dutch/english text it would spit out english translations for some of the dutch text. Strange bug/feature since from all appearances it had understood the dutch text perfectly fine.

Best for English, but I've found it pretty decent for Spanish.

It's even better for some languages other than English (e. g. Spanish), see: https://github.com/openai/whisper?tab=readme-ov-file#availab...

Re: FFmpeg 8.0 adds Whisper support

#117

Post by abdusco »

MaxikCZ wrote: post #114

I tried to use whisper to generate non-english subs from english audio, but wasnt able to figure out. I know it can do english subs from non-english audio, and that earlier (less precise) versions could do any language audio -> any language subs, but latest whisper only to english subs. Anyone found a way?

I solved it by generating English subtitles, then passing those to an LLM in chunks that are ~20 entries in size. Include preceding and following subtitles as context for better translation. Make sure to replace the timestamps with simple integer ids, because LLMs like to mangle those, no matter how hard you prompt.

I could share a python script that is working pretty reliably for me.

Re: FFmpeg 8.0 adds Whisper support

#118

Post by [deleted] »

miki123211 wrote: post #57
re wrote: post #40

Earlier quoted context omitted.

I've heard that, but that doesn't sound like a useful approach for videos where (1) non-speech segments can have plenty of other sound (music, noise) and (2) you want timestamps to match up with the original video, like for subtitles. But maybe there are known mitigations for both of those issues that I'm not aware of. And if they do exist maybe they can be included in the ffmpeg whisper integration.

By "delete", people mostly mean "detect", so that you can avoid processing such segments through Whisper. There's no reason to actually cut the silence out from the original audio file.

[deleted]

Re: FFmpeg 8.0 adds Whisper support

#119

Post by thunderfork »

dzhiurgis wrote: post #92
Lio wrote: post #45

Once local transcription is in more places hopefully we can persuade content creator not to burn bouncing sub-titles into their videos. I've seen professionally produced recordings on dry and technical subjects with good sound quality where they've decided to use distracting sub-titles with no way to disable them. It seems so unnecessary if you're not making novelty videos about cats. Also local transcription allows…

It's just so annyoing how someone like Netflix offers like 3-4 languages for most of its content when you can basically get it for free via browser extensions (if you watch on browser). Must be union thing.

[dead]

Re: FFmpeg 8.0 adds Whisper support

#120

Post by zoobab »

majewsky wrote: post #58
zoobab wrote: post #56

Not sure it will be packaged in Debian, with an external binary model god knows how it was produced...

It looks like the model file needs to be supplied at invocation time, so the binary blob would not be required for packaging.

so 'apt install ffmpeg' won't be enough to have the feature?
Post reply on HN