Does this mean that any software which uses ffmpeg can now add a transcription option? Audacity, Chrome, OBS etc
If they want to support it out-of-the box, they'll still have to embed a model file (roughly 500 MB - 3GB, varying size and quality)
FFmpeg 8.0 adds Whisper support
51–60 of 341 posts
Re: FFmpeg 8.0 adds Whisper support
#52I've been playing with whisper to try to do local transcription of long videos, but one issue I've found is that long (>15 seconds) spans without any speech tend to send it into a hallucination loops that it often can't recover from. I wonder if, with direct integration into ffmpeg, they will be able to configure it in a way that can improve that situation.
Whisper is supposed to be used with voice activity detection and all production implementations that I've seen do that. The raw model is known to make up nonsense for silence because, as I understand it, it was never trained not to do that, assuming everyone will use VAD
[deleted]
Re: FFmpeg 8.0 adds Whisper support
#53Can whisper do multilingual yet? Last time I tried it on some mixed dutch/english text it would spit out english translations for some of the dutch text. Strange bug/feature since from all appearances it had understood the dutch text perfectly fine.
Best for English, but I've found it pretty decent for Spanish.
Re: FFmpeg 8.0 adds Whisper support
#54Re: FFmpeg 8.0 adds Whisper support
#55Once local transcription is in more places hopefully we can persuade content creator not to burn bouncing sub-titles into their videos. I've seen professionally produced recordings on dry and technical subjects with good sound quality where they've decided to use distracting sub-titles with no way to disable them. It seems so unnecessary if you're not making novelty videos about cats. Also local transcription allows…
The other problem with burned-in subtitles is you can't change the language.
True, but (as someone who not infrequently has to rewind content on just about all streaming apps because it decided one particular subtitle only needed to be display for less than 200ms this time around) sometimes burned-in seems like a good idea.
I don't understand why the problem seems so pervasive (I've seen it on Netflix, Viki, and Apple TV, at least) and so transient.
Re: FFmpeg 8.0 adds Whisper support
#56Not sure it will be packaged in Debian, with an external binary model god knows how it was produced...
Re: FFmpeg 8.0 adds Whisper support
#57Earlier quoted context omitted.
You usually delete silence before using something like whisper.
I've heard that, but that doesn't sound like a useful approach for videos where (1) non-speech segments can have plenty of other sound (music, noise) and (2) you want timestamps to match up with the original video, like for subtitles. But maybe there are known mitigations for both of those issues that I'm not aware of. And if they do exist maybe they can be included in the ffmpeg whisper integration.
By "delete", people mostly mean "detect", so that you can avoid processing such segments through Whisper. There's no reason to actually cut the silence out from the original audio file.
Re: FFmpeg 8.0 adds Whisper support
#58Not sure it will be packaged in Debian, with an external binary model god knows how it was produced...
It looks like the model file needs to be supplied at invocation time, so the binary blob would not be required for packaging.
Re: FFmpeg 8.0 adds Whisper support
#59Can whisper do multilingual yet? Last time I tried it on some mixed dutch/english text it would spit out english translations for some of the dutch text. Strange bug/feature since from all appearances it had understood the dutch text perfectly fine.
I think the Dutch/English is probably the worst combination for this. Languages are rather close.
Re: FFmpeg 8.0 adds Whisper support
#60Earlier quoted context omitted.
Or read the documentation for the new whisper filter: https://ffmpeg.org/ffmpeg-filters.html#whisper-1
That also works, I assumed the ffmpeg website would also be behind Anubis if the git server is, but it doesn't actually seem to be.
Anubis is not all that useful for static websites since serving them does not generate high load (unlike when a bot traverses a Git server UI).