Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FFmpeg 8.0 adds Whisper support

code.ffmpeg.org

Post reply on HN
Print view View original

201–210 of 341 posts

Re: FFmpeg 8.0 adds Whisper support

#201

Post by numpad0 »

waltbosz wrote: post #123
ethan_smith wrote: post #111

Earlier quoted context omitted.

Whisper can indeed transcribe Japanese and translate it to English, though quality varies by dialect and audio clarity. You'll need the "large-v3" model for best results, and you can use ffmpeg's new integration with a command like `ffmpeg -i movie.mp4 -af whisper=model=large-v3:task=translate output.srt`.

I wonder how the results of an AI Japanese-audio-to-English-subtitles would compare to a fansub-ed anime. I'm guessing it would be a more literal translation vs. contextual or cultural. I found an interesting article about trollsubs, which I guess are fansubs made with a contemptuous flare. https://neemblog.home.blog/2020/08/19/the-lost-art-of-fan-ma... Tangent: I'm one of those people who watch movies with closed ca…

I was recently just playing around with Google Cloud ASR as well as smaller Whisper models, and I can say it hasn't gotten to that point: Japanese ASRs/STTs all generate final kanji-kana mixed text, and since kanji:pronunciation is n:n maps, it's non-trivial enough that it currently need hands from human native speakers to fix misheard texts in a lot of cases. LLMs should be theoretically good at this type of tasks, but they're somehow clueless about how Japanese pronunciation works, and they just rubber-stamp inputs as written.

The conversion process from pronunciation to intended text is not deterministic either, so it probably can't be solved by "simply" generating all-pronunciation outputs. Maybe a multimodal LLM as ASR/STT, or a novel dual input as-spoken+estimated-text validation model could be made? I wouldn't know, though. It seemed like a semi-open question.

Re: FFmpeg 8.0 adds Whisper support

#202

Post by jhatemyjob »

I wish they worked with the mpv folks instead of shoehorning this in. Based on the docs it looks like getting live transcription for a video will involve running the demuxer/decoder on one thread, and this whisper filter on another thread, using ffmpeg's AVIO (or to a REST API [1].... shudders) to synchronize those two parallel jobs. It could have been way simpler.

Other than for the "live transcription" usecase (that they made unnecessarily complicated), I don't see how this is any better than running Whisper.cpp directly. Other people in this thread are basically saying "ffmpeg's interface is better understood" [2] but LLMs make that point moot since you can just ask them to do the drudgery for you.

[1] https://medium.com/@vpalmisano/run-whisper-audio-transcripti...

[2] https://news.ycombinator.com/item?id=44890067

Re: FFmpeg 8.0 adds Whisper support

#203

Post by eigenvalue »

xrd wrote: post #163
notatallshaw wrote: post #140

Earlier quoted context omitted.

> uv pip install --system torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 uv has a feature to get the correct version of torch based on your available cuda (and some non-cuda) drivers (though I suggest using a venv not the system Python): > uv pip install torch torchvision torchaudio --torch-backend=auto More details: https://docs.astral.sh/uv/guides/integration/pytorch/#automa... This…

I love uv and really feel like I only need to know "uv add" and "uv sync" to be effective using it with python. That's an incredible feat. But, when I hear about these kinds of extras, it makes me even more excited. Getting cuda and torch to work together is something I have struggled countless times. The team at Astral should be nominated for a Nobel Peace Prize.

They’ve definitely saved me many hours of wasted time between uv and ruff.

Re: FFmpeg 8.0 adds Whisper support

#204

Post by herbcso »

smallpipe wrote: post #181
0cf8612b2e1e wrote: post #174

Earlier quoted context omitted.

The quality of subtitles implies that almost no effort is being put into their creation. Watch even a high budget movie/TV show and be aghast at how frequently they diverge.

A good subtitle isn't a perfect copy of what was said.

Tom Scott would agree with you. https://m.youtube.com/watch?v=pU9sHwNKc2c

Re: FFmpeg 8.0 adds Whisper support

#205

Post by danudey »

xrd wrote: post #163
notatallshaw wrote: post #140

Earlier quoted context omitted.

> uv pip install --system torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 uv has a feature to get the correct version of torch based on your available cuda (and some non-cuda) drivers (though I suggest using a venv not the system Python): > uv pip install torch torchvision torchaudio --torch-backend=auto More details: https://docs.astral.sh/uv/guides/integration/pytorch/#automa... This…

I love uv and really feel like I only need to know "uv add" and "uv sync" to be effective using it with python. That's an incredible feat. But, when I hear about these kinds of extras, it makes me even more excited. Getting cuda and torch to work together is something I have struggled countless times. The team at Astral should be nominated for a Nobel Peace Prize.

> "uv add"

One life-changing thing I've been using `uv` for:

System python version is 3.12:

    $ python3 --version
    Python 3.12.3
A script that requires a library we don't have, and won't work on our local python:

    $ cat test.py
    #!/usr/bin/env python3

    import sys
    from rich import print

    if sys.version_info
It fails:

    $ python3 test.py
    Traceback (most recent call last):
    File "/tmp/tmp/test.py", line 10, in 
        from rich import print
    ModuleNotFoundError: No module named 'rich'
Tell `uv` what our requirements are

    $ uv add --script=test.py --python '3.13' rich
    Updated `test.py`
`uv` updates the script:

    $ cat test.py
    #!/usr/bin/env python3
    # /// script
    # requires-python = ">=3.13"
    # dependencies = [
    #     "rich",
    # ]
    # ///

    import sys
    from rich import print

    if sys.version_info
`uv` runs the script, after installing packages and fetching Python 3.13

    $ uv run test.py
    Downloading cpython-3.13.5-linux-x86_64-gnu (download) (33.8MiB)
    Downloading cpython-3.13.5-linux-x86_64-gnu (download)
    Installed 4 packages in 7ms
    Hello world, this is python 3.13.5 (main, Jun 12 2025, 12:40:22) [Clang 20.1.4 ]
And if we run it with Python 3.12, we can see that errors:

    $ uv run --python 3.12 test.py
    warning: The requested interpreter resolved to Python 3.12.3, which is incompatible with the script's Python requirement: `>=3.13`
    Installed 4 packages in 7ms
    This script will not work on Python 3.12
Works for any Python you're likely to want:

    $ uv python list
    cpython-3.14.0b2-linux-x86_64-gnu                 
    cpython-3.14.0b2+freethreaded-linux-x86_64-gnu    
    cpython-3.13.5-linux-x86_64-gnu                   /home/dan/.local/share/uv/python/cpython-3.13.5-linux-x86_64-gnu/bin/python3.13
    cpython-3.13.5+freethreaded-linux-x86_64-gnu      
    cpython-3.12.11-linux-x86_64-gnu                  
    cpython-3.12.3-linux-x86_64-gnu                   /usr/bin/python3.12
    cpython-3.12.3-linux-x86_64-gnu                   /usr/bin/python3 -> python3.12
    cpython-3.11.13-linux-x86_64-gnu                  /home/dan/.local/share/uv/python/cpython-3.11.13-linux-x86_64-gnu/bin/python3.11
    cpython-3.10.18-linux-x86_64-gnu                  /home/dan/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/bin/python3.10
    cpython-3.9.23-linux-x86_64-gnu                   
    cpython-3.8.20-linux-x86_64-gnu                   
    pypy-3.11.11-linux-x86_64-gnu                     
    pypy-3.10.16-linux-x86_64-gnu                     
    pypy-3.9.19-linux-x86_64-gnu                      
    pypy-3.8.16-linux-x86_64-gnu                      
    graalpy-3.11.0-linux-x86_64-gnu                   
    graalpy-3.10.0-linux-x86_64-gnu                   
    graalpy-3.8.5-linux-x86_64-gnu

Re: FFmpeg 8.0 adds Whisper support

#206

Post by rkomorn »

t-3 wrote: post #179
rkomorn wrote: post #55

Earlier quoted context omitted.

True, but (as someone who not infrequently has to rewind content on just about all streaming apps because it decided one particular subtitle only needed to be display for less than 200ms this time around) sometimes burned-in seems like a good idea. I don't understand why the problem seems so pervasive (I've seen it on Netflix, Viki, and Apple TV, at least) and so transient.

It's a newer problem IME, so I'd guess it's cause by people using auto-transcription/translation tools to generate subtitles. For eg. Chinese content, I'll see stuff on Viki where the OG Mandarin subs are formatted sanely and the English is piecemeal follow-the-audio style. I can't imagine this happening in any other way than use of a transcription+translation tool without review.

I don't think it's an automation-related thing. It happens even on big name shows on big apps.

I think it's a toolkit thing where some sort of event or timer goes off at the wrong time and the subtitles get cleared when they shouldn't. And then if you rewind and replay, it doesn't happen again (because spurious event/timer issue).

Re: FFmpeg 8.0 adds Whisper support

#207

Post by kstrauser »

smallpipe wrote: post #181
0cf8612b2e1e wrote: post #174

Earlier quoted context omitted.

The quality of subtitles implies that almost no effort is being put into their creation. Watch even a high budget movie/TV show and be aghast at how frequently they diverge.

A good subtitle isn't a perfect copy of what was said.

Hard disagree. When I'm reading a transcript, I want word-for-word what the people said, not a creative edit. I want the speakers' voice, not the transcriptionist's.

And when I'm watching subtitles in my own language (say because I want the volume low so I'm not disturbing others), I hate when the words I see don't match the words I hear. It's the quickest way I can imagine to get sucked out of the content and into awareness of the delivery of the content.

Re: FFmpeg 8.0 adds Whisper support

#208

Post by guipsp »

dotancohen wrote: post #185
kwar13 wrote: post #5

Fantastic! I am working on a speech-to-text GNOME extension that would immensely benefit from this. https://github.com/kavehtehrani/gnome-speech2text

Why is this a Gnome extension? I would love to use this in KDE.

Likely because they are a GNOME user and the APIs are DE specific.

Re: FFmpeg 8.0 adds Whisper support

#209

Post by t-3 »

rkomorn wrote: post #206
t-3 wrote: post #179

Earlier quoted context omitted.

It's a newer problem IME, so I'd guess it's cause by people using auto-transcription/translation tools to generate subtitles. For eg. Chinese content, I'll see stuff on Viki where the OG Mandarin subs are formatted sanely and the English is piecemeal follow-the-audio style. I can't imagine this happening in any other way than use of a transcription+translation tool without review.

I don't think it's an automation-related thing. It happens even on big name shows on big apps. I think it's a toolkit thing where some sort of event or timer goes off at the wrong time and the subtitles get cleared when they shouldn't. And then if you rewind and replay, it doesn't happen again (because spurious event/timer issue).

At least with vtt and srt, the chunk of text displayed is explicitly associated with a chunk of time, so something like that really shouldn't be happening. Maybe there is some sort of subtitle-writing on the fly like what is sometimes done with transcoding video, but that would be really strange for a plaintext format that is so light compared to the video and audio coming with it.

Re: FFmpeg 8.0 adds Whisper support

#210

Post by guipsp »

SSLy wrote: post #148
demurgos wrote: post #125

Earlier quoted context omitted.

I'm not very familiar with them, but I always assumed that there is a lot of overlap between the maintainers of both projects.

Well, they are just unrelated. VLC has a plugin to access ffmpeg codecs via libav*, that's about it.

They are not completly unrelated. There is significant overlap. FFMPEG also uses libs from VLC.
Post reply on HN