How can I run Whisper or this software in Linux or Android as a non-technical user?
Basically a simple audio-to-text for personal use?
FFmpeg 8.0 adds Whisper support
161–170 of 341 posts
Re: FFmpeg 8.0 adds Whisper support
#162Once local transcription is in more places hopefully we can persuade content creator not to burn bouncing sub-titles into their videos. I've seen professionally produced recordings on dry and technical subjects with good sound quality where they've decided to use distracting sub-titles with no way to disable them. It seems so unnecessary if you're not making novelty videos about cats. Also local transcription allows…
I think this trend is partially driven by the silent auto play that happens on YouTube. Baked in subtitles help draw people into the video.
Re: FFmpeg 8.0 adds Whisper support
#163Whisper is genuinely amazing - with the right nudging. It's the one AI thing that has genuinely turned my life upside-down in an unambiguously good way. People should check out Subtitle Edit (and throw the dev some money) which is a great interface for experimenting with Whisper transcription. It's basically Aegisub 2.0, if you're old, like me. HOWTO: Drop a video or audio file to the right window, then go to Video >…
> uv pip install --system torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 uv has a feature to get the correct version of torch based on your available cuda (and some non-cuda) drivers (though I suggest using a venv not the system Python): > uv pip install torch torchvision torchaudio --torch-backend=auto More details: https://docs.astral.sh/uv/guides/integration/pytorch/#automa... This…
I love uv and really feel like I only need to know "uv add" and "uv sync" to be effective using it with python. That's an incredible feat.
But, when I hear about these kinds of extras, it makes me even more excited. Getting cuda and torch to work together is something I have struggled countless times.
The team at Astral should be nominated for a Nobel Peace Prize.
Re: FFmpeg 8.0 adds Whisper support
#164Whisper is genuinely amazing - with the right nudging. It's the one AI thing that has genuinely turned my life upside-down in an unambiguously good way. People should check out Subtitle Edit (and throw the dev some money) which is a great interface for experimenting with Whisper transcription. It's basically Aegisub 2.0, if you're old, like me. HOWTO: Drop a video or audio file to the right window, then go to Video >…
I found this online demo of it: https://www.nikse.dk/subtitleedit/online
Re: FFmpeg 8.0 adds Whisper support
#165Does this have the ability to edit historic words as more info becomes available? Eg. If I say "I scream", it sounds phonetically identical to "Ice cream". Yet the transcription of "I scream is the best dessert" makes a lot less sense than "Ice cream is the best dessert". Doing this seems necessary to have both low latency and high accuracy, and things like transcription on android do that and you can see the adjusti…
It makes me curious about how human subtitlers or even scriptwriters choose to transcribe intentionally ambiguous speech, puns and narratively important mishearings. It's like you need to subtitle what is heard not what is said. Do those born profoundly deaf specifically study word sounds in order to understand/create puns, rhymes and such so they don't need assistance understanding narrative mishearings? It must fee…
I had similar thoughts when reading Huck Finn. It's not just phonetically spelled, it's much different. Almost like Twain came up with a list of words, and then had a bunch of 2nd graders tell him the spelling of words they had seen. I guess at some point, you just get good at bad spelling?
Re: FFmpeg 8.0 adds Whisper support
#166Earlier quoted context omitted.
As a hard of hearing person, I can now download any video from the internet (e.g. youtube) and generate subtitles on the fly, not having to struggle to understand badly recorded or unintelligible speech.
IF the dialog is badly recorded or unintelligible speech, how would a transcription process get it correct?
Because it can use the full set of information of the audio - people with hearing difficulties cannot. Also interesting, people with perfectly functional hearing, but whom have "software" bugs (i.e. I find it extremely hard to process voices with significant background nose) can also benefit :)
Re: FFmpeg 8.0 adds Whisper support
#167Can whisper do multilingual yet? Last time I tried it on some mixed dutch/english text it would spit out english translations for some of the dutch text. Strange bug/feature since from all appearances it had understood the dutch text perfectly fine.
Whisper has been multilingual for 5 years at least.
Except it’s only been released in September 2022 (not even 3 years ago).
Re: FFmpeg 8.0 adds Whisper support
#168Earlier quoted context omitted.
As a hard of hearing person, I can now download any video from the internet (e.g. youtube) and generate subtitles on the fly, not having to struggle to understand badly recorded or unintelligible speech.
IF the dialog is badly recorded or unintelligible speech, how would a transcription process get it correct?
The definition of "unintelligible" varies by person, especially by accent. Like, I got no problem with understanding the average person from Germany... but someone from the deep backwaters of Saxony, forget about that.
Re: FFmpeg 8.0 adds Whisper support
#169Earlier quoted context omitted.
so if "I scream" is in one chunk, and "is the best dessert" is in the next, then there is no way to edit the first chunk to correct the mistake? That seems... suboptimal! I don't think other streaming transcription services have this issue since, whilst they do chunk up the input, past chunks can still be edited. They tend to use "best of N" decoding, so there are always N possible outputs, each with a probability as…
The right way to do this would be to use longer, overlapping chunks. E.g. do thranscription every 3 seconds, but transcribe the most recent 15s of audio (or less if it's the beginning of the recording). This would increase processing requirements significantly, though. You could probably get around some of that with clever use of caching, but I don't think any (open) implementation actually does that.
If real-time transcription is so bad, why force it to be real-time. What happens if you give it a 2-3 second delay? That's pretty standard in live captioning. I get real-time being the ultimate goal, but we're not there yet. So working within the current limitations is piss poor transcription in real-time really more desirable/better than better transcriptions 2-3 second delay?
Re: FFmpeg 8.0 adds Whisper support
#170I guess that there is no streaming option for sending generated tokens to, say, an LLM service to process the text in real-time.
Whisper has the encoder-decoder architecture, so it's hard to run streaming efficiently, though whisper-streaming is a thing. https://kyutai.org/next/stt is natively streaming STT.
There are many streaming ASR models based on CTC or RNNT. Look for example at sherpa (https://github.com/k2-fsa/sherpa-onnx), which can run streaming ASR, VAD, diarization, and many more.