Is Whisper still SOTA 3 years later? It does not seem there is a clearly better open model. Alec Radford really is a genius!
NVIDIA Nemo Parakeet for English.
Mistral’s recent Voxtral is supposed to be nice and open source
FFmpeg 8.0 adds Whisper support
221–230 of 341 posts
Re: FFmpeg 8.0 adds Whisper support
#222Once local transcription is in more places hopefully we can persuade content creator not to burn bouncing sub-titles into their videos. I've seen professionally produced recordings on dry and technical subjects with good sound quality where they've decided to use distracting sub-titles with no way to disable them. It seems so unnecessary if you're not making novelty videos about cats. Also local transcription allows…
Those burned in subtitles still aren’t as cool as theme-matched anime subtitles during intro music sequences from fansubs 15 years ago. Those are still cool IMO
Or how the fansubbers will create masks to translate diegetic text like signage and written notes
Re: FFmpeg 8.0 adds Whisper support
#223Earlier quoted context omitted.
A slight segue to this; I was made aware of the phenomena that - The language in which you think in, sets the constraints to which you level of expanse the brain can think and parse information in. I think in English fortunately and it's an ever evolving language so, expanding as the world does. That is compared to the majority of people where I'm from; English was a second language they had to learn and the people t…
This is called linguist relativity (nee. The Sapir-Whorf hypothesis) and the strong form you describe has fallen out of favour in modern linguistics. A surprising number of monolingual people think their own language is the most adaptable and modern language, but this is obviously untrue. All languages evolve to fit the needs of speakers. Also, the idea that people "think in language X" is heavily disputed. One obvio…
> if you truly did think in the language you speak, how could this situation happen?
As far as how it happens to me is concerned, either something closer to speech than raw thoughts reports back the data in shared memory is invalid for selected language, or I find there's no text representation exist for what I am trying to say.
The "raw" thoughts work with the currently active language, for me, so at least for me, I just know strong Sapir-Whorf hypothesis is not even a hypothesis, but just a reasonable verbalization closely matching my own observations.
I don't get why people can't take it, even in the age of LLMs. It is what it is and that old guy is just never correct even for once.
Re: FFmpeg 8.0 adds Whisper support
#224Why would one use FFmpeg with Whisper support, instead of using Whisper directly?
At least whisper.cpp only supports a few input formats like WAV and MP3. To get subtitles for videos I always have to first run ffmpeg to get an audio file and then run whisper.cpp. Guess this new feature may mean that I can do it in just one step, so slightly more convenient?
Re: FFmpeg 8.0 adds Whisper support
#225How can I run Whisper or this software in Linux or Android as a non-technical user? Basically a simple audio-to-text for personal use?
I don't think installing (i.e. compiling) whisper.cpp and using it to do audio-to-text is very difficult. If the documentation is too technical I am sure you can ask some LLM to walk you through it. I have used it on Android in termux and on my FreeBSD desktop computer. Would not expect any difficulties on any modern Linux.
Re: FFmpeg 8.0 adds Whisper support
#226Can whisper do multilingual yet? Last time I tried it on some mixed dutch/english text it would spit out english translations for some of the dutch text. Strange bug/feature since from all appearances it had understood the dutch text perfectly fine.
Isn't that a bit much for ASR models? Humans can't handle simultaneous multilingual dictation task either, I have to stop and reinitialize ears before switching languages between English and my primary one.
In South Asia, it's quite common for people to speak a combination of their local language and English. Not just alternating sentences between the two languages, but in fact, constructing sentences using compound phrases from the two languages.
"Madam, please believe me, maine homework kiya ha" [I did my homework].
Re: FFmpeg 8.0 adds Whisper support
#227Earlier quoted context omitted.
A good opportunity to point people to the paper with my favorite title of all time: "How to wreck a nice beach you sing calm incense" https://dl.acm.org/doi/10.1145/1040830.1040898
For folks like me puzzling over what the correct transcription of the title should be, I think it's "How to recognize speech using common sense"
This is the correct parsing of it.
(I can't take credit for coming up with the title, but I worked on the project.)
Re: FFmpeg 8.0 adds Whisper support
#228Why would one use FFmpeg with Whisper support, instead of using Whisper directly?
[deleted]
Re: FFmpeg 8.0 adds Whisper support
#229I had a small bash pipeline for doing this until now. ffmpeg -f pulse -i "$(pactl get-default-source)" -t 5 -f wav -ar 16000 -ac 1 -c:a pcm_s16le - \ | ./main - \ | head -2 \ | tail -1 \ | cut -d] -f2 \ | awk '{$1=$1};1' The reading from mic part (-f pulse, pactl...) is linux-specific rest of it should be cross platform. The `main` executable is the whisper.cpp executable (see whisper.cpp github readme, it's just the…
> which I then.... Yes, please, go on...
The LLM turns my unstructured command into structured command (a limited set of commands hardcoded in the prompt) and a script takes that and executes it. I have it do stuff like interact with google keep/google calendar using the CLI. Those are the most used actions but there's a few others . Of course all actions can be scheduled.
The LLM can screw up now and then and output absolute garbage. But I've got a knack now for figuring out what prompts it's gonna be hopeless on and I manually enter those.
Example:
Saying
Remove makhana from shopping list
Ends up running the command
gkeep items edit shopping_list --check makhana
There is a direct text interface too that skips the voice transcription.
The main thing is it does in a background window without interrupting my screen or me needing to wait for whatever slow webpage to load. I had it do a few things on GitHub like remind me when checks pass on PRs. You could potentially connect it to various things like your amazon account to check on your order, etc,.. as I write this I now realise I did what basically amounts to what folks do with MCP today. Maybe I should update it to use the protocol.
These days I have a little more idle time as a grad student than I did in a tech company, and I don't really need to manage home/cooking/... so I don't really use some of the more complicated features. I mostly just use it to schedule 1on1s with my guide and add reminders about assignments and TA work and talks and my music class.
Re: FFmpeg 8.0 adds Whisper support
#230Whisper is genuinely amazing - with the right nudging. It's the one AI thing that has genuinely turned my life upside-down in an unambiguously good way. People should check out Subtitle Edit (and throw the dev some money) which is a great interface for experimenting with Whisper transcription. It's basically Aegisub 2.0, if you're old, like me. HOWTO: Drop a video or audio file to the right window, then go to Video >…
Can you give an example why it made your life that much better?
I used it like sibling commenter to get subtitles for downloaded videos. My hearing is bad. Whisper seems much better that YouTube's built-in auto-subtitles, so sometimes it is worth the extra trouble for me to download a video just to generate good subtitles and then watch it offline.
I also used whisper.cpp to transcribe all my hoarded podcast episodes. Took days of my poor old CPU working at 100% on all cores (and then a few shorter runs to transcribe new episodes I have downloaded since). Worked as good as I could possibly hope. Of course it gets the spelling of names wrong, but I don't expect anything (or anyone) to do much better. It is great to be able to run ripgrep to find old episodes on some topic and sometimes now I read an episode instead of listen, or listen to it with mpv with subtitles.