41–50 of 341 posts
Can whisper do multilingual yet? Last time I tried it on some mixed dutch/english text it would spit out english translations for some of the dutch text. Strange bug/feature since from all appearances it had understood the dutch text perfectly fine.
I've been playing with whisper to try to do local transcription of long videos, but one issue I've found is that long (>15 seconds) spans without any speech tend to send it into a hallucination loops that it often can't recover from. I wonder if, with direct integration into ffmpeg, they will be able to configure it in a way that can improve that situation.
I've been playing with whisper to try to do local transcription of long videos, but one issue I've found is that long (>15 seconds) spans without any speech tend to send it into a hallucination loops that it often can't recover from. I wonder if, with direct integration into ffmpeg, they will be able to configure it in a way that can improve that situation.
You usually delete silence before using something like whisper.
I've seen professionally produced recordings on dry and technical subjects with good sound quality where they've decided to use distracting sub-titles with no way to disable them.
It seems so unnecessary if you're not making novelty videos about cats.
Also local transcription allows for automatic translation and again overlaying subtitles on top of an existing burnt in set is a really poor reading experience.
Once local transcription is in more places hopefully we can persuade content creator not to burn bouncing sub-titles into their videos. I've seen professionally produced recordings on dry and technical subjects with good sound quality where they've decided to use distracting sub-titles with no way to disable them. It seems so unnecessary if you're not making novelty videos about cats. Also local transcription allows…
Can whisper do multilingual yet? Last time I tried it on some mixed dutch/english text it would spit out english translations for some of the dutch text. Strange bug/feature since from all appearances it had understood the dutch text perfectly fine.
Whisper has been multilingual for 5 years at least.
Earlier quoted context omitted.
Whisper works on 30 second chunks. So yes it can do that and that’s also why it can hallucinate quite a bit.
The ffmpeg code seems to default to three second chunks ( https://ffmpeg.org/ffmpeg-filters.html#whisper-1 ): queue The maximum size that will be queued into the filter before processing the audio with whisper. Using a small value the audio stream will be processed more often, but the transcription quality will be lower and the required processing power will be higher. Using a large value (e.g. 10-20s) will produce m…
I don't think other streaming transcription services have this issue since, whilst they do chunk up the input, past chunks can still be edited. They tend to use "best of N" decoding, so there are always N possible outputs, each with a probability assigned, and as soon as one word is the same in all N outputs then it becomes fixed.
The internal state of the decoder needs to be duplicated N times, but that typically isn't more than a few kilobytes of state so N can be hundreds to cover many combinations of ambiguities many words back.
Does this have the ability to edit historic words as more info becomes available? Eg. If I say "I scream", it sounds phonetically identical to "Ice cream". Yet the transcription of "I scream is the best dessert" makes a lot less sense than "Ice cream is the best dessert". Doing this seems necessary to have both low latency and high accuracy, and things like transcription on android do that and you can see the adjusti…
Does this have the ability to edit historic words as more info becomes available? Eg. If I say "I scream", it sounds phonetically identical to "Ice cream". Yet the transcription of "I scream is the best dessert" makes a lot less sense than "Ice cream is the best dessert". Doing this seems necessary to have both low latency and high accuracy, and things like transcription on android do that and you can see the adjusti…
I find that in languages I don't speak well, my ability to understand degrades much more quickly as the audio quality goes down. But in my native language, even with piss poor audio quality, my brain fills in the garbled words with its prior expectation of what those words should be, based on context.