Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FFmpeg 8.0 adds Whisper support

code.ffmpeg.org

Post reply on HN
Print view View original

171–180 of 341 posts

Re: FFmpeg 8.0 adds Whisper support

#171

Post by brcmthrowaway »

yvdriess wrote: post #94
londons_explore wrote: post #23

Does this have the ability to edit historic words as more info becomes available? Eg. If I say "I scream", it sounds phonetically identical to "Ice cream". Yet the transcription of "I scream is the best dessert" makes a lot less sense than "Ice cream is the best dessert". Doing this seems necessary to have both low latency and high accuracy, and things like transcription on android do that and you can see the adjusti…

A good opportunity to point people to the paper with my favorite title of all time: "How to wreck a nice beach you sing calm incense" https://dl.acm.org/doi/10.1145/1040830.1040898

Do AI voice recognition still use markov models for this?

Re: FFmpeg 8.0 adds Whisper support

#173

Post by numpad0 »

cenamus wrote: post #127
numpad0 wrote: post #62

Earlier quoted context omitted.

Isn't that a bit much for ASR models? Humans can't handle simultaneous multilingual dictation task either, I have to stop and reinitialize ears before switching languages between English and my primary one.

Isn't that exactly what intepreters do?

If they're like what I am, they seem to just coordinate constant staggered resets for sub-systems of language processing pipeline while keeping internal representations of inputs in half-text state so that input come back out through the pipeline in the other configurations.

That's how I anecdotally feel and interpret how my own brain appear to work, so it could be different from how interpreters work or how actual human brains work, but as far as I see it, professional simultaneous interpreters don't seem to be agnostic for relevant pairs of languages at all.

Re: FFmpeg 8.0 adds Whisper support

#174

Post by 0cf8612b2e1e »

Fluorescence wrote: post #110
londons_explore wrote: post #23

Does this have the ability to edit historic words as more info becomes available? Eg. If I say "I scream", it sounds phonetically identical to "Ice cream". Yet the transcription of "I scream is the best dessert" makes a lot less sense than "Ice cream is the best dessert". Doing this seems necessary to have both low latency and high accuracy, and things like transcription on android do that and you can see the adjusti…

It makes me curious about how human subtitlers or even scriptwriters choose to transcribe intentionally ambiguous speech, puns and narratively important mishearings. It's like you need to subtitle what is heard not what is said. Do those born profoundly deaf specifically study word sounds in order to understand/create puns, rhymes and such so they don't need assistance understanding narrative mishearings? It must fee…

The quality of subtitles implies that almost no effort is being put into their creation. Watch even a high budget movie/TV show and be aghast at how frequently they diverge.

Re: FFmpeg 8.0 adds Whisper support

#175

Post by mjw_byrne »

chipsrafferty wrote: post #158
efilife wrote: post #135

Earlier quoted context omitted.

Thanks. Now I know that I'm not that stupid and this actually makes no sense

It actually does make sense. Not saying you're stupid, but in standard English, if you say it quickly, the two sentences are nearly identical.

They're pretty different in British English, I struggled to figure it out until I started thinking about how it would sound with an American accent.

Re: FFmpeg 8.0 adds Whisper support

#176

Post by t-3 »

anonymousiam wrote: post #96
ph4evers wrote: post #32

Earlier quoted context omitted.

Whisper works on 30 second chunks. So yes it can do that and that’s also why it can hallucinate quite a bit.

Whisper is excellent, but not perfect. I used Whisper last week to transcribe a phone call. In the transcript, the name of the person I was speaking with (Gem) was alternately transcribed as either "Jim" or "Jem", but never "Gem."

That's at least as good as a human, though. Getting to "better-than-human" in that situation would probably require lots of potentially-invasive integration to allow the software to make correct inferences about who the speakers are in order to spell their names correctly, or manually supplying context as another respondent mentioned.

Re: FFmpeg 8.0 adds Whisper support

#177

Post by johnisgood »

voxadam wrote: post #30
johnisgood wrote: post #15

Earlier quoted context omitted.

Yes. From the documentation: > It runs automatic speech recognition using the OpenAI's Whisper model.

Thanks, I was being tripped up by DDOS protection on code.ffmpeg.org for a minute and couldn't read the patch. The combo of Firefox and the fact that Quantum/Lumen/CenturyLink seems to get off by rotating my dynamic IP for no reason occasionally triggers various DDOS protections schemes.

No problem. :) Yeah, it took me 8 seconds to get through. It seems your issue was worse.

Re: FFmpeg 8.0 adds Whisper support

#178

Post by iambvk »

Is anyone able to get streaming audio to text conversion working with whisper.cpp?

I tried several times to get this into a reasonable shape, but all have been failures. If anyone has pointers I really appreciate it.

Re: FFmpeg 8.0 adds Whisper support

#179

Post by t-3 »

rkomorn wrote: post #55
HPsquared wrote: post #46

Earlier quoted context omitted.

The other problem with burned-in subtitles is you can't change the language.

True, but (as someone who not infrequently has to rewind content on just about all streaming apps because it decided one particular subtitle only needed to be display for less than 200ms this time around) sometimes burned-in seems like a good idea. I don't understand why the problem seems so pervasive (I've seen it on Netflix, Viki, and Apple TV, at least) and so transient.

It's a newer problem IME, so I'd guess it's cause by people using auto-transcription/translation tools to generate subtitles. For eg. Chinese content, I'll see stuff on Viki where the OG Mandarin subs are formatted sanely and the English is piecemeal follow-the-audio style. I can't imagine this happening in any other way than use of a transcription+translation tool without review.

Re: FFmpeg 8.0 adds Whisper support

#180

Post by abdusco »

vevoe wrote: post #145
abdusco wrote: post #117

Earlier quoted context omitted.

I solved it by generating English subtitles, then passing those to an LLM in chunks that are ~20 entries in size. Include preceding and following subtitles as context for better translation. Make sure to replace the timestamps with simple integer ids, because LLMs like to mangle those, no matter how hard you prompt. I could share a python script that is working pretty reliably for me.

I'd love to see that script, do you have a link?

https://gist.github.com/abdusco/5bd5c909547f5f9b935dbd2fb2fe...
Post reply on HN