Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Automatically synchronize subtitles with video

github.com

Post reply on HN
Print view View original

41–50 of 129 posts

Re: Show HN: Automatically synchronize subtitles with video

#41

Post by smacke »

robert-boehnke wrote: post #40

> Instead, we use the fact that "scoring all alignments" is a convolution operation and can be implemented with the Fast Fourier Transform (FFT), bringing the complexity down to O(n log n). Not sure I understand this right - is this basically treating both binary strings as square waves, converting them to the frequency domain and determining the offset as a pitch shift between the two spectrograms?

Precisely. If I could more easily get at the raw classifier output of webrtcvad, it should be possible to be even smarter (we could have square waves with any amplitude between -1 and +1, not just either -1 and +1, which should take into account the classifier uncertainty).

EDIT: err, I'm actually not sure about the pitch shift part, that's a bit of vocabulary I'm not familiar with. If you've seen the fast polynomial multiplication algo from CLRS, it's basically that. E.g. if we have strings 1101 and 0101, we can find the best alignment by looking at the exponent of the largest coefficient after multiplying

polynomial(1101)*polynomial(reverse(0101))

where polynomial(1101) = x^3 + x^2 - x + 1

and polynomial(reverse(0101)) = polynomial(1010) = x^3 - x^2 + x - 1

Re: Show HN: Automatically synchronize subtitles with video

#42

Post by smacke »

loeg wrote: post #34

> I have yet to find a case where the automatic synchronization has been off by more than ~1 second That's actually quite bad synchronization error for A-V; maybe it's less noticeable for subtitles. I wonder if there is a good way to reduce that error down to milliseconds.

It's actually fairly annoying for subtitles, but it's useful for getting videos in unfamiliar languages to the right ballpark. Afterwards, my go-to so far has been to just perform a manual one-off adjustment in VLC after the video starts (you can offset subtitles in increments / decrements of 50ms with hotkeys h and j).

Re: Show HN: Automatically synchronize subtitles with video

#43

Post by clairity »

anyfoo wrote: post #22

From the README: "[...] the naive O(n^2) strategy for scoring all alignments is unacceptable. Instead, we use the fact that "scoring all alignments" is a convolution operation and can be implemented with the Fast Fourier Transform (FFT), bringing the complexity down to O(n log n)." I absolutely love it when something that, at very first glance, has no business in being solved in the frequency domain, gets solved in t…

the fourier series was one of the things that blew my mind. any periodic function can be decomposed into a bunch of sine waves! even (the interesting portions of) many other functions can be approximated by a (potentially infinite) series of sine waves! it's simply madness i tell you.

Re: Show HN: Automatically synchronize subtitles with video

#44

Post by smacke »

zeotroph wrote: post #25
guessmyname wrote: post #16

Earlier quoted context omitted.

I haven’t tried this command, but I believe the use case is as follows: • Open audio/video file with —supposedly— synchronized file • After a few seconds, I realize the subtitles appear before/after the dialogues • I immediately close the multimedia player, and open the Terminal • I execute the “subsync” command which does who knows what • Open the SRT and discover that the subtitles are now in the correct timestamps…

Previously: Fiddle with the subtitle/audio offset factor (and then they drift apart again slowly, driving you mad!) Now: subsync Soon: Players run subsync internally the press of one button or commandline switch. The voice audio detection and then mapping is such a neat solution. I would have embedded parts of the surrounding audio in some base64 format into the subtitle file and then used that as an alignment clue.…

Actually, it unfortunately doesn't work if the drift gets worse over time -- so far, it only works with constant drift. Maybe fixing constant 1st derivative drift is the next step!

Re: Show HN: Automatically synchronize subtitles with video

#46

Post by draz »

This is an area I am actively looking into at the moment for my employer. It's an interesting project, and I am excited to see work in this domain. Note that this will not pass QC at any broadcaster/OTT. Our tolerance is typically within 1-12 frames (and a bunch of other requirements around shot changes, etc.)

Re: Show HN: Automatically synchronize subtitles with video

#47

Post by derefr »

smacke wrote: post #26
IshKebab wrote: post #11

Surely you can do much better than this using a speech recogniser?

By speech recognizer, are you referring to a method that would perform speech-to-text transcription? If so, you could probably do better when your video and subtitles are in the same language, but you would lose the ability to be language-agnostic (unless you manage to solve machine translation well enough :) Interestingly enough, I haven't actually found any cases where the synchronization doesn't work (assuming the…

How about if you just do speech-to-timed-phonemes (which doesn't require any kind of language model, just a model trained with people speaking in accents, to produce "canonical" phonemes) and text-to-phonemes (which just needs a dictionary with IPA metadata), and then attempt to synchronize your text phonemes to the audio's phonemes? It'd be the same as the current solution, but with higher dimensionality on the symbols.

Re: Show HN: Automatically synchronize subtitles with video

#48

Post by hosay123 »

anyfoo wrote: post #22

From the README: "[...] the naive O(n^2) strategy for scoring all alignments is unacceptable. Instead, we use the fact that "scoring all alignments" is a convolution operation and can be implemented with the Fast Fourier Transform (FFT), bringing the complexity down to O(n log n)." I absolutely love it when something that, at very first glance, has no business in being solved in the frequency domain, gets solved in t…

I thought I understood FFT, but totally not getting the relationship to this problem. Someone ELI5 please? :)

Re: Show HN: Automatically synchronize subtitles with video

#49

Post by aasasd »

zeotroph wrote: post #18
satori99 wrote: post #3

This is cool. I am constantly impressed by ffmpeg and it capabilities. Last week I was able to make it act as a proxy for live streaming video that reduced the volume of ad-breaks by 50% automatically by listening for SCTE35 cue packets in the stream and adjusting a volume filter accordingly. My housemate has a reality TV addiction, and the ads were getting on my nerves. I just intended to see if it was possible, but…

A neat subtitle feature I found out mpv has, but no other video player: Hide those parts of the subtitle which are mainly for those with impaired hearing like '[loud noise]' etc, but keep the rest: `mpv --sub-filter-sdh ...` That logic could probably be extracted into a separate subtitle filtering tool.

When going through Miami Vice I discovered that the SDH parts sometimes also act as notes for the culture-impaired. (Song titles and lyrics were a particularly nice touch, though not strictly SDH.)

And for English-as-second-language folks, it's occasionally curious which expressions are used in the SDH descriptions.

Post reply on HN