Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Automatically synchronize subtitles with video

github.com

Post reply on HN
Print view View original

51–60 of 129 posts

Re: Show HN: Automatically synchronize subtitles with video

#51

Post by aasasd »

smacke wrote: post #13
kojackst wrote: post #9

Sorry for the dumb question, but what exactly does this do? `subsync reference.srt -i unsynchronized.srt -o synchronized.srt` I mean, I already have a synchronized srt file, so what would I be syncing here?

Great question. The main use case I can think of is when reference.srt and unsynchronized.srt are in different languages, and you want to eventually merge them into a single dual-language subtitle file. EDIT: Oh, I should also mention that you don't need a reference.srt -- it can look at the video directly and use that as a reference.

> when reference.srt and unsynchronized.srt are in different languages

I just want to thank you for including this use-case, because it's exactly the thing I'm regularly running into. Subs in one language are bundled with the vid, all subs from OpenSubtitles are desynchronized.

Re: Show HN: Automatically synchronize subtitles with video

#52

Post by aasasd »

smacke wrote: post #44
zeotroph wrote: post #25

Earlier quoted context omitted.

Previously: Fiddle with the subtitle/audio offset factor (and then they drift apart again slowly, driving you mad!) Now: subsync Soon: Players run subsync internally the press of one button or commandline switch. The voice audio detection and then mapping is such a neat solution. I would have embedded parts of the surrounding audio in some base64 format into the subtitle file and then used that as an alignment clue.…

Actually, it unfortunately doesn't work if the drift gets worse over time -- so far, it only works with constant drift. Maybe fixing constant 1st derivative drift is the next step!

> only works with constant drift

I hoped that this solution would sync parts that have really different offsets between the audio and the subs, including changes from negative to positive offsets. Because that's the cases where automatic fixes in e.g. Aegisub don't suffice.

This happens when the vid and the subs are from different releases which apparently were edited for some reason―regional releases or something. Like, after some point the subs are suddenly off by a minute.

Re: Show HN: Automatically synchronize subtitles with video

#54

Post by Iburinoc »

hosay123 wrote: post #48
anyfoo wrote: post #22

From the README: "[...] the naive O(n^2) strategy for scoring all alignments is unacceptable. Instead, we use the fact that "scoring all alignments" is a convolution operation and can be implemented with the Fast Fourier Transform (FFT), bringing the complexity down to O(n log n)." I absolutely love it when something that, at very first glance, has no business in being solved in the frequency domain, gets solved in t…

I thought I understood FFT, but totally not getting the relationship to this problem. Someone ELI5 please? :)

The problem being solved essentially is: you have two binary strings, and you want to offset one of them so that they match up the best. For each offset, you're taking a dot product of one sequence with the offset version of the other. This is the same as computing the convolution of the two sequences together (https://en.wikipedia.org/wiki/Convolution). Computing this naively would be O(n^2) (doing linear work for each possible offset).

One property of the Fourier transform is that convolution in the time domain corresponds to element-wise multiplication in the frequency domain (https://en.wikipedia.org/wiki/Convolution_theorem), so you can compute the convolution efficiently by taking the FFT of both series, doing element-wise multiplication, and then taking the inverse FFT of the result.

Re: Show HN: Automatically synchronize subtitles with video

#55

Post by aabeshou »

hosay123 wrote: post #48
anyfoo wrote: post #22

From the README: "[...] the naive O(n^2) strategy for scoring all alignments is unacceptable. Instead, we use the fact that "scoring all alignments" is a convolution operation and can be implemented with the Fast Fourier Transform (FFT), bringing the complexity down to O(n log n)." I absolutely love it when something that, at very first glance, has no business in being solved in the frequency domain, gets solved in t…

I thought I understood FFT, but totally not getting the relationship to this problem. Someone ELI5 please? :)

ELI5: You can turn this problem into finding the best "convolution index", and fourier transforms make computing convolutions cheaper.

ELIUndergrad: (note that \* means multiplication, there doesnt seem to be a way to escape an asterisk)

Lets start by seeing how this is a convolution. We have the videoSpeech sequence, and the subtitle sequence - each is a vector, indexed by time, of 0's and 1's indicating whether there is speech in that time. We can imagine padding the sequences out on either side with 0's, and consider the alignment task as shifting the subtitle sequence left and right in time until we get the best alignment with the 1's in the videoSpeech sequence. We can express the goodness of alignment as the number of matching 1's, aka the sum over all times t of videoSpeech(t) \* subtitle(t). This is the definition of a convolution: the convolution of two sequences gives a new sequence where the value at index i is this sum above where one of the sequences is shifted by i. Mathematically, conv(videoSpeech, subtitle)(i) = sum( videoSpeech(t)\* subtitle(t-i)). So we can rephrase this problem as, find the index i which maximizes the value of the convolution sequence.

The discrete fourier transform is a function that takes a sequence and gives another sequence. It's relevant here because it "turns convolution into multiplication": fourier(videoSpeech)(i) \* fourier(subtitle)(i) = fourier(conv(videoSpeech, subtitle))(i).

So finally to solve the problem, we get the pointwise product sequence S = fourier(videoSpeech) \* fourier(subtitle), do the inverse fourier transform on it invFourier(S), and maximize invFourier(S)(i) over i.

Re: Show HN: Automatically synchronize subtitles with video

#56

Post by scarmig »

clairity wrote: post #43
anyfoo wrote: post #22

From the README: "[...] the naive O(n^2) strategy for scoring all alignments is unacceptable. Instead, we use the fact that "scoring all alignments" is a convolution operation and can be implemented with the Fast Fourier Transform (FFT), bringing the complexity down to O(n log n)." I absolutely love it when something that, at very first glance, has no business in being solved in the frequency domain, gets solved in t…

the fourier series was one of the things that blew my mind. any periodic function can be decomposed into a bunch of sine waves! even (the interesting portions of) many other functions can be approximated by a (potentially infinite) series of sine waves! it's simply madness i tell you.

Interesting bit of history, Kolmogorov became famous when he published his first scientific paper (at 19!) on the construction of a function whose Fourier series diverges (almost) everywhere.

Re: Show HN: Automatically synchronize subtitles with video

#57

Post by kumarharsh »

This blows my mind (like everyone else's here)!

While playing digital copies of somewhat older movies (before BluRay rips came into vogue), a problem which surfaces frequently is that the frame rates of the video and the subtitle track are slightly mismatched, say video at 24fps, subtitle at 25.6fps(?). This makes the subtitles drift away from the video and require a manual intervention every few minutes. If I can't hear the audio properly for some reason, then I just become fed up and don't watch the movie at all. Add to that the existence of different 'cuts' for films, which add another dimension to the subtitle problem.

How would one even go about solving this problem?

Re: Show HN: Automatically synchronize subtitles with video

#58

Post by kumarharsh »

smacke wrote: post #44
zeotroph wrote: post #25

Earlier quoted context omitted.

Previously: Fiddle with the subtitle/audio offset factor (and then they drift apart again slowly, driving you mad!) Now: subsync Soon: Players run subsync internally the press of one button or commandline switch. The voice audio detection and then mapping is such a neat solution. I would have embedded parts of the surrounding audio in some base64 format into the subtitle file and then used that as an alignment clue.…

Actually, it unfortunately doesn't work if the drift gets worse over time -- so far, it only works with constant drift. Maybe fixing constant 1st derivative drift is the next step!

Oh! Yeah, that makes absolutely perfect sense! I just asked the same question here regarding the difference in FPS of video and subtitle.

Re: Show HN: Automatically synchronize subtitles with video

#59

Post by kumarharsh »

Completely tangential question to this awesome discussion: who even writes subtitles? It feels like a thankless job to write subtitles for rips of movies and TV shows old and new. I get that maybe they source original from, say, Netflix, but .set files existed long before Netflix, and for lots of movies not on Netflix. Writing all those tags for hearing impaired seems like a lot of work, which anyone other than the first-person movie production team would be loathe to indulge in. Yet I see a lot of subtitles which don't seem 'official'. Then there are subtitles which seem like they were written as a loose translation of the audio. Both are in English, so I don't see why they can't just transcribe what's being said. Instead, for dialogues such as "The greatest trick the Devil ever pulled was convincing the world he didn't exist.", the subtitle is written as "The devil tricked the world into thinking he didn't exist."

Re: Show HN: Automatically synchronize subtitles with video

#60

Post by xfer »

kumarharsh wrote: post #57

This blows my mind (like everyone else's here)! While playing digital copies of somewhat older movies (before BluRay rips came into vogue), a problem which surfaces frequently is that the frame rates of the video and the subtitle track are slightly mismatched, say video at 24fps, subtitle at 25.6fps(?). This makes the subtitles drift away from the video and require a manual intervention every few minutes. If I can't…

If you use vlc there is a subtitle fps that you can modify.
Post reply on HN