Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Automatically synchronize subtitles with video

github.com

Post reply on HN
Print view View original

21–30 of 129 posts

Re: Show HN: Automatically synchronize subtitles with video

#21

Post by smacke »

acqq wrote: post #8

"Although it can usually work if all you have is the video file, it will be faster (and potentially more accurate) if you have a correctly synchronized "reference" srt file, in which case you can do the following: subsync reference.srt -i unsynchronized.srt -o synchronized.srt" I really don't understand what the reference.srt is supposed to be here. Edit: If I have a "reference" which is synchronized why do I have to…

Thanks for flagging -- this seems to be a common point of confusion; I'll work on updating the readme.

Let's say reference.srt is a set of English subtitles, and you have an out-of-alignment French subtitles file unsynchronized.srt. You could use the video directly as a reference in order to synchronize the French subtitles, but it will take a bit longer since we need to extract audio and perform voice detection over the whole video file. In this case, it will be faster to use your already-synchronized English subtitles file reference.srt.

Re: Show HN: Automatically synchronize subtitles with video

#22

Post by anyfoo »

From the README:

"[...] the naive O(n^2) strategy for scoring all alignments is unacceptable. Instead, we use the fact that "scoring all alignments" is a convolution operation and can be implemented with the Fast Fourier Transform (FFT), bringing the complexity down to O(n log n)."

I absolutely love it when something that, at very first glance, has no business in being solved in the frequency domain, gets solved in the frequency domain.

Re: Show HN: Automatically synchronize subtitles with video

#23

Post by jmkb »

Outpox wrote: post #14
acqq wrote: post #8

"Although it can usually work if all you have is the video file, it will be faster (and potentially more accurate) if you have a correctly synchronized "reference" srt file, in which case you can do the following: subsync reference.srt -i unsynchronized.srt -o synchronized.srt" I really don't understand what the reference.srt is supposed to be here. Edit: If I have a "reference" which is synchronized why do I have to…

I believe it's a typo and should read "Although it can usually work if all you have is the subtitles file [...]". This means that you would use a reference file that would allow you to synchronize your subtitles.

I could imagine having a file with correctly-synced subs in the wrong language. Maybe that would be a good reference file.

(I think you're correct subbing "subtitles file" for "video file".)

Re: Show HN: Automatically synchronize subtitles with video

#24

Post by iforgotpassword »

dearrifling wrote: post #4

Awesome! According to the description of the internals this only works with offsets and doesn't adjust the subtitle's playback speed. I think playback speed variations are not that broad, it's usually caused by playing a subtitle written for a given FPS played back on a different FPS video. It might worth it to try common FPS ratios at once and keep the best match.

Yes that would be great, happens a lot when moving between pal and NTSC regions.

Re: Show HN: Automatically synchronize subtitles with video

#25

Post by zeotroph »

guessmyname wrote: post #16
kojackst wrote: post #9

Sorry for the dumb question, but what exactly does this do? `subsync reference.srt -i unsynchronized.srt -o synchronized.srt` I mean, I already have a synchronized srt file, so what would I be syncing here?

I haven’t tried this command, but I believe the use case is as follows: • Open audio/video file with —supposedly— synchronized file • After a few seconds, I realize the subtitles appear before/after the dialogues • I immediately close the multimedia player, and open the Terminal • I execute the “subsync” command which does who knows what • Open the SRT and discover that the subtitles are now in the correct timestamps…

Previously: Fiddle with the subtitle/audio offset factor (and then they drift apart again slowly, driving you mad!)

Now: subsync

Soon: Players run subsync internally the press of one button or commandline switch.

The voice audio detection and then mapping is such a neat solution. I would have embedded parts of the surrounding audio in some base64 format into the subtitle file and then used that as an alignment clue. But this won't work when the languages don't match.

Re: Show HN: Automatically synchronize subtitles with video

#26

Post by smacke »

IshKebab wrote: post #11

Surely you can do much better than this using a speech recogniser?

By speech recognizer, are you referring to a method that would perform speech-to-text transcription? If so, you could probably do better when your video and subtitles are in the same language, but you would lose the ability to be language-agnostic (unless you manage to solve machine translation well enough :)

Interestingly enough, I haven't actually found any cases where the synchronization doesn't work (assuming the only problem with the subtitles file is a time offset), so it actually looks like simple voice detection might be good enough for the target use-case of longer TV episodes and movies, although further evaluation is necessary.

Re: Show HN: Automatically synchronize subtitles with video

#28

Post by iforgotpassword »

IshKebab wrote: post #11

Surely you can do much better than this using a speech recogniser?

Why? Detecting whether there is speech or not should be enough to align video and subs. It's not like the word order would be scrambled in the subtitle file and needs to be sorted according to the video. :)

Re: Show HN: Automatically synchronize subtitles with video

#29

Post by aasasd »

dearrifling wrote: post #4

Awesome! According to the description of the internals this only works with offsets and doesn't adjust the subtitle's playback speed. I think playback speed variations are not that broad, it's usually caused by playing a subtitle written for a given FPS played back on a different FPS video. It might worth it to try common FPS ratios at once and keep the best match.

In the case when resulting subtitles turn out to be shorter than the input file, subtitle editing software like Aegisub can automatically trim ending times so they don't overlap with starting times.

This could probably be similarly done in the script itself in the future.

If the result gets longer and the times for each line are a bit too short―I guess you'll have to read faster, har har.

---

Actually, iirc Aegisub can also do automatic stretching for different FPS, since that's a rather easy case.

However, it's not enough when the subs are from a different media release or an edit―which made me long for exactly the kind of solution like in the post.

Re: Show HN: Automatically synchronize subtitles with video

#30

Post by smacke »

symstym wrote: post #27

Nice! Have you seen this similar project? If not sure if there are any ideas to be borrowed, but it sounds like it achieved excellent accuracy: https://github.com/AlbertoSabater/subtitle-synchronization

Interesting! I was not aware of that project before. It looks like it could be worthwhile to incorporate their neural net-based VAD, if it's not too slow. At a first glance, it looks like the main difference is the postprocessing step -- they use some heuristics to avoid trying all possible alignments, while subsync uses FFT as its secret sauce to get away with trying out all of them. :)

I should include in the readme that for >1 hour movies, it usually finishes the synchronization in ~20 seconds, which compares favorably to the project linked in your comment (13 minutes unoptimized, 2 minutes optimized).

Post reply on HN