AI could end foreign-language subtitles • Hacker News Board

Re: AI could end foreign-language subtitles

#111

Post by marcan_42 » Thu, Jan 20, 2022, 2:29 AM UTC

aeturnum wrote: post #58

jmckib wrote: post #43

Earlier quoted context omitted.

I love subtitles, but for me they tend to spoil the timing of the lines being delivered, especially with comedy. Unfortunately subtitles are necessary even when I'm able to play audio as loud as I want, because it's almost always hard to understand at least 10-20% of the words that actors are saying. Maybe it would be nice to have subtitles that appear a word at a time as they are spoken, but I imagine that's not ver…

I would also love to a more responsive subtitle format! I've seen some really wonderful custom jobs where people use different subtitle colors for different speakers, place subtitles near the speaker in the frame, etc. I think there is a ton of room for improving the expressiveness of the medium.

The format exists, it's called ASS. It can do things like this:

https://youtu.be/djrJP_Pi03U

https://youtu.be/R3shsgKbd_M?t=12s

That's a pure text format; no bitmaps involved, just markup and TTF fonts. It can do vector shapes and perspective positioning. These are somewhat overdone examples, but one more common nice use case is literally translating signs on buildings, objects and such, seamlessly, while preserving the image style. The whole thing is rendered in real time and you can toggle the subs off and see the original frame.

https://youtu.be/n_c-eKTisI0

Again, that's all markup rendered on the video in real time from fonts. Yes, really.

It is the standard in the anime fansub community, but has seen little adoption elsewhere. Partially this is because it seems commercial operations rarely have much interest in subtitle quality or technology.

Incidentally, even YouTube supports an undocumented subtitle format that can support a decent amount of styling (YTT). Unfortunately, support is inconsistent across web/mobile versions, and it looks best when viewed in a browser. I've used it to do things like this (turn on captions):

https://www.youtube.com/watch?v=dh1BCyjCbes

https://www.youtube.com/watch?v=IxLI7KmzOjE

You can convert a subset of ASS to that format with this tool: https://github.com/arcusmaximus/YTSubConverter

Re: AI could end foreign-language subtitles

#112

Post by rexreed » Thu, Jan 20, 2022, 2:33 AM UTC

This article is a thinly veiled promotion for Veritone's MARVEL.ai which doesn't even do translation into foreign languages. It's a text-to-speech encoder that has a variety of voices trained for different languages. The actual translations and subtitles need to be done most likely still by human workers or human-supervised machine translation. The article never really makes the point of the title which is that foreign-language subtitles will be ending any time soon.

There should be a disclaimer on this advertorial piece.

Re: AI could end foreign-language subtitles

#113

Post by purplepatrick » Thu, Jan 20, 2022, 2:33 AM UTC

I second the importance of subtitles. I grew up in a country where dubbing is a huge industry and everything is dubbed. I’ve also lived in another country where almost nothing is dubbed. After moving to the US and seeing many of the films I had previously only known in their dubbed version, I can no longer stand dubbing. I can tell that something is dubbed without having to look at the screen. I’d never want to not have subtitles — I’d rather not watch at all…

That being said, as I understand the article this marvel ai thing is supposed to replicate the original performance using the original voice in another language? Of course that would be fantastic a solution, but I doubt that the technology is anything near what it wants to do…

Re: AI could end foreign-language subtitles

#114

Post by pavlov » Thu, Jan 20, 2022, 2:42 AM UTC

I want the opposite: subtitles on everything, just not directly on the picture.

Opera houses have subtitles on a separate text-only screen (or screens embedded in seats). Why can’t I have a device like that at home?

A narrow subtitle display that sits below the TV, sort of like the Touch Bar on the previous-generation MacBooks. I’d buy that.

Re: AI could end foreign-language subtitles

#115

Post by joemelonyeah » Thu, Jan 20, 2022, 2:44 AM UTC

AI can generate voices similar to the original actor, but the script will still have to be translated and proofread by expert translators. No AI can fully replace them.

Re: AI could end foreign-language subtitles

#116

Post by throwaway675309 » Thu, Jan 20, 2022, 2:47 AM UTC

aeturnum wrote: post #3

This would be a lovely option to have and I find it hilarious that anyone imagines that subtitles are some kind of problem we are trying to get out of. Adding text to movies has a huge array of advantages: it preserves the original performances, it makes the dialog accessible to people with hearing or auditory processing disabilities, they can be multiplexed in a way that audio cannot, etc. Obviously it would be wond…

Then you're simply not thinking about it deeply enough. There are plenty of shows that someone may want to watch but may not be able to invest 100% of my attention watching that show, therefore with a show dubbed in their native language, they can still have the show in the background while being able to listen to the audio.

With subtitles one has to constantly be looking at the actual show itself. Furthermore if one makes use picture in picture such as on a mobile platform or a tablet device the subtitles are so correspondingly small as to be basically hieroglyphic at that point.

Re: AI could end foreign-language subtitles

#117

Post by KeepFlying » Thu, Jan 20, 2022, 2:47 AM UTC

pavlov wrote: post #114

I want the opposite: subtitles on everything, just not directly on the picture. Opera houses have subtitles on a separate text-only screen (or screens embedded in seats). Why can’t I have a device like that at home? A narrow subtitle display that sits below the TV, sort of like the Touch Bar on the previous-generation MacBooks. I’d buy that.

Well now you've got me thinking. That has to be very feasible for any content you own the file for, but I wonder how steaming services currently handle subtitles. Are they embedded into the video stream or are they their own channel? Could a third party pull them out and pass them along to a separate device like that?

Does anyone with more experience have any idea?

Re: AI could end foreign-language subtitles

#118

Post by edgyquant » Thu, Jan 20, 2022, 2:52 AM UTC

tompetry wrote: post #80

aeturnum wrote: post #3

This would be a lovely option to have and I find it hilarious that anyone imagines that subtitles are some kind of problem we are trying to get out of. Adding text to movies has a huge array of advantages: it preserves the original performances, it makes the dialog accessible to people with hearing or auditory processing disabilities, they can be multiplexed in a way that audio cannot, etc. Obviously it would be wond…

I am very surprised more people watched the dubbed version of Squid Game over the subtitles. I saw a friend watching the dubbed version and to me it didn't "preserve the original performances" as you say. Surprised subtitles are a barrier, I enjoy them personally, but I guess they have the data to know. I have tested out services that translate your "own voice" to another language and it was pretty cool, although I c…

Most people are fine with them. My dad has hearing problems so we always watched movies with the old fashion “white on black bar” closed captioning. As an adult I just get distracted by them and it prevents me from watching the movie.

Re: AI could end foreign-language subtitles

#119

Post by endisneigh » Thu, Jan 20, 2022, 2:53 AM UTC

thaumasiotes wrote: post #93

endisneigh wrote: post #12

What’s really surprising to me is that synching subtitles isn’t a solved problem. Why can’t someone just loosely transcribe without time stamps and sync it to the video?

Because synching subtitles to the video isn't even a goal. Most people read more slowly than they can listen. Many other people can read faster than they can listen. But you need subtitles that stick around long enough for the first group, and really, you want a comfortable margin so that people aren't frantically trying to finish reading every subtitle before it disappears. I watched the show 大江大河 ("Like a Flowing R…

A system that could do what I’m describing would also solve the problem you mention tho

Re: AI could end foreign-language subtitles

#120

Post by kpozin » Thu, Jan 20, 2022, 3:05 AM UTC

fxtentacle wrote: post #50

aeturnum wrote: post #3

This would be a lovely option to have and I find it hilarious that anyone imagines that subtitles are some kind of problem we are trying to get out of. Adding text to movies has a huge array of advantages: it preserves the original performances, it makes the dialog accessible to people with hearing or auditory processing disabilities, they can be multiplexed in a way that audio cannot, etc. Obviously it would be wond…

Agree. Dolby Atmos releases tend to have a lot of noisy background sounds all around you, and then people silently mumble on the center channel. My English is really good, but some movies are still difficult to watch without subtitles. For example, Hawkeye jumped in between "painfully loud effects" and "barely audible dialog".

This is actually an improvement over stereo soundtracks, as you can just raise the relative volume of the center channel!