Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI could end foreign-language subtitles

axios.com

Post reply on HN
Print view View original

51–60 of 221 posts

Re: AI could end foreign-language subtitles

#51

Post by cellis »

echelon wrote: post #9

Absolutely coming. Goupon founder Andrew Mason's new startup Descript does a fantastic job editing podcasts from text. This same sort of tooling is absolutely going to make it into film. This sort of stuff is already trivially possible with zero effort: https://fakeyou.com/w2l/result/WR:t5m9xayqcwkjf9wng3nb4sxgzh... In ten years, we'll be able to reposition actors and fix flubbed takes. We won't have to visit set, us…

The problem is it's not very good and there is a vast uncanny valley. Phoneme / voice conversion is an incredibly difficult problem and not likely to be solved with any techniques known today, least ways not cheaply. Transformers show some ability to get it right , but if you wanted to use e.g. a Seq2Seq model you'd be hard pressed to come up with enough data that would generalize.

Re: AI could end foreign-language subtitles

#52

Post by ASalazarMX »

fxtentacle wrote: post #50
aeturnum wrote: post #3

This would be a lovely option to have and I find it hilarious that anyone imagines that subtitles are some kind of problem we are trying to get out of. Adding text to movies has a huge array of advantages: it preserves the original performances, it makes the dialog accessible to people with hearing or auditory processing disabilities, they can be multiplexed in a way that audio cannot, etc. Obviously it would be wond…

Agree. Dolby Atmos releases tend to have a lot of noisy background sounds all around you, and then people silently mumble on the center channel. My English is really good, but some movies are still difficult to watch without subtitles. For example, Hawkeye jumped in between "painfully loud effects" and "barely audible dialog".

Eating pop corn while watching a movie has become increasingly difficult with this kind of sound mixing.

Re: AI could end foreign-language subtitles

#53

Post by cellis »

nostrademons wrote: post #36
dmitriid wrote: post #17

This will not happen for many, many years to come. That is, companies will try it, there will be some bad and ridiculous results, there will be pusheback and the tech will largely be ignored for a decade or so. Main problems will be intonation and inflection. What is a question to an English speaker is a regular sentence to a Russian. And so on. "Petabytes of data from primarily English-speaking libraries" mean jack…

I'd encourage you to listen to some cutting edge speech synthesis like Google Duplex: https://www.youtube.com/watch?v=ijwHj2HaOT0 This is live in production today - business owners are getting calls from robots, and you can trigger it yourself with Google Assistant. [Disclaimer: I work(ed) on Google Assistant - I'm in the process of transferring out - and have listened to some presentations about how Duplex works. It…

It's still not very good. The state of the art vocoders ( StyleTron et al ) are years away from getting intonation right. But then if you could get intonation correct, you still need to be able to get pausing and pitch correct, let alone timbre and a few other things I'm forgetting from my research.

Re: AI could end foreign-language subtitles

#55

Post by throw_away »

xiii1408 wrote: post #10
throw_away wrote: post #5

I would much rather we use AI to improve subtitling. Better auto speech recognition, per-word millisecond synchronization, positioning based on speaker/screen content-- all these could improve the subtitled experience and would be way less creepy than deepfake voices.

Yes! My perennial problem is that most Chinese TV content has excellent subtitles, but they're burned in to the videos. So if I want to watch a Chinese show with a friend who doesn't read Chinese, there's no option even for auto-translated subtitles. I've often thought of writing a script to generate subtitle files using text recognition, but haven't gotten around to it.

Ah, but one thing I miss about the burned-in subtitles was how, on VHS at least, one could fast forward like 2-3x and still be able to read the captions. I double-lazied my way out of more than one high school book review where a boring book was made into a boring movie.

Re: AI could end foreign-language subtitles

#56

Post by jonpurdy »

nkozyra wrote: post #42
aeturnum wrote: post #3

This would be a lovely option to have and I find it hilarious that anyone imagines that subtitles are some kind of problem we are trying to get out of. Adding text to movies has a huge array of advantages: it preserves the original performances, it makes the dialog accessible to people with hearing or auditory processing disabilities, they can be multiplexed in a way that audio cannot, etc. Obviously it would be wond…

> I find it hilarious that anyone imagines that subtitles are some kind of problem we are trying to get out of. I think there's a lot of gatekeeping about this. People - perhaps rightfully so - believe subtitled versions to be more ... "pure" than dubbed versions. I don't watch any language dubbed movies, but I wouldn't chide anyone who did. Not everyone wants to spend the whole two hours glancing up and down across…

> I think there's a lot of gatekeeping about this. People - perhaps rightfully so - believe subtitled versions to be more ... "pure" than dubbed versions.

Well, they are more pure than dubbed versions. There is so much nuance and tone in the original language that is either left out or changed in the dub. Not to mention word order (subject-verb-object in English/Chinese, subject-object-verb in Japanese/Korean) that further changes the timing of impactful words.

I am not looking forward to having to watch the dub versions of Totoro and other Miyazaki films when my daughter is between 2-7 years old...

Re: AI could end foreign-language subtitles

#57

Post by aeturnum »

commandlinefan wrote: post #39
aeturnum wrote: post #3

This would be a lovely option to have and I find it hilarious that anyone imagines that subtitles are some kind of problem we are trying to get out of. Adding text to movies has a huge array of advantages: it preserves the original performances, it makes the dialog accessible to people with hearing or auditory processing disabilities, they can be multiplexed in a way that audio cannot, etc. Obviously it would be wond…

> subtitles are some kind of problem Heck, they're not even limited to foreign languages. I put them on in English because I'm starting to lose my hearing now in my late 40's and if I turn up the volume loud enough that I can hear the dialogue ok, the sound effects would be so loud they'd set off the neighbor's car alarm.

I also use them regularly in my native language! Very helpful in noisy environments.

Re: AI could end foreign-language subtitles

#58

Post by aeturnum »

jmckib wrote: post #43
aeturnum wrote: post #3

This would be a lovely option to have and I find it hilarious that anyone imagines that subtitles are some kind of problem we are trying to get out of. Adding text to movies has a huge array of advantages: it preserves the original performances, it makes the dialog accessible to people with hearing or auditory processing disabilities, they can be multiplexed in a way that audio cannot, etc. Obviously it would be wond…

I love subtitles, but for me they tend to spoil the timing of the lines being delivered, especially with comedy. Unfortunately subtitles are necessary even when I'm able to play audio as loud as I want, because it's almost always hard to understand at least 10-20% of the words that actors are saying. Maybe it would be nice to have subtitles that appear a word at a time as they are spoken, but I imagine that's not ver…

I would also love to a more responsive subtitle format! I've seen some really wonderful custom jobs where people use different subtitle colors for different speakers, place subtitles near the speaker in the frame, etc. I think there is a ton of room for improving the expressiveness of the medium.

Re: AI could end foreign-language subtitles

#60

Post by epolanski »

commandlinefan wrote: post #39
aeturnum wrote: post #3

This would be a lovely option to have and I find it hilarious that anyone imagines that subtitles are some kind of problem we are trying to get out of. Adding text to movies has a huge array of advantages: it preserves the original performances, it makes the dialog accessible to people with hearing or auditory processing disabilities, they can be multiplexed in a way that audio cannot, etc. Obviously it would be wond…

> subtitles are some kind of problem Heck, they're not even limited to foreign languages. I put them on in English because I'm starting to lose my hearing now in my late 40's and if I turn up the volume loud enough that I can hear the dialogue ok, the sound effects would be so loud they'd set off the neighbor's car alarm.

many reasons to use them:

- family is sleeping so you keep volume low

- hard to understand accents or lingo

- bad audio mixing or bad audio systems that muffle words and rest of the sound

Post reply on HN