Live data from Hacker News

Open-source tool translates and dubs videos into other languages using AI

github.com

81–90 of 119 posts

Re: Open-source tool translates and dubs videos into other languages using AI

#81
post #65

This could be useful in combating fake news. In many videos especially in political news, foreign languages are dub over with sometimes nuanced translation that can skew audiences to (mis)understand the content in certain ways.

A translation lacking nuance/precision (due to being the work of machine learning) can also cause significant misunderstandings, though. I'm not sure you win or lose in that regard by switching from humans to machines.

Re: Open-source tool translates and dubs videos into other languages using AI

#82
post #55

Unlike other methods of automation, AI is replacing human beings too fast. And before you say, "new jobs will be created" -- look at history. After the computer, new jobs have been created, but what kind of jobs? Every year, we are becoming more entwined in wage slavery as the wealth accumulates at the top and jobs become more meaningless. So, no, new jobs will not be created, except the kind of jobs that crush the h…

> After the computer, new jobs have been created, but what kind of jobs? Every year, we are becoming more entwined in wage slavery as the wealth accumulates at the top and jobs become more meaningless. What are you talking about? Many of us have tech jobs with much more comfort, creativity and autonomy than the jobs they displaced, and computerisation has made it much more practical for those who dare to strike out t…

I know, right. Jobs created by computers have by far been the best paying, and most comfortable jobs that exist in human history. We're privileged.

Re: Open-source tool translates and dubs videos into other languages using AI

#83
post #76
post #69

Earlier quoted context omitted.

> (if we can ignore the ethics of using the actor's voice and the loss of work for the dubbing companies and actors) This is a shocking parenthetical.

Optimistically, maybe we can replace the original actor with a synthetic voice as well. Then the issue is largely moot.

I realise that for dubbing (and voice actors in general), they have the least amount of "potential fallback legal protections" in the context of automation via generative AI:

Translating? Machine translation is already well established.

Copyright? IIRC the most popular voice generation company (ElevenLabs) uses copyright-safe models, where the sources for the base model were already consenting.

Likeness? As you said, just use a synthetic voice.

Replacement of jobs? Not really a legal issue. It's not much worse than self checkouts or driverless cars, for example. The only reason we're talking about it is because it affects white-collar workers and not blue-collar workers, and voice actors are more likely to be celebrities than cashiers, for example.

Re: Open-source tool translates and dubs videos into other languages using AI

#84
post #65

This could be useful in combating fake news. In many videos especially in political news, foreign languages are dub over with sometimes nuanced translation that can skew audiences to (mis)understand the content in certain ways.

A translation lacking nuance/precision (due to being the work of machine learning) can also cause significant misunderstandings, though. I'm not sure you win or lose in that regard by switching from humans to machines.

the errors in ML translation would be systemic and less likely to be of human arbitration, which is less dangerous. Human translation could be deliberately misleading at a specific phrase in a specific interview.

Re: Open-source tool translates and dubs videos into other languages using AI

#85

Earlier quoted context omitted.

Voice acting (dubbing) is acting, which is a "high form of labor". In some languages/countries, most of the dubbing is performed by a few extremely good voice actors.

they wont replace voice actors for that. Rather it will replace voice actors for niche languages for niche content. Just like AI generated images doesnt replace artists working on the latest pixar movie but rather lets some small blog ad an image they would never had otherwise.

But there's a lot of people who have been working as voice actors for small/medium visibility content. Toothpaste ads, Kickstarter promo videos, corporate training videos, mid-range video games... a chunk of that is going away, and it's going to hurt a bunch of people who were previously able to make a decent living off it without being a star.

Re: Open-source tool translates and dubs videos into other languages using AI

#86

Localization and dubbing is a sad endeavour. By trying to accomodate everyone's individual preference for information transmission we accomplish nothing more than reducing our ability to understand each other in the long run. Having a Babelfish is all well and good. Until it stops working, and you realise no one can understand each other any more. Ironically localization is often pushed by well meaning Americans who…

Are you saying it's a bad thing if the creator of a work decides they want it localized or dubbed into other languages? I don't understand why you want to take that choice out of their hands. Which languages someone speaks isn't simply a matter of "individual preference". Learning a new language takes a lot of time and energy, and people only have the time to learn a handful of languages in most cases unless they can…

Nah the localization itself is fine. Where it becomes problematic is when there's no opt-out at the app level. Or perhaps it should even be opt-in.

Re: Open-source tool translates and dubs videos into other languages using AI

#87

Huh, if it's good enough, movies/TV shows dubbed with AI-clone of the original voice would be great (if we can ignore the ethics of using the actor's voice and the loss of work for the dubbing companies and actors). For example here's how weird Friends is in German: https://www.youtube.com/watch?v=nCoNSZV--z0 . Or Italian: https://www.youtube.com/watch?v=wO5qTzvyQ1s Can AI detect the emotional tone of sentences yet,…

In my experience, human dubbing never captures the original tone anyway. Probably never can unless it's done by people fluent in both source and target languages and're also good at voice acting. And so I have a huge preference for subs so I can appreciate the nuance in the original voices.

Re: Open-source tool translates and dubs videos into other languages using AI

#88
post #65

This could be useful in combating fake news. In many videos especially in political news, foreign languages are dub over with sometimes nuanced translation that can skew audiences to (mis)understand the content in certain ways.

The AI is quite capable of inserting its own translation errors.

Re: Open-source tool translates and dubs videos into other languages using AI

#89

Earlier quoted context omitted.

Who's upset here about the hordes of horse manure cleaners that lost their jobs due to the advent of cars? Does anyone miss the stench of a 1800s street in the summer heat? No. Lets ask the influencers, the twitch celebs, the podcasters id they would prefer to shovel horse crap, or play with their e-device all day. And to be completely fair, the manufacturing jobs may be gone because elected leaders were told it was…

> Who's upset here about the hordes of horse manure cleaners that lost their jobs due to the advent of cars? False dichotomy. Not every modern-day twitch celeb would be shoveling manure. And another false dichotomy: I was not arguing for either NOW or THE TIME before cars. And I wonder how many people would trade their modern influence job for shoveling manure in return for living in an area with cleaner air. Also, a…

Technology disappears when it becomes ubiquitous. It never actually becomes less, because people don't want less. I can't see for example people wanting there to be fewer lightbulbs, or even refrigerators (mainly for those in warm areas). Everybody wants safe lighting and their food unspoiled after a day (once exposed to these things), and likely won't consider the means to be "tech". And it only improves over time. AI is just another tech that's in the early stage but also on it's way to ubiquity.

Re: Open-source tool translates and dubs videos into other languages using AI

#90

Localization and dubbing is a sad endeavour. By trying to accomodate everyone's individual preference for information transmission we accomplish nothing more than reducing our ability to understand each other in the long run. Having a Babelfish is all well and good. Until it stops working, and you realise no one can understand each other any more. Ironically localization is often pushed by well meaning Americans who…

This is a terrible take, and you should have at least included the "forced" dub disclaimer from your comment below. Without at least one of sub/dub, foreign (relative to your current location!) language content basically doesn't get consumed at all except by a very small minority of people who are very keen on the content anyway - or are speakers of the language anyway.

Now, as veterans of anime forum wars will know, subtitling is nearly always better than dubbing, and I hope this tech is capable of that as well. Most media systems let you put a whole load of subtitle tracks on and then pick one.

There's far, far too much content out there for more than a fraction of it to be ever professionally translated. While we should expect human translation review and a spot of localization for officially released works, most of the internet is just free content being given away for very little return. And that's where automatic translation is going to shine: release the non-English meme champions! Let us have a look in Bilibili!

Post reply on HN