Weak supervision to isolate sign language communicators in crowded news videos • Hacker News Board

Re: Weak supervision to isolate sign language communicators in crowded news videos

#51

Post by matroid » Fri, Aug 16, 2024, 4:47 PM UTC

zie wrote: post #47

1st: I sign ASL not ISL like the OP is talking about. In the ASL world, most news translations into ASL are delayed or sped up from the person talking and/or the captions if they happen to also be available. You are going to have sync problems. Secondly, it's not just moving the hands, body movements, facial expressions, etc all count in ASL , I'm betting they count in ISL as well. Thirdly the quality of interpretati…

Thanks Zie for the message. I'm sorry to hear about your "interpreter" encounter :(

I do think these problems are much, much worse for ISL as you rightly noted.

I think I should have been careful when I said "solve" in my post. But that really came from a place of optimism/excitement.

Re: Weak supervision to isolate sign language communicators in crowded news videos

#52

Post by canjobear » Sat, Aug 17, 2024, 12:53 AM UTC

wizzwizz4 wrote: post #41

canjobear wrote: post #21

Earlier quoted context omitted.

“Throw an LM at it” is the only approach that has ever produced human level machine translation. For theory on how a strong target-language-side LM can improve translation, even in the extreme scenario where no parallel “texts” are available, https://proceedings.neurips.cc/paper_files/paper/2023/file/7...

You're mixing up cause and effect. The transformer architecture was invented for machine translation – and it's pretty good at it! ( Very far from human-level, but still mostly comprehensible, and a significant improvement over the state-of-the-art at time of first publication.) But we shouldn't treat this as anything more than "special-purpose ML architecture achieves decent results". The GPT architecture, using tra…

It’s true that the Transformer architecture was developed for seq2seq MT, but you can get similar performance with Mamba or RWKV or other new non-Transformer architectures. It seems that what is important is having a strong general sequence-learning architecture plus tons of data.

> The GPT architecture, using transformers to do iterated predictive text, is a modern version of the Markov bot.

The Markov nature only matters if the text falls outside the context window.

> Perhaps surprisingly so, until you step back, look at the training data, and look at the information flow: the conditional probability of the next token isn't mostly coming from the source text.

I’m not sure what you’re getting at here. If it’s that you can predict the next token in many cases without looking at the source language, then that’s also true for traditional encoder-decoder architectures, so it’s not a problem unique to prompting. Or are you getting at problems arising from teacher-forcing?

Basically the question was how an LM could possibly help translation, and the answer is that it gives you a strong prior for the decoder. That’s also the basic idea in the theoretical UMT paper: you are trying to find a function from source to target language that produces a sensible distribution as defined by an LM.

Re: Weak supervision to isolate sign language communicators in crowded news videos

#53

Post by zie » Sun, Aug 18, 2024, 1:49 AM UTC

matroid wrote: post #51

zie wrote: post #47

1st: I sign ASL not ISL like the OP is talking about. In the ASL world, most news translations into ASL are delayed or sped up from the person talking and/or the captions if they happen to also be available. You are going to have sync problems. Secondly, it's not just moving the hands, body movements, facial expressions, etc all count in ASL , I'm betting they count in ISL as well. Thirdly the quality of interpretati…

Thanks Zie for the message. I'm sorry to hear about your "interpreter" encounter :( I do think these problems are much, much worse for ISL as you rightly noted. I think I should have been careful when I said "solve" in my post. But that really came from a place of optimism/excitement.

I'm glad you are excited about it. I don't want to discourage you, keep working at it, who knows what might come of it!

Just know, it won't be a weekend hack to solve the problem(s) though.