Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Weak supervision to isolate sign language communicators in crowded news videos

vrroom.github.io

Post reply on HN
Print view View original

11–20 of 53 posts

Re: Weak supervision to isolate sign language communicators in crowded news videos

#11

Post by kobalsky »

akira2501 wrote: post #2

> I believe that we can solve continuous sign language translation convincingly American Sign Language is not English, in fact, it's not even particularly close to English. Much of the language is conveyed with body movements outside of the hands and fingers, particularly with facial expressions and "named placeholders." > All this is to say, that we need to build a 5000 hour scale dataset for Sign Language Translati…

> It's more of a bad and broken transliteration that if you struggle to think about you can parse out and understand.

it seems to be more common to see sign language interpreters now. is it just virtue signaling to have that instead of just closed captions?

Re: Weak supervision to isolate sign language communicators in crowded news videos

#12

Post by jallmann »

kobalsky wrote: post #11
akira2501 wrote: post #2

> I believe that we can solve continuous sign language translation convincingly American Sign Language is not English, in fact, it's not even particularly close to English. Much of the language is conveyed with body movements outside of the hands and fingers, particularly with facial expressions and "named placeholders." > All this is to say, that we need to build a 5000 hour scale dataset for Sign Language Translati…

> It's more of a bad and broken transliteration that if you struggle to think about you can parse out and understand. it seems to be more common to see sign language interpreters now. is it just virtue signaling to have that instead of just closed captions?

Many deaf people do prefer sign language as an accessibility option over reading captions, even if the interpreting can be hit-or-miss.

Re: Weak supervision to isolate sign language communicators in crowded news videos

#13

Post by matroid »

kobalsky wrote: post #11
akira2501 wrote: post #2

> I believe that we can solve continuous sign language translation convincingly American Sign Language is not English, in fact, it's not even particularly close to English. Much of the language is conveyed with body movements outside of the hands and fingers, particularly with facial expressions and "named placeholders." > All this is to say, that we need to build a 5000 hour scale dataset for Sign Language Translati…

> It's more of a bad and broken transliteration that if you struggle to think about you can parse out and understand. it seems to be more common to see sign language interpreters now. is it just virtue signaling to have that instead of just closed captions?

Also, in India, many hearing-impaired people know only ISL.

Re: Weak supervision to isolate sign language communicators in crowded news videos

#14

Post by wizzwizz4 »

matroid wrote: post #9
akira2501 wrote: post #2

> I believe that we can solve continuous sign language translation convincingly American Sign Language is not English, in fact, it's not even particularly close to English. Much of the language is conveyed with body movements outside of the hands and fingers, particularly with facial expressions and "named placeholders." > All this is to say, that we need to build a 5000 hour scale dataset for Sign Language Translati…

Thanks for the feedback. You raise great points and this was the reason why we wrote this post, so that we can hear from people where the actual problem lies. On a related note, this sort of explains why our model is struggling to fit on 500 hours of our current dataset (even on the training set). Even so, the current state of automatic translation for Indian Sign Language is that, in-the-wild, even individual words…

I think you think it's a magic box. There's not actually such thing as a "strong language model", not in the way you're using the concept.

> We hope that what we are building might at least improve the state-of-the-art there.

Do you have any theoretical arguments for how and why it would improve it? If not, my concern is that you're just sucking the air out of the room. (Research into "throw a large language model at the problem" doesn't tend to produce any insight that could be used by other approaches, and doesn't tend to work, but it does funnel a lot of grant funding into cloud providers' pockets.)

Re: Weak supervision to isolate sign language communicators in crowded news videos

#15

Post by jallmann »

Sign languages have such enormous variability that I have always thought having fluent sign language recognition / translation probably means we have solved AGI.

Detecting the presence of sign language in a video is an interesting subset of the problem and is important for building out more diverse corpora. I would also try to find more conversational sources of data, since news broadcasts can be clinical as others have mentioned. Good luck.

Re: Weak supervision to isolate sign language communicators in crowded news videos

#16

Post by WesternWind »

akira2501 wrote: post #2

> I believe that we can solve continuous sign language translation convincingly American Sign Language is not English, in fact, it's not even particularly close to English. Much of the language is conveyed with body movements outside of the hands and fingers, particularly with facial expressions and "named placeholders." > All this is to say, that we need to build a 5000 hour scale dataset for Sign Language Translati…

Just to note this is for ISL, Indian Sign Language, not ASL, American Sign Language.

Re: Weak supervision to isolate sign language communicators in crowded news videos

#17

Post by hi-v-rocknroll »

I'm wondering how long it will take for LLMs to be able to generate complete (one of many) sign language(s) on-the-fly and put the various sign language(s) translators out of a job. The crux seems to be that sign language differs significantly from spoken language and includes facial movements and nonverbal emotional tonality.

Re: Weak supervision to isolate sign language communicators in crowded news videos

#18

Post by bluGill »

kobalsky wrote: post #11
akira2501 wrote: post #2

> I believe that we can solve continuous sign language translation convincingly American Sign Language is not English, in fact, it's not even particularly close to English. Much of the language is conveyed with body movements outside of the hands and fingers, particularly with facial expressions and "named placeholders." > All this is to say, that we need to build a 5000 hour scale dataset for Sign Language Translati…

> It's more of a bad and broken transliteration that if you struggle to think about you can parse out and understand. it seems to be more common to see sign language interpreters now. is it just virtue signaling to have that instead of just closed captions?

Sign languages is nothing like spoken language and so reading a spoken language is hard.

Re: Weak supervision to isolate sign language communicators in crowded news videos

#19

Post by School-Cotton »

akira2501 wrote: post #2

> I believe that we can solve continuous sign language translation convincingly American Sign Language is not English, in fact, it's not even particularly close to English. Much of the language is conveyed with body movements outside of the hands and fingers, particularly with facial expressions and "named placeholders." > All this is to say, that we need to build a 5000 hour scale dataset for Sign Language Translati…

> American Sign Language is not English

I'm not sure I understand your point. Chinese is also not English but machine translation of Chinese to English can be done.

You're right that laypeople often assume, wrongly, that a given country's sign language is an encoding of the local spoken language. In reality it's usually a totally different language in its own right. But that shouldn't mean that translation is fundamentally impossible.

Re: Weak supervision to isolate sign language communicators in crowded news videos

#20

Post by KeepFlying »

hi-v-rocknroll wrote: post #17

I'm wondering how long it will take for LLMs to be able to generate complete (one of many) sign language(s) on-the-fly and put the various sign language(s) translators out of a job. The crux seems to be that sign language differs significantly from spoken language and includes facial movements and nonverbal emotional tonality.

The fact that hearing impaired people prefer ASL interpreters to closed captioning tells me automated translation will never be enough.

It's the same reason we prefer interpreters to Google Translate when the message is important.

Interpretation adds the nuance of a human that all the automatic tools miss.

I'm sure it could make a small dent in the market for interpreters but only a small one.

Post reply on HN