Weak supervision to isolate sign language communicators in crowded news videos
1–10 of 53 posts
Re: Weak supervision to isolate sign language communicators in crowded news videos
#2> I believe that we can solve continuous sign language translation convincingly
American Sign Language is not English, in fact, it's not even particularly close to English. Much of the language is conveyed with body movements outside of the hands and fingers, particularly with facial expressions and "named placeholders."
> All this is to say, that we need to build a 5000 hour scale dataset for Sign Language Translation and we are good to go. But where can we find this data? Luckily news broadcasters often include special news segments for the hearing-impaired.
You need _way_ more than just 5000 hours of video. People who are deaf of hard of hearing, in my experience, dislike the interpreters in news broadcasts. It's very difficult, as an interpreter, to provide _worthwhile_ translations of what is being spoken _as_ it is being spoken.
It's more of a bad and broken transliteration that if you struggle to think about you can parse out and understand.
The other issue is most interpreters are hearing and so use the language slightly differently from actual deaf persons, and training on this on news topics will make it very weak when it comes to understanding and interpreting anything outside of this context. ASL has "dialects" and "slang."
Hearing people always presume this will be simple. They should really just take an ASL class and worth with deaf and hearing impaired people first.
Re: Weak supervision to isolate sign language communicators in crowded news videos
#3> I believe that we can solve continuous sign language translation convincingly American Sign Language is not English, in fact, it's not even particularly close to English. Much of the language is conveyed with body movements outside of the hands and fingers, particularly with facial expressions and "named placeholders." > All this is to say, that we need to build a 5000 hour scale dataset for Sign Language Translati…
Lifeprint.org has plenty of free asl courses taught by a deaf person. Highly recommended for everyone but as with any language it takes a lot of study to be useful.
Re: Weak supervision to isolate sign language communicators in crowded news videos
#4> I believe that we can solve continuous sign language translation convincingly American Sign Language is not English, in fact, it's not even particularly close to English. Much of the language is conveyed with body movements outside of the hands and fingers, particularly with facial expressions and "named placeholders." > All this is to say, that we need to build a 5000 hour scale dataset for Sign Language Translati…
The blog post references translating between English and Indian Sign Language (ISL). I interpreted that to mean translating between spoken English and ISL, not ASL and ISL.
Regardless, I’m curious how (dis)similar ISL is to ASL.
Re: Weak supervision to isolate sign language communicators in crowded news videos
#5> I believe that we can solve continuous sign language translation convincingly American Sign Language is not English, in fact, it's not even particularly close to English. Much of the language is conveyed with body movements outside of the hands and fingers, particularly with facial expressions and "named placeholders." > All this is to say, that we need to build a 5000 hour scale dataset for Sign Language Translati…
I know an interpreter who is a CODA. Her first language was sign language, which I think helps a lot. I once asked her if she thought in English or ASL and she said ASL.
During the pandemic she’d get very frustrated by the ASL she saw on the news. Her mom and deaf friends couldn’t understand them. It wasn’t long before she was on the news regularly to make sure better information was going out. She kept getting COVID, because she refused to wear a mask while working, because coving up the face would make it more difficult to convey the message. I had to respect the dedication.
Re: Weak supervision to isolate sign language communicators in crowded news videos
#6> I believe that we can solve continuous sign language translation convincingly American Sign Language is not English, in fact, it's not even particularly close to English. Much of the language is conveyed with body movements outside of the hands and fingers, particularly with facial expressions and "named placeholders." > All this is to say, that we need to build a 5000 hour scale dataset for Sign Language Translati…
[deleted]
Re: Weak supervision to isolate sign language communicators in crowded news videos
#7> I believe that we can solve continuous sign language translation convincingly American Sign Language is not English, in fact, it's not even particularly close to English. Much of the language is conveyed with body movements outside of the hands and fingers, particularly with facial expressions and "named placeholders." > All this is to say, that we need to build a 5000 hour scale dataset for Sign Language Translati…
Lifeprint.org has plenty of free asl courses taught by a deaf person. Highly recommended for everyone but as with any language it takes a lot of study to be useful.
.org landed on a squatting page, I suppose you mean https://lifeprint.com/asl101/lessons/lessons.htm
Re: Weak supervision to isolate sign language communicators in crowded news videos
#8Earlier quoted context omitted.
Lifeprint.org has plenty of free asl courses taught by a deaf person. Highly recommended for everyone but as with any language it takes a lot of study to be useful.
.org landed on a squatting page, I suppose you mean https://lifeprint.com/asl101/lessons/lessons.htm
Just spent 5 minutes following along with the first video. Very clear and friendly instructor
Re: Weak supervision to isolate sign language communicators in crowded news videos
#9> I believe that we can solve continuous sign language translation convincingly American Sign Language is not English, in fact, it's not even particularly close to English. Much of the language is conveyed with body movements outside of the hands and fingers, particularly with facial expressions and "named placeholders." > All this is to say, that we need to build a 5000 hour scale dataset for Sign Language Translati…
Thanks for the feedback. You raise great points and this was the reason why we wrote this post, so that we can hear from people where the actual problem lies.
On a related note, this sort of explains why our model is struggling to fit on 500 hours of our current dataset (even on the training set). Even so, the current state of automatic translation for Indian Sign Language is that, in-the-wild, even individual words cannot be detected very well. We hope that what we are building might at least improve the state-of-the-art there.
> It's more of a bad and broken transliteration that if you struggle to think about you can parse out and understand.
Can you elaborate a bit more on this. Do you think if we make a system for bad/broken transliteration and funnel it through ChatGPT, it might give meaningful results? That is ChatGPT might be able to correct for errors as it is a strong language model.
Re: Weak supervision to isolate sign language communicators in crowded news videos
#10> I believe that we can solve continuous sign language translation convincingly American Sign Language is not English, in fact, it's not even particularly close to English. Much of the language is conveyed with body movements outside of the hands and fingers, particularly with facial expressions and "named placeholders." > All this is to say, that we need to build a 5000 hour scale dataset for Sign Language Translati…
The blog post references translating between English and Indian Sign Language (ISL). I interpreted that to mean translating between spoken English and ISL, not ASL and ISL. Regardless, I’m curious how (dis)similar ISL is to ASL.
That is correct. We want to translate between English and ISL. English, because it is by and large the language of the Web and I think we should try to connect ISL to it rather than Indian Languages.
From my understanding, they are quite dissimilar. A person who knows ISL will not understand ASL, for example.