Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Weak supervision to isolate sign language communicators in crowded news videos

vrroom.github.io

Post reply on HN
Print view View original

31–40 of 53 posts

Re: Weak supervision to isolate sign language communicators in crowded news videos

#31

Post by lolinder »

matroid wrote: post #13
kobalsky wrote: post #11

Earlier quoted context omitted.

> It's more of a bad and broken transliteration that if you struggle to think about you can parse out and understand. it seems to be more common to see sign language interpreters now. is it just virtue signaling to have that instead of just closed captions?

Also, in India, many hearing-impaired people know only ISL.

This is also true in the US. People pick up enough written English to get by most of the time, but it's often quite broken and clearly a second language. I know hearing impaired people native to the US with substantially worse English than the average European.

Re: Weak supervision to isolate sign language communicators in crowded news videos

#32

Post by [deleted] »

fsckboy wrote: post #23
al_borland wrote: post #5

Earlier quoted context omitted.

I know an interpreter who is a CODA. Her first language was sign language, which I think helps a lot. I once asked her if she thought in English or ASL and she said ASL. During the pandemic she’d get very frustrated by the ASL she saw on the news. Her mom and deaf friends couldn’t understand them. It wasn’t long before she was on the news regularly to make sure better information was going out. She kept getting COVID…

> I once asked her if she thought in English or ASL and she said ASL It's common to think that we think in languages, but at a fundamental level, we simply don't. Ever have the experience of not being able to remember the word for something? you know exactly what you are thinking, but can't come up with the word. If you thought in your language, this wouldn't happen.

[deleted]

Re: Weak supervision to isolate sign language communicators in crowded news videos

#33

Post by lolinder »

fsckboy wrote: post #29
lolinder wrote: post #27

Earlier quoted context omitted.

There are layers to thought, and the layer that is most conscious is, at least for me (and from what I hear a lot of other people), in my native language. Further, achieving fluency in a second language is often associated with skipping the intermediate step of translating from English, with thoughts materializing first in the second language. You're correct that there does seem to be a layer lower than that—one that…

you can talk to yourself in your head using your native language. But that's not evidence that you are thinking in your language, it's thinking of your language. when you do math, are you thinking in your language? when you drive a car or play a video game or a sport, or make love, are you thinking in your language? I'll answer for you, no, you aren't. Do people who grow up without language (there have been plenty ex…

I don't think we actually disagree on the facts so much as on the buckets we sort them into.

I very much count my internal English monologue as thinking in my language and would understand that to be what someone is asking about if they asked what language I think in. At the same time, I totally agree that there are layers to thought that never rise into the monologue.

Re: Weak supervision to isolate sign language communicators in crowded news videos

#34

Post by [deleted] »

refulgentis wrote: post #22
al_borland wrote: post #5

Earlier quoted context omitted.

I know an interpreter who is a CODA. Her first language was sign language, which I think helps a lot. I once asked her if she thought in English or ASL and she said ASL. During the pandemic she’d get very frustrated by the ASL she saw on the news. Her mom and deaf friends couldn’t understand them. It wasn’t long before she was on the news regularly to make sure better information was going out. She kept getting COVID…

I got a insanely dumb question, that probably has an obvious answer: why is it so critically important to have ASL? It seems to me you could skip having someone to the side adding the layers of emoting imperfectly and just watch the anchor and captions? Note, I'm definitely the one missing something. When first met with a contradiction, check your premises, is my motto, not lecture on why they're wrong.

[deleted]

Re: Weak supervision to isolate sign language communicators in crowded news videos

#35

Post by refulgentis »

School-Cotton wrote: post #25
refulgentis wrote: post #22

Earlier quoted context omitted.

I got a insanely dumb question, that probably has an obvious answer: why is it so critically important to have ASL? It seems to me you could skip having someone to the side adding the layers of emoting imperfectly and just watch the anchor and captions? Note, I'm definitely the one missing something. When first met with a contradiction, check your premises, is my motto, not lecture on why they're wrong.

What captions? There’s no widely-used written form of ASL. If you meant the English-language captions, well, it should be apparent why some people prefer content to be dubbed in their native language rather than reading subtitles in a different language that they understand less well.

Yes I meant $NATIVE_LANGUAGE_OF_VIEWER, and that example didn't help me I'm afraid, I'm quite a dense one! I appreciated you checking if I meant written ASL captions. :) The example of dubs left me confused because dubs brings in an extra sense that isn't applicable in the ASL case. There, in either case, we're watching someone who isn't the character. I can't square that with the existential fervor the woman in OP felt

Re: Weak supervision to isolate sign language communicators in crowded news videos

#36

Post by thaumasiotes »

amdsn wrote: post #24
refulgentis wrote: post #22

Earlier quoted context omitted.

I got a insanely dumb question, that probably has an obvious answer: why is it so critically important to have ASL? It seems to me you could skip having someone to the side adding the layers of emoting imperfectly and just watch the anchor and captions? Note, I'm definitely the one missing something. When first met with a contradiction, check your premises, is my motto, not lecture on why they're wrong.

Think of it more like "why is it important to have Spanish as an option for captions in an area that knows it has a large audience of L1 Spanish speakers?". English/ is pretty much a foreign language to many signers. Their native language is ASL or whatever other sign language they know, and these languages aren't just 1:1 mappings of words in the local dominant language to hand signals. They have many dimensions of…

> For example, where an English speaker would use words like "very" or "extremely" or choose adjectives with more intense connotations an ASL signer would repeat or exaggerate the sign they want to emphasize

小小比小更小

大大比大更大

If you search for "a big big guy", you'll find no end of English examples, either. I would be surprised if any language didn't use repetition for emphasis.

Re: Weak supervision to isolate sign language communicators in crowded news videos

#37

Post by thaumasiotes »

matroid wrote: post #10
voidingw wrote: post #4

Earlier quoted context omitted.

The blog post references translating between English and Indian Sign Language (ISL). I interpreted that to mean translating between spoken English and ISL, not ASL and ISL. Regardless, I’m curious how (dis)similar ISL is to ASL.

That is correct. We want to translate between English and ISL. English, because it is by and large the language of the Web and I think we should try to connect ISL to it rather than Indian Languages. From my understanding, they are quite dissimilar. A person who knows ISL will not understand ASL, for example.

> A person who knows ISL will not understand ASL, for example.

That isn't the best example; it would still be true of language pairs that were very similar to each other.

Re: Weak supervision to isolate sign language communicators in crowded news videos

#38

Post by drdeca »

fsckboy wrote: post #29
lolinder wrote: post #27

Earlier quoted context omitted.

There are layers to thought, and the layer that is most conscious is, at least for me (and from what I hear a lot of other people), in my native language. Further, achieving fluency in a second language is often associated with skipping the intermediate step of translating from English, with thoughts materializing first in the second language. You're correct that there does seem to be a layer lower than that—one that…

you can talk to yourself in your head using your native language. But that's not evidence that you are thinking in your language, it's thinking of your language. when you do math, are you thinking in your language? when you drive a car or play a video game or a sport, or make love, are you thinking in your language? I'll answer for you, no, you aren't. Do people who grow up without language (there have been plenty ex…

There are kinds of thought which are facilitated by language.

Re: Weak supervision to isolate sign language communicators in crowded news videos

#39

Post by rasur »

al_borland wrote: post #5
akira2501 wrote: post #2

> I believe that we can solve continuous sign language translation convincingly American Sign Language is not English, in fact, it's not even particularly close to English. Much of the language is conveyed with body movements outside of the hands and fingers, particularly with facial expressions and "named placeholders." > All this is to say, that we need to build a 5000 hour scale dataset for Sign Language Translati…

I know an interpreter who is a CODA. Her first language was sign language, which I think helps a lot. I once asked her if she thought in English or ASL and she said ASL. During the pandemic she’d get very frustrated by the ASL she saw on the news. Her mom and deaf friends couldn’t understand them. It wasn’t long before she was on the news regularly to make sure better information was going out. She kept getting COVID…

During Covid, my son (who is deaf and attends a deaf school) has issued masks with transparent windows at the front, especially for assisting with lip-reading for deaf users. This is in Switzerland though - I don't know if this innovation reached across the Atlantic ;)

Re: Weak supervision to isolate sign language communicators in crowded news videos

#40

Post by agarsev »

Sign language researcher here! I would recommend you look a bit at the scientific literature on the topic. I know it can be a bit overwhelming and hard to know to separate the actual info from the garbage, so I can try and select for you a few hand picked papers. IMO, trying to understand sign language oneself, or at least getting basic notions, is fundamental to understand where the real problems lie.

Unfortunately there's no getting away from that. While the scarcity of data indeed is an issue, and your idea is nice (congratulations!) the actual problem is the scarcity of useful data. Since sign language doesn't correspond to the oral language, there are many problems with alignment and just what to translate to. Glosses (oral language words used as representation for signs) are not enough at all, since they don't capture the morphology and grammar of the language, which among other things heavily relies on space and movement. Video + audio/audio captions is nearly useless.

Good luck with your efforts, this is a fascinating area where we get to combine the best of CS, AI, linguistics... but it's hard! As I said, let me know if you want some literature, by PM/email if you want, and I'll get back to you later.

Post reply on HN