Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Sign language translation with smart glasses

github.com

Post reply on HN
Print view View original

1–10 of 12 posts

Show HN: Sign language translation with smart glasses

#1

Post by aadisang »

Hi!

I have relatives that speak sign language, and always found it odd that despite unbelievable advances in AI in recent years, the problem still felt somewhat neglected.

Especially with the advent of wearable tech; to my knowledge, this is the first project that integrates the meta glasses w/ fingerspelling translation software.

As for the technical aspect, I trained a neural net on Google's FSboard dataset modeling a CNN + GRU temporal encoder architecture (trained with CTC), and then decoded the output with CTC beam search and a KenLM language model to solve some of the deficiencies with my model.

I've also made it cross-platform (web + iOS, with both web screen sharing and the ability to use non smart-glasses). The project is entirely FOSS.

I'm currently working on making the on-device model work cleanly, but unfortunately performance takes a bit of hit on lower-end devices so for now I've opted for hosting the model on Modal.

There's so much room for improvement, but I'm happy with this starting point. Let me know what you think, and check out the YouTube demo!

Show HN: Sign language translation with smart glasses
github.com

Re: Show HN: Sign language translation with smart glasses

#2

Post by loicalleyne »

fingerspelling != sign language

In sign languages facial expressions and body language function as core grammatical tools known as non-manual markers, used in grammar and syntax, distinguishing adverbs and adjectives, indicating tone and intensity. Classifiers, abbreviated as CL, are signed morphemes (particularly, classifier handshapes) that represent a group of nouns or referents. Fingerspelling hand shapes are the building blocks for the hand-related elements of the language (signs done with the hands) whose meaning is dependent on the context, much like a spoken language's word's meaning can be affected by where in a sentence it is placed, tone/pitch, etc.

From a technical viewpoint, if you intend to push beyond where most projects similar to this one have ended up, it might be interesting to consider SL handshapes as analogous to tokens in an LLM, vector encoding them to capture the context and semantic meaning.

From the human relationship POV, making an effort to learn full natural sign language to communicate with your relatives will no doubt make them very happy and enrich their lives and yours.

Re: Show HN: Sign language translation with smart glasses

#3

Post by quizzical8432 »

loicalleyne wrote: post #2

fingerspelling != sign language In sign languages facial expressions and body language function as core grammatical tools known as non-manual markers, used in grammar and syntax, distinguishing adverbs and adjectives, indicating tone and intensity. Classifiers, abbreviated as CL, are signed morphemes (particularly, classifier handshapes) that represent a group of nouns or referents. Fingerspelling hand shapes are the…

This was my first thought, too (after “which sign language?”).

That said, the video does show the model identifying facial and other body features, so it’d be interesting To know if this model _could_ be capable of working with a full sign language (ASL or otherwise).

It also got me thinking about how LLMs might handle signs as tokens, and it seems you did too! Would you just detect unique handshapes and hope the LLM could do the rest? Could you somehow encode orientation/movement/direction/locations and facial expressions? I presume current LLMs don’t have much training input on this kind of language - would we need to train new LLMs to even have a hope of them “understanding” 3-dimensional language?

Re: Show HN: Sign language translation with smart glasses

#10

Post by forestrywat »

I know this is well meaning, but you should know that in the disability community it's very common to have people invent tech for us without fully understanding our needs. (Smart canes, for instance.)

What your demo shows is a model understanding fingerspelling. Fingerspelling is ... a very small part of sign language.

To make the claim you are translating sign language you need to understand facial expressions, hand positioning/motion (help you vs help me, same hand shape, different positions/motions), classifiers, and a ton more stuff.

It's a really beautiful language, and I encourage everyone to learn it, but for your project I'd be very careful to not make a claim about translating sign language.

In the mean time, it sounds like you've got some passion for accessibility work? We're always looking for more skilled engineers to work in the a11y space. It does require some humility and willingness to listen to others needs.

Post reply on HN