Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I made a talking emoji using regular emojis and JavaScript

hackernoon.com

Post reply on HN
Print view View original

1–10 of 63 posts

Re: I made a talking emoji using regular emojis and JavaScript

#3

Post by _nalply »

I'm Deaf and read lips.

Thanks for this extremely intriguing idea to use emoji to depict a talking person visually.

You know, some animated films get it right. Sometimes I clearly see utterances like «Thank you!», «Okay», but more often not. Especially older animations don't care and just let the character move the mouth in very simplistic ways: «Babbabbabaa Baababba ba baaa.»

Similarly for that emoji. It is moving the mouth in quite arbitrary ways. It looks like: «Sobbabbabee be seebee babbaa».

To solve this problem we need about twelve emoji for utterances: «ah», «ay», «ee», «oh», «oo», «b», «d», «f», «l», «n», «r» and «s». The «r» emoji must be animated (so we see the trilling tongue). The other sounds are either invisible like «k» or cover several different sounds like «b» also covering «m» and «p».

«Okay» would be rendered by: «oh», «oo», «oo» (standing for «k» wich is invisible) «ay» and «ee».

Edit: Add «th». This is an extremely simple sound to lipread. I remember my delight when I learnt English in my youth and in an movie suddenly realised that an actor said «Thank you».

Re: I made a talking emoji using regular emojis and JavaScript

#4

Post by twiss »

_nalply wrote: post #3

I'm Deaf and read lips. Thanks for this extremely intriguing idea to use emoji to depict a talking person visually. You know, some animated films get it right. Sometimes I clearly see utterances like «Thank you!», «Okay», but more often not. Especially older animations don't care and just let the character move the mouth in very simplistic ways: «Babbabbabaa Baababba ba baaa.» Similarly for that emoji. It is moving t…

Halfway through the article, he shows a picture with mouths for a-z, sh and th: https://cdn-images-1.medium.com/max/1600/0*gwbWfYm-iQIPEXXP%... There are others if you search images for "lip sync animation". It would be interesting to see if the result is understandable if you simply replace the emojis with those images.

Re: I made a talking emoji using regular emojis and JavaScript

#5

Post by _nalply »

twiss wrote: post #4
_nalply wrote: post #3

I'm Deaf and read lips. Thanks for this extremely intriguing idea to use emoji to depict a talking person visually. You know, some animated films get it right. Sometimes I clearly see utterances like «Thank you!», «Okay», but more often not. Especially older animations don't care and just let the character move the mouth in very simplistic ways: «Babbabbabaa Baababba ba baaa.» Similarly for that emoji. It is moving t…

Halfway through the article, he shows a picture with mouths for a-z, sh and th: https://cdn-images-1.medium.com/max/1600/0*gwbWfYm-iQIPEXXP%... There are others if you search images for "lip sync animation". It would be interesting to see if the result is understandable if you simply replace the emojis with those images.

Lipreading is different. For example the «sh» sound is made by slightly pursing the lips but not all speakers do this. In German it gives a very clear visual picture.

Re: I made a talking emoji using regular emojis and JavaScript

#6

Post by k__ »

_nalply wrote: post #3

I'm Deaf and read lips. Thanks for this extremely intriguing idea to use emoji to depict a talking person visually. You know, some animated films get it right. Sometimes I clearly see utterances like «Thank you!», «Okay», but more often not. Especially older animations don't care and just let the character move the mouth in very simplistic ways: «Babbabbabaa Baababba ba baaa.» Similarly for that emoji. It is moving t…

I once read lipreading doesn't work very reliable, even for very skilled lipreaders.

Maybe that's why most animation studios don't bother.

Re: I made a talking emoji using regular emojis and JavaScript

#7

Post by exogen »

One improvement might be to use a pronunciation dictionary and map the emojis to actual sounds rather than characters. Having recently needed such a thing, I found that there are two widely available datasets: CMUdict and Moby Pronunciator:

http://onlinebooks.library.upenn.edu/webbin/gutbook/lookup?n...

https://github.com/cmusphinx/cmudict

The problem is that they'd be pretty hefty payloads to load on the client, so you'd want to do the text -> phone mapping elsewhere. Then use your character mapping as backup for words that aren't in the dictionaries.

Re: I made a talking emoji using regular emojis and JavaScript

#8

Post by evanb »

_nalply wrote: post #3

I'm Deaf and read lips. Thanks for this extremely intriguing idea to use emoji to depict a talking person visually. You know, some animated films get it right. Sometimes I clearly see utterances like «Thank you!», «Okay», but more often not. Especially older animations don't care and just let the character move the mouth in very simplistic ways: «Babbabbabaa Baababba ba baaa.» Similarly for that emoji. It is moving t…

You could actually propose a whole set of Unicode characters that are making realistic-for-lip-reading movements.
Post reply on HN