Live data from Hacker News

I made a talking emoji using regular emojis and JavaScript

hackernoon.com

21–30 of 63 posts

Re: I made a talking emoji using regular emojis and JavaScript

#21
> an emoji, in reality, looks like this: “\xF0\x9F\x98\x81”

… well, that’s one encoding of the UTF-8 encoding of the Unicode code point.

> This is because setInterval(_=>{ },99) executes the function every 99ms

This is categorically wrong. You can’t trust setInterval to heed your request precisely at all: it’s a request that browsers take as a minimum only. Most browsers will call your function after the number of milliseconds you requested plus up to 16ms more, but some might wait even more than that (I think Safari in power saving mode doubles its tick time, to operate at 30fps; and background tabs typically won’t fire more than once a second these days).

Try running this snippet in your dev tools; it logs the number of milliseconds between calls:

  t=Date.now();setInterval(()=>console.log(-t+(t=Date.now())), 99)
On Firefox I’m getting mostly 108–111, with the odd one a little higher, and a couple of 99s after running it for a few minutes.

(Some trivia on similar techniques for this measurement: `+new Date` is rather slow, `Date.now()` is about 7× faster in at least Firefox and Chrome, and `performance.now()` gets you microsecond precision (it returns a floating-point number in milliseconds, tied to an unspecified epoch instead of real-world time), and is a little slower than Date.now().)

Re: I made a talking emoji using regular emojis and JavaScript

#23
post #3

I'm Deaf and read lips. Thanks for this extremely intriguing idea to use emoji to depict a talking person visually. You know, some animated films get it right. Sometimes I clearly see utterances like «Thank you!», «Okay», but more often not. Especially older animations don't care and just let the character move the mouth in very simplistic ways: «Babbabbabaa Baababba ba baaa.» Similarly for that emoji. It is moving t…

I always thought an interesting project would be to use facial recognition to detect key mouth shapes to make better gifs of people talking.

Only a few sounds use lip shapes. Most sounds are based on tongue positioning. Compare ‘m’ & ‘n’ to… not making any sound with your lips together or slightly parted.

The only sounds I would be very confident at recovering sounds from images for ‘th’, ‘w’, ‘o’, & indistinguishable voiced/unvoiced pairs like ‘b’/‘p’ or ‘v’/‘f’. Basically, the sounds a ventriloquist has to cheat.

Re: I made a talking emoji using regular emojis and JavaScript

#24
post #8

Earlier quoted context omitted.

You could actually propose a whole set of Unicode characters that are making realistic-for-lip-reading movements.

That would be a rabbit hole though, if you consider that Unicode isn't English or even Indo-European centric. *(just another rabbit hole actually, Unicode got plenty of them already)

> Unicode isn't English or even Indo-European centric

It definitely is! The Latin alphabet gets a special place at the start of the range, in order to be compatible with ASCII, an English-language character set.

Re: I made a talking emoji using regular emojis and JavaScript

#25

> an emoji, in reality, looks like this: “\xF0\x9F\x98\x81” … well, that’s one encoding of the UTF-8 encoding of the Unicode code point. > This is because setInterval(_=>{ },99) executes the function every 99ms This is categorically wrong. You can’t trust setInterval to heed your request precisely at all: it’s a request that browsers take as a minimum only. Most browsers will call your function after the number of mi…

Hi Chris, thanks for the corrections, I know you cannot trust the browser, but for the sake of simplicity, and because the purpose of the intro was describing the code I found on the internet, I opt for a simple "this is the trick", doesn't always work, but this is the idea

Re: I made a talking emoji using regular emojis and JavaScript

#27
post #5

Earlier quoted context omitted.

Lipreading is different. For example the «sh» sound is made by slightly pursing the lips but not all speakers do this. In German it gives a very clear visual picture.

I would only purse my lips if I were performing a shush. In words like ‘hash’ I’d just close my mouth slightly & leave lips parted and wide from the ‘ha’ sound. I agree that the mouth shapes are garbage in this demo. A shame because that is the most interesting & important part of animating speech.

This demo is done with my nearly 0 knowledge in lips reading, the effect I was searching was to make it more realistic, but with the limited amount of usable emojis I cannot have it very realistic, just better than random. At the end of the article I propose some gotchas to improve it: use better images, use a better map.

I'm happy that this article has triggered some people curiosity and can maybe evolve in something more advanced

Re: I made a talking emoji using regular emojis and JavaScript

#29
post #6
post #3

I'm Deaf and read lips. Thanks for this extremely intriguing idea to use emoji to depict a talking person visually. You know, some animated films get it right. Sometimes I clearly see utterances like «Thank you!», «Okay», but more often not. Especially older animations don't care and just let the character move the mouth in very simplistic ways: «Babbabbabaa Baababba ba baaa.» Similarly for that emoji. It is moving t…

I once read lipreading doesn't work very reliable, even for very skilled lipreaders. Maybe that's why most animation studios don't bother.

Some do bother, because even some hearing people can tell the mouth movements are off. This adds to the authenticity of a movie.

Re: I made a talking emoji using regular emojis and JavaScript

#30
post #5

Earlier quoted context omitted.

Lipreading is different. For example the «sh» sound is made by slightly pursing the lips but not all speakers do this. In German it gives a very clear visual picture.

I would only purse my lips if I were performing a shush. In words like ‘hash’ I’d just close my mouth slightly & leave lips parted and wide from the ‘ha’ sound. I agree that the mouth shapes are garbage in this demo. A shame because that is the most interesting & important part of animating speech.

That's why I am bad at lipreading in English. People have mouth shapes all over the place. I'm sure I could learn lipreading a friend I meet often well, however this doesn't help lipreading a stranger. When traveling in the US I always have a notebook ready.
Post reply on HN