Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

High-Speed Face-Tracking for Dynamic Facial Projection Mapping

vision.ict.e.titech.ac.jp

Post reply on HN
Print view View original

11–20 of 24 posts

Re: High-Speed Face-Tracking for Dynamic Facial Projection Mapping

#11

Post by vessenes »

Ooooh, this is so cool. I didn’t realize I was seeing projected ‘makeup’ on faces until they showed some lower latency prior work. Once I saw it, I could see occasional misalignment. But, this is just an awesome and fun thing. I can imagine it being just amazing for live theater costuming and story telling.

That said, it looks like the tech currently works at 1-2m range. I’d guess this is because that’s an easy range for a projector to work at and provide high res, in which case there’s no reason you couldn’t get a zoom setup going.

Perhaps some enterprising director of the next Disney theater production will put the money in to get this working reliably on stage.

Re: High-Speed Face-Tracking for Dynamic Facial Projection Mapping

#12

Post by tsumnia »

bhouston wrote: post #2

Neat stuff. I wonder if one could use a neural network to try to do prediction as well. Human movements are super stereotyped in the face and you could likely predict with decent accuracy the next frame given the previous frames.

Check out a lot of DisneyResearchHub's videos on YouTube. They are regularly posting research on object/face tracking and projections.

That said as you mentioned faces are pretty stereotyped. We "solved" face mapping two decades ago with Tim Cootes and Paul Ekman's work. We're able to quickly map rough estimates using traditional haar cascade classifiers and Viola-Jones with AdaBoost. Neural Networks may help, but we have other solutions that also handle the problem with "relative" ease (ignoring lighting, occlusion, etc.)

Re: High-Speed Face-Tracking for Dynamic Facial Projection Mapping

#13

Post by croes »

grayhatter wrote: post #8
croes wrote: post #7

Earlier quoted context omitted.

„Leaked“ videos of political enemies saying abysmal things The Protocols of the Elders of Zion on steroids.

Uh... this is a white paper about projecting an image onto a moving face. Not super imposing, but projecting... with a projector... using light... so what exactly are you worried about with this tech? Or is it just that you're afraid of face tracking in general? And it's not really about this topic specifically?

It’s about high speed face tracking.

Can you imagine other use cases than projection?

Re: High-Speed Face-Tracking for Dynamic Facial Projection Mapping

#14

Post by sporkl »

_flux wrote: post #3

Pretty cool. I've had this idea of converting face data into MIDI CC data to use with DAWs or synths, this could be a useful algorithm for it. Is the source code available?

Not this project, I've seen this something like this done with Google's MediaPipe models to get face tracking data into Max/MSP.

https://cycling74.com/forums/n4m-facemesh-handpose-google-me...

Re: High-Speed Face-Tracking for Dynamic Facial Projection Mapping

#18

Post by grayhatter »

croes wrote: post #13
grayhatter wrote: post #8

Earlier quoted context omitted.

Uh... this is a white paper about projecting an image onto a moving face. Not super imposing, but projecting... with a projector... using light... so what exactly are you worried about with this tech? Or is it just that you're afraid of face tracking in general? And it's not really about this topic specifically?

It’s about high speed face tracking. Can you imagine other use cases than projection?

> Can you imagine other use cases than projection?

yes

vtubers

human interface device for quadriplegics

automated camera tracking/zoom for video calls

emotion recognition support software for people with difficulty reading or understanding emotions (ASD)

truly smart, smart home that can track and respond to gaze, e.g. "can you turn off that light" and it will turn off the light they're looking at

sleep detection for smart locks

liveness detection for identify verification systems

vr tracking for interactions that both escape the uncanny valley, and feel more real time

need more?

Re: High-Speed Face-Tracking for Dynamic Facial Projection Mapping

#19

Post by echelon »

m3kw9 wrote: post #17

Is good but when you need that extra lens shift camera, it goes from mass market to super niche.

It seems like they've done two things here - the physical optical apparatus setup and the algorithm.

I'm wondering why the physical alignment is so important. Are camera distortion models and mapping, view projection, etc. just too slow or low quality to run?

I suppose I'll have to take a look at their paper later.

Re: High-Speed Face-Tracking for Dynamic Facial Projection Mapping

#20

Post by croes »

grayhatter wrote: post #18
croes wrote: post #13

Earlier quoted context omitted.

It’s about high speed face tracking. Can you imagine other use cases than projection?

> Can you imagine other use cases than projection? yes vtubers human interface device for quadriplegics automated camera tracking/zoom for video calls emotion recognition support software for people with difficulty reading or understanding emotions (ASD) truly smart, smart home that can track and respond to gaze, e.g. "can you turn off that light" and it will turn off the light they're looking at sleep detection for…

Yes, list the malicious ones too.
Post reply on HN