How is this ML? They use a CNN for foreground segmentation, a minor step in their pipeline. But the major contribution seems to be putting the silhouettes in a common reference frame. I sincerely hope sciencemag isn’t putting ML in the title purely to jump on the bandwagon.
ML turns video of a 360° turn into 3D model of a person
11–20 of 33 posts
Re: ML turns video of a 360° turn into 3D model of a person
#12This is awesome. I wish someone will implement this as a piece of open source software. Imagine the potential!
Source code seems to be available :)
Re: ML turns video of a 360° turn into 3D model of a person
#13Re: ML turns video of a 360° turn into 3D model of a person
#14It seems determined to put visible toes on everybody, no matter that they're wearing socks. Is this a bug or a feature?
I'm going to guess they start with a generic human model that includes all limbs and extremities and then the "machine learning" process attempts to fit that model to the silhouettes extracted from the video.
Re: ML turns video of a 360° turn into 3D model of a person
#15Link to the paper: https://arxiv.org/abs/1803.04758
And to the video: https://www.youtube.com/watch?v=nPOawky2eNk
Re: ML turns video of a 360° turn into 3D model of a person
#16Oh how I hate marketing speech.
First of all, the title should include "video of a predefined 360° turn".
And then they say something along the lines of "average accuracy of about 5mm" for joining the constructed modeled joints to their model, while you see the body wobbling around happily.
This is an impressive demo, but gah!
Re: ML turns video of a 360° turn into 3D model of a person
#17As an artist, my first thought is I wonder what happens if you try giving this a series of drawings .
you'd probably need a lot of drawings, I wonder what's the sampling rate the thing uses
it's a cool idea though :)
Re: ML turns video of a 360° turn into 3D model of a person
#18It seems determined to put visible toes on everybody, no matter that they're wearing socks. Is this a bug or a feature?
I'm going to guess they start with a generic human model that includes all limbs and extremities and then the "machine learning" process attempts to fit that model to the silhouettes extracted from the video.
Which implies that the technique uses domain knowledge of people to make assumptions about their morphology.
Re: ML turns video of a 360° turn into 3D model of a person
#19I wonder if will see a future soon where a director can fully edit the positions and physical actions of the actors at post production.
basically, the whole scenes will be transferred to believable 3d models seemlessly, and you can reanimate parts of everything. I feel like that's doing to happen for sure, for big Hollywood productions at least (like the Marvel stuff)
Re: ML turns video of a 360° turn into 3D model of a person
#20Structure from motion is an existing technique. What is the contribution of ML in this case (it seems like joint positioning maybe?)? https://en.m.wikipedia.org/wiki/Structure_from_motion
99%¹ of computer vision problems are 80% solved. The problem is, you need 95+% solution to be practically useful.
Binocular stereo vision has just approached general applicability, and SfM is mostly used in very constrained environments (traffic analysis) or with large computational resources with manual correction (offline 3D mapping from aerial data).
¹ Numbers are metaphoric only, based on experience in scientific and industrial CV.