TLDR They use a neural net to process an image to find a similar known 3D model to the object in the image as well as parameters for how to deform that model to be like the object in the image, and also then perform a linear combination of some related meshes (this is a follow up to their prior work "Compact Model Representation for 3D Reconstruction"). Pretty exciting work, but still far from being able to robustly do Image2Mesh. I happen to have worked on something quite similar, if any of you are curious: https://deformnet-site.github.io/DeformNet-website/
Image2Mesh: Single Image 3D Reconstruction
11–17 of 17 posts
Re: Image2Mesh: Single Image 3D Reconstruction
#12Surprised this is getting any notice at all. I work at a facial recognition firm pioneering this technique 20 years ago. (www.CyberExtruder.com) The TLDR description by andreyk pretty much sums the process, as it originated long ago. It is very similar now, just with a team of very smart people iterating over the pipeline for so long has made it sophisticated to the point the TLDR description could be argued as too light.
I even made a 3D avatar API & service, now closed, from the technology 10 years ago. The twitter site is still around, if you want to see the quality of reconstruction possible: https://twitter.com/3DAvatarStore/media
Re: Image2Mesh: Single Image 3D Reconstruction
#13Arxiv Vanity: https://www.arxiv-vanity.com/papers/1711.10669/
Re: Image2Mesh: Single Image 3D Reconstruction
#14TLDR They use a neural net to process an image to find a similar known 3D model to the object in the image as well as parameters for how to deform that model to be like the object in the image, and also then perform a linear combination of some related meshes (this is a follow up to their prior work "Compact Model Representation for 3D Reconstruction"). Pretty exciting work, but still far from being able to robustly…
Interesting, thanks for the link. I tried a few published methods for doing this a year or so and I found they were quite slow. As in, you'd rather just use a stereo camera to quickly generate a decent pointcloud than use a single view plus a network like these. What did you find your performance (ms/frame) was like?
Re: Image2Mesh: Single Image 3D Reconstruction
#15TLDR They use a neural net to process an image to find a similar known 3D model to the object in the image as well as parameters for how to deform that model to be like the object in the image, and also then perform a linear combination of some related meshes (this is a follow up to their prior work "Compact Model Representation for 3D Reconstruction"). Pretty exciting work, but still far from being able to robustly…
Interesting, thanks for the link. I tried a few published methods for doing this a year or so and I found they were quite slow. As in, you'd rather just use a stereo camera to quickly generate a decent pointcloud than use a single view plus a network like these. What did you find your performance (ms/frame) was like?
With separate dedicated identifiers and matching reconstruction pipelines, while only caring about a limited number of things, as in our specializing in faces, we can reconstruct 25 to 30 independent people, their heads and shoulder tops only, in real time on a 3.4 GHz i7. Note the reconstruction is accurate only as far as it matters for FR, as in we tend to ignore hair. So the models are bald, but they have the frame to frame facial expressions for everyone. Here's a gif of a degraded lighting test at 70 fps with one head doing a 3D reconstruction independently performed each frame: https://www.dropbox.com/s/0jtee4ofh2vdeg0/BlakeHeadMeshes.gi... I'm just visualizing the face, but the entire head and shoulder tops are reconstructed.
Re: Image2Mesh: Single Image 3D Reconstruction
#16Earlier quoted context omitted.
Interesting, thanks for the link. I tried a few published methods for doing this a year or so and I found they were quite slow. As in, you'd rather just use a stereo camera to quickly generate a decent pointcloud than use a single view plus a network like these. What did you find your performance (ms/frame) was like?
With separate dedicated identifiers and matching reconstruction pipelines, while only caring about a limited number of things, as in our specializing in faces, we can reconstruct 25 to 30 independent people, their heads and shoulder tops only, in real time on a 3.4 GHz i7. Note the reconstruction is accurate only as far as it matters for FR, as in we tend to ignore hair. So the models are bald, but they have the fram…
Ah this is cool too! I assume no neural networks are involved here?
EDIT: why the changing colors in that gif?
Re: Image2Mesh: Single Image 3D Reconstruction
#17Earlier quoted context omitted.
With separate dedicated identifiers and matching reconstruction pipelines, while only caring about a limited number of things, as in our specializing in faces, we can reconstruct 25 to 30 independent people, their heads and shoulder tops only, in real time on a 3.4 GHz i7. Note the reconstruction is accurate only as far as it matters for FR, as in we tend to ignore hair. So the models are bald, but they have the fram…
Ah this is cool too! I assume no neural networks are involved here? EDIT: why the changing colors in that gif?
The changing colors and grain are part of a degraded video while tracking the face test. The face detection and portions of the reconstruction are NNs.