Why wouldn't they use 3d renderings as a large part of their training data set? You could have perfectly generated depth outputs generated alongside the image input, and you could adjust things like focal length to all kinds of values that would make this able to understand how shifting items correlate to depth across a variety of focal lengths. To be honest I'm not even sure how they're training them with live footage, how are they even getting the depth maps from the training footage to begin with?
Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
11–20 of 45 posts
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#12You can tell that the authors have a very fast internet connection by the fact that this website weights in at 91.6 Mbyte and takes over a minute to fully load on a 25 Mbit connection.
Jesus. Why couldn't they use embedded video files instead of 30 megabyte gifs?
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#13I can’t wait until techniques like this find their way in to open source photogrammetry pipelines. I came up with a way of training neural nets for robotics using a monocular camera, photogrammetry, and a simulation environment with the captured 3D scene, but the photogrammetry was error prone and computationally intensive even on a beefy cloud server. I’d love for OpenSFM or OpenMVS (check github) to get this kind o…
I personally do not believe that depth generated purely from deep learning can be used as input to photogrammetry anytime soon.
Photogrammetry works exceedingly well because the depth maps that they generate are quite precise and accurate, and mesh reconstruction usually assumes that these points are quite close to ground truth.
Deep learning approaches usually have medium accuracy but low precision, which causes the flickering and smooth surfaces that you see on the person. Even the background has flickering despite being computed through stereo, likely because the camera motion is primarily forward-backward (vs. more accurate side-to-side motion), the baseline is likely small, and the depth isn't globally optimized.
This type of research is super great for applications requiring lower accuracy, typically visual-only applications (e.g. selective blurring, faking stereo on a frame, etc.). But as an input to photogrammetry — probably not anytime soon, until the problems above get resolved.
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#14You can tell that the authors have a very fast internet connection by the fact that this website weights in at 91.6 Mbyte and takes over a minute to fully load on a 25 Mbit connection.
Jesus. Why couldn't they use embedded video files instead of 30 megabyte gifs?
A state-of-the-art deep learning neural net designed by digital video experts within one of the most technology savvy companies in the world...
What do they use to reveal it to the world? GIFs!
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#15Why wouldn't they use 3d renderings as a large part of their training data set? You could have perfectly generated depth outputs generated alongside the image input, and you could adjust things like focal length to all kinds of values that would make this able to understand how shifting items correlate to depth across a variety of focal lengths. To be honest I'm not even sure how they're training them with live foota…
> we make use of an existing source of data for supervision: YouTube videos in which people imitate mannequins by freezing in a wide variety of natural poses, while a hand-held camera tours the scene. Because the entire scene is stationary (only the camera is moving), triangulation-based methods--like multi-view-stereo (MVS)--work, and we can get accurate depth maps for the entire scene including the people in it
I suspect the reason for not using 3D rendering is the desire to cope with the noise and variability of real video.
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#16Can other companies use YouTube database for free say for research in Computer vision?
Sure, but having it fully tagged in a data center across from you helps.
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#17Can other companies use YouTube database for free say for research in Computer vision?
I think it's a gray area, but researchers often just do it. Better to ask for forgiveness than permission I guess. You could never collect datasets like ImageNet if you had to obtain individual permissions.
At least some jurisdictions have research exemptions in their copyright laws, so at least I don't need the copyright owner's permission to use any data for research purposes.
I'd still prefer to use explicitly open datasets because it allows for simpler data sharing and easier reproducibility, however in cases where that's not possible whatever is available will do even if I'm restricted in how I can redistribute that data.
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#18I can’t wait until techniques like this find their way in to open source photogrammetry pipelines. I came up with a way of training neural nets for robotics using a monocular camera, photogrammetry, and a simulation environment with the captured 3D scene, but the photogrammetry was error prone and computationally intensive even on a beefy cloud server. I’d love for OpenSFM or OpenMVS (check github) to get this kind o…
I personally do not believe that depth generated purely from deep learning can be used as input to photogrammetry anytime soon. Photogrammetry works exceedingly well because the depth maps that they generate are quite precise and accurate, and mesh reconstruction usually assumes that these points are quite close to ground truth. Deep learning approaches usually have medium accuracy but low precision, which causes the…
Interesting. Perhaps my idea of this being inserted in to existing algorithms would not work.
However I do ultimately seek a low accuracy “visually approximate” 3D scene that I could use for simulation purposes. I guess I could rephrase my desire as: I’d love to see this kind of approach used to train an end to end deep learning photogrammetry system. I feel like the parallel nature of neural nets as well as their ability to approximate results could result in a much less computationally intensive solution to my photogrammetry desires.
(I want to train my four wheel drive robot to follow forest trails using the training method described in the “world models” research paper, which requires a simulation to work.)
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#19Earlier quoted context omitted.
I personally do not believe that depth generated purely from deep learning can be used as input to photogrammetry anytime soon. Photogrammetry works exceedingly well because the depth maps that they generate are quite precise and accurate, and mesh reconstruction usually assumes that these points are quite close to ground truth. Deep learning approaches usually have medium accuracy but low precision, which causes the…
Interesting. Perhaps my idea of this being inserted in to existing algorithms would not work. However I do ultimately seek a low accuracy “visually approximate” 3D scene that I could use for simulation purposes. I guess I could rephrase my desire as: I’d love to see this kind of approach used to train an end to end deep learning photogrammetry system. I feel like the parallel nature of neural nets as well as their ab…
Some of my friends recently put out http://gibsonenv.stanford.edu/
Full simulation with realistic 3D spaces, enables embodied agents to interact and learn from real-world spaces. Not forest trails, but a real world environment.
If you really want to create a 3D model of forest trails, photogrammetry should be sufficient, because forest scenes are richly-textured.
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#20Why wouldn't they use 3d renderings as a large part of their training data set? You could have perfectly generated depth outputs generated alongside the image input, and you could adjust things like focal length to all kinds of values that would make this able to understand how shifting items correlate to depth across a variety of focal lengths. To be honest I'm not even sure how they're training them with live foota…
The reason why is because there is no training data of the sort you describe out there.
By using MVS-based approaches, they are able to get over the data hurdle by compiling a dataset of your average YouTube video, instead of creating 3D renderings that include dynamic people. Importantly, MVS is really quite accurate, and in many cases can be considered ground truth.
Being able to forgo 3D renderings to use video only is almost certainly a reason why their results are so good.