DeepMind seems to be conducting research in a similar direction. Any insight on how are the two projects related or different?
https://deepmind.com/blog/neural-scene-representation-and-re...
Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
21–30 of 45 posts
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#22The predictions seem to have rapid flickering, which means the model is saying lots of items are moving back and forth extremely quickly. Since this seems common in video analysis (rapid changes per frame) is it that smoothing or taking into consideration multiple frames is slow? Or does it cause more issues than it solves?
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#23Always worth looking at a point cloud versus a disparity map.
Grayscale disparity/depth maps are somewhat misleading - the large regions of constant intensity suggest that the algorithm is good at segmenting areas of constant depth. However, the flickering in the map suggests that if you actually tried to plot this in 3D, it'd be pretty noisy. Not to disparage the result, but 2D depth/disparity maps tend to look better than what they represent.
You can see this in the synthetic camera wiggle video, focus on the actor's hands, for example.
You can also see this effect in the Stereolabs Zed promo video.
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#24I can’t wait until techniques like this find their way in to open source photogrammetry pipelines. I came up with a way of training neural nets for robotics using a monocular camera, photogrammetry, and a simulation environment with the captured 3D scene, but the photogrammetry was error prone and computationally intensive even on a beefy cloud server. I’d love for OpenSFM or OpenMVS (check github) to get this kind o…
I personally do not believe that depth generated purely from deep learning can be used as input to photogrammetry anytime soon. Photogrammetry works exceedingly well because the depth maps that they generate are quite precise and accurate, and mesh reconstruction usually assumes that these points are quite close to ground truth. Deep learning approaches usually have medium accuracy but low precision, which causes the…
> I personally do not believe that depth generated purely from deep learning can be used as input to photogrammetry anytime soon.
6d.ai uses depthnets in its mobile photogrammetry pipeline. demo: https://twitter.com/mattmiesnieks/status/1106722396889702406
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#25You can tell that the authors have a very fast internet connection by the fact that this website weights in at 91.6 Mbyte and takes over a minute to fully load on a 25 Mbit connection.
From guidelines -
Be kind. Don't be snarky. Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#26Seems related to the Tesla video-based depth perception work?
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#27Is this how Tesla does their magic? Brilliant!
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#28Earlier quoted context omitted.
I personally do not believe that depth generated purely from deep learning can be used as input to photogrammetry anytime soon. Photogrammetry works exceedingly well because the depth maps that they generate are quite precise and accurate, and mesh reconstruction usually assumes that these points are quite close to ground truth. Deep learning approaches usually have medium accuracy but low precision, which causes the…
> I personally do not believe that depth generated purely from deep learning can be used as input to photogrammetry anytime soon. 6d.ai uses depthnets in its mobile photogrammetry pipeline. demo: https://twitter.com/mattmiesnieks/status/1106722396889702406
How do you know that 6D AI uses deep learning to predict depth maps?
I'm very familiar with their work (they're doing a great job), but the demo video you linked appears to be a photogrammetric-based approach. You can tell because highly-textured surfaces are readily mapped, but low-texture regions remain unmapped, despite high coverage by the camera.
Maybe they use learned features for things like persistent AR, but I'm quite certain that they do not use deep learning to predict depth maps ab initio.
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#29I'm not sure if this is due to different mapping into greyscale or is their method completely killing far distance details?
Compared to "Chen et al" which is a bit flickery in the foreground, but full of stable background details, their result is almost completely black 3m in.
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#30Earlier quoted context omitted.
Interesting. Perhaps my idea of this being inserted in to existing algorithms would not work. However I do ultimately seek a low accuracy “visually approximate” 3D scene that I could use for simulation purposes. I guess I could rephrase my desire as: I’d love to see this kind of approach used to train an end to end deep learning photogrammetry system. I feel like the parallel nature of neural nets as well as their ab…
Some of my friends recently put out http://gibsonenv.stanford.edu/ Full simulation with realistic 3D spaces, enables embodied agents to interact and learn from real-world spaces. Not forest trails, but a real world environment. If you really want to create a 3D model of forest trails, photogrammetry should be sufficient, because forest scenes are richly-textured.
Yes I did come across Gibsonenv and it looks great for indoor scenes.
As far as photogrammetry of forest trails, I found it to be very computationally intensive (taking a GCE 32 core instance 30+ hours using 90+GB of ram to compute a scene, only with errors that made it unusable). It felt very heavy handed and given all the great work I've seen in scene understanding using neural nets, it seems like deep learning would be a promising approach here. Maybe there is commercial photogrammetry software that has better pipelines, but I want to be able to compute my scenes on linux and use hundreds of images.
I did my computation with OpenSFM and OpenMVS. Both wonderful projects for being free and open source. I did get a lot of great results. But I am convinced a simpler way is possible with deep learning.