Could this technology one day become so good as to eliminate the need for lidar for self-driving cars? Or will lidar be so inexpensive by that point that there will be no need to eliminate it?
Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
31–40 of 45 posts
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#32Seems related to the Tesla video-based depth perception work?
That was my thought too, particularly because Tesla described extracting distance and velocity for moving objects by processing video frames two at a time in the upcoming hardwire.
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#33I can’t wait until techniques like this find their way in to open source photogrammetry pipelines. I came up with a way of training neural nets for robotics using a monocular camera, photogrammetry, and a simulation environment with the captured 3D scene, but the photogrammetry was error prone and computationally intensive even on a beefy cloud server. I’d love for OpenSFM or OpenMVS (check github) to get this kind o…
I personally do not believe that depth generated purely from deep learning can be used as input to photogrammetry anytime soon. Photogrammetry works exceedingly well because the depth maps that they generate are quite precise and accurate, and mesh reconstruction usually assumes that these points are quite close to ground truth. Deep learning approaches usually have medium accuracy but low precision, which causes the…
Couldn't depth estimates and camera movement estimates based on NNs still substantially speed up the feature matching? Even if it isn't accurate, it seems like it should be able to reject like 90% of the candidate matches with high reliability.
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#34You can tell that the authors have a very fast internet connection by the fact that this website weights in at 91.6 Mbyte and takes over a minute to fully load on a 25 Mbit connection.
Jesus. Why couldn't they use embedded video files instead of 30 megabyte gifs?
This is an image processing project. It's tradition to demo it to the world exclusively using grainy low resolution images. In this case it needs animation too, so gifs are the obvious choice!
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#35At this point I feel like I'm psychic. Every single time I see an image processing project posted on here I think to myself "I bet the only examples are tiny low resolution thumbnails" and every. single. time. I'm proven right. Whyyyyyyy?
To be fair, this particular application doesn't really need more to show it's improvement over other approaches, but still.
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#36Earlier quoted context omitted.
Some of my friends recently put out http://gibsonenv.stanford.edu/ Full simulation with realistic 3D spaces, enables embodied agents to interact and learn from real-world spaces. Not forest trails, but a real world environment. If you really want to create a 3D model of forest trails, photogrammetry should be sufficient, because forest scenes are richly-textured.
Yes I did come across Gibsonenv and it looks great for indoor scenes. As far as photogrammetry of forest trails, I found it to be very computationally intensive (taking a GCE 32 core instance 30+ hours using 90+GB of ram to compute a scene, only with errors that made it unusable). It felt very heavy handed and given all the great work I've seen in scene understanding using neural nets, it seems like deep learning wou…
OpenSFM is quite out of date, so it's quite inefficient and rather inaccurate (e.g. exhaustive matching is O(n^2), and there are a lot of smarter ways that are closer to O(n))
Also, one of the main steps of mesh reconstruction is depth map generation. It typically takes anywhere from 30-75% of compute time for dense reconstruction, IF it's parallelized thru GPU. If you're using the CPU only to calculate depth maps, you're probably slowing yourself down by an order of magnitude.
If you have a GPU, and use a better SFM-MVS solution, then you can quite easily reconstruct datasets of 1k-10k images within 24 hours.
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#37Is it as good as LIDAR
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#38Seems related to the Tesla video-based depth perception work?
Tesla approach works chiefly on video scenes with static objects like parked cars.
They train a DepthCNN to infer depth from monocular images (lidar or stereo for supervision) and make sure it's temporally consistent by adjusting with pixel transformations from the previous and next frame using a PoseCNN. https://arxiv.org/abs/1704.07813
The guys at Google use Optical flow (only previous frame) to make sure their model trained on static object video sequences works when the scene is dynamic using a mask for a specific class an object (humans here). They do have to make sure nothing but humans are dynamic in the scene.
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#39Is it as good as LIDAR
I do not think so
Re: Moving Camera, Moving People: A Deep Learning Approach to Depth Prediction
#40The best part of this paper is that they used mannequin challenge videos as their training dataset. That's super clever.
I have seen people suggest that the "10 year challenge" was created to build an age-related training dataset. While the mannequin challenge was probably just spontaneous, I wonder if we will see an increasing number of viral challenges in the future that center around the creation of structured information.