Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HybridNeRF: Efficient Neural Rendering

haithemturki.com

Post reply on HN
Print view View original

21–30 of 53 posts

Re: HybridNeRF: Efficient Neural Rendering

#21

Post by naet »

ttul wrote: post #3

Does anyone else look forward to a game that lets you transform your house or neighbor into a playable level with destructible objects? How far are we from recognizing the “car” and making it drivable, or the “tree” and making it choppable?

My parents had a floor plan of our house drawn up for some reason, and when I was in late middle school I found it and modeled the house in the hammer editor so my friends and I could play Counter Strike source in there.

It wasn't very well done but I figured out how to make the basic walls and building, add stairs, add some windows, grab some pre existing props like simple couches beds and a TV, and it was pretty recognizable. After adding a couple ladders to the outside so you could climb in the windows or on the roof the map was super fun just as a map, and doubly so since I could do things like hide in my own bedroom closet and recognize the rooms.

Took some work since I didn't know how to do anything but totally worth it. I feel like there has to be a much more accessible level editor in some game out there today, not sure what it would be though.

I thought my school had great architecture for another map but someone rightfully convinced me that would be a very bad idea to add to a shooting game. So I never made any others besides the house.

Re: HybridNeRF: Efficient Neural Rendering

#22

Post by refibrillator »

turkihaithem wrote: post #17

One of the paper authors here - happy to answer any questions about the work or chat about neural rendering in general!

Congrats on the paper! Any chance the code will be released?

Also I’d be curious to hear, what are you excited about in terms of future research ideas?

Personally I’m excited by the trend of eliminating the need for traditional SfM preprocessing (sparse point clouds via colmap, camera pose estimation, etc).

Re: HybridNeRF: Efficient Neural Rendering

#23

Post by aerodog »

turkihaithem wrote: post #17

One of the paper authors here - happy to answer any questions about the work or chat about neural rendering in general!

Thanks for your work!

From my experience, NERF works great, but depends on highly accurate camera location information. Unless the VR device has this baked in, one must run a Colmap-style or SFM-style process to generate those camera extrinsics. Is there anything special HybridNeRF does around this?

Re: HybridNeRF: Efficient Neural Rendering

#24

Post by modeless »

aerodog wrote: post #23
turkihaithem wrote: post #17

One of the paper authors here - happy to answer any questions about the work or chat about neural rendering in general!

Thanks for your work! From my experience, NERF works great, but depends on highly accurate camera location information. Unless the VR device has this baked in, one must run a Colmap-style or SFM-style process to generate those camera extrinsics. Is there anything special HybridNeRF does around this?

The method in this paper relies on precomputed camera poses as input, but there have been tons of papers published on the topic of eliminating this requirement. Here are a few: https://dust3r.europe.naverlabs.com/ https://arxiv.org/abs/2102.07064 https://arxiv.org/abs/2312.08760v1 https://x.com/_akhaliq/status/1734803566802407901

Re: HybridNeRF: Efficient Neural Rendering

#25

Post by TeMPOraL »

smokel wrote: post #8
TeMPOraL wrote: post #5

Earlier quoted context omitted.

I dreamed of that since being a kid, so for nearly three decades now. It's been entirely possible even then - it was just a matter of using enough elbow grease. The problem is, the world is full of shiny happy people ready to call you a terrorist, assert their architectural copyright, or bring in the "creepiness factor", to shut down anyone who tries this.

Having destructible objects is in no way possible on contemporary hardware, unless you simplify the physics to the extreme. Perhaps I'm misunderstanding your statement? Recognising objects for what they are has only recently become somewhat possible. Separating them in a 3D scan is still pretty much impossible.

My statement applies even without the destructive environment part - even though that was already mainstream 23 years ago! See Red Faction. No, just making a real-life place a detailed part of a video game is going to cause the pushback I mentioned.

Re: HybridNeRF: Efficient Neural Rendering

#26

Post by TeMPOraL »

idiotsecant wrote: post #10
smokel wrote: post #8

Earlier quoted context omitted.

Having destructible objects is in no way possible on contemporary hardware, unless you simplify the physics to the extreme. Perhaps I'm misunderstanding your statement? Recognising objects for what they are has only recently become somewhat possible. Separating them in a 3D scan is still pretty much impossible.

Destructible environments have been a thing for like....a decade or so? There's plenty of tricks to make it realistic enough to be fun without simulating every molecule.

Two decades - see Red Faction, which is a first-person shooter from 2001.

Re: HybridNeRF: Efficient Neural Rendering

#27

Post by totalview »

modeless wrote: post #20
totalview wrote: post #12

Earlier quoted context omitted.

I work in the rendering and gaming industry and also run a 3D scanning company. I have similarly wished for this capability, especially the destructability part. What you speak of is still pretty far off for several reasons: -No Collision/poor collision on NERFs and GS: to have a proper interactive world, you usually need accurate character collision so that your character or vehicle can move along the floor/ground (…

I think collision detection is solvable. And the scanning process should be no harder than 3D modeling to the same quality level. Probably much easier, honestly. Modeling is labor intensive. I'm not sure why you say "there’s no scanner available that provides both good 3-D information and good photo realistic textures" because these new techniques don't use "scanners", all you need is regular cameras. The 3D informat…

You are correct; most of these new techniques are using a camera. In my line of work I consider a camera sensor a scanner of sorts, as we do a lot of photogrammetry and “scan” with a 45MP full frame. The inferred 3D from cameras is pretty bad when it comes to accuracy, especially from dimly lit areas or where you dip into a closet or closed space that doesn’t have a good structural tie back to the main space you are trying to recreate in 3D. Laser scanners are far preferable to tie your photo pose estimation to, and most serious reality capture for video games is done with both a camera a and $40,000+ LiDAR Scanner. Have you ever tried to scan every corner of a house with only a traditional DSLR or point and shoot camera? I have and the results are pretty bad from a 3D standpoint without a ton of post process.

The collision detection problem is related heavily to having clean 3D as mentioned above. My company is doing development on computing collision on reality capture right now in a clean way and I would be interested in any thoughts you have. We are chunking collision on the dataset at a fixed distance from the player character (can’t go too fast in a vehicle or it will outpace the collision and fall thru the floor) and have a tunable LOD that influences collision resolution.

Re: HybridNeRF: Efficient Neural Rendering

#28

Post by turkihaithem »

refibrillator wrote: post #22
turkihaithem wrote: post #17

One of the paper authors here - happy to answer any questions about the work or chat about neural rendering in general!

Congrats on the paper! Any chance the code will be released? Also I’d be curious to hear, what are you excited about in terms of future research ideas? Personally I’m excited by the trend of eliminating the need for traditional SfM preprocessing (sparse point clouds via colmap, camera pose estimation, etc).

Thank you! The code is unlikely to be released (it's built upon Meta-internal codebases that I no longer have access to post-internship), at least not in the form that we specifically used at submission time. The last time I caught up with the team someone was expressing interest in releasing some broadly useful rendering code, but I really can't speak on their behalf so no guarantees.

IMHO it's a really exciting time to be in the neural rendering / 3D vision space - the field is moving quickly and there's interesting work across all dimensions. My personal interests lean towards large-scale 3D reconstruction, and to that effect eliminating the need for traditional SfM/COLMAP preprocessing would be great. There's a lot of relevant recent work (https://dust3r.europe.naverlabs.com/, https://cameronosmith.github.io/flowmap/, https://vggsfm.github.io/, etc), but scaling these methods beyond several dozen images remains a challenge. I’m also really excited about using learned priors that can improve NeRF quality in underobserved regions (https://reconfusion.github.io). IMO using these priors will be super important to enabling dynamic 4D reconstruction (since it’s otherwise unfeasible to directly observe every space-time point in a scene). Finally, making NeRF environments more interactive (as other posts have described) would unlock many use cases especially in simulation (ie: for autonomous driving). This is kind of tricky for implicit representations (like the original NeRF and this work), but there have been some really cool papers in the 3D Gaussian space (https://xpandora.github.io/PhysGaussian/) that are exciting.

Re: HybridNeRF: Efficient Neural Rendering

#29

Post by turkihaithem »

modeless wrote: post #24
aerodog wrote: post #23

Earlier quoted context omitted.

Thanks for your work! From my experience, NERF works great, but depends on highly accurate camera location information. Unless the VR device has this baked in, one must run a Colmap-style or SFM-style process to generate those camera extrinsics. Is there anything special HybridNeRF does around this?

The method in this paper relies on precomputed camera poses as input, but there have been tons of papers published on the topic of eliminating this requirement. Here are a few: https://dust3r.europe.naverlabs.com/ https://arxiv.org/abs/2102.07064 https://arxiv.org/abs/2312.08760v1 https://x.com/_akhaliq/status/1734803566802407901

Your understanding is correct!

Re: HybridNeRF: Efficient Neural Rendering

#30

Post by sneak »

totalview wrote: post #27
modeless wrote: post #20

Earlier quoted context omitted.

I think collision detection is solvable. And the scanning process should be no harder than 3D modeling to the same quality level. Probably much easier, honestly. Modeling is labor intensive. I'm not sure why you say "there’s no scanner available that provides both good 3-D information and good photo realistic textures" because these new techniques don't use "scanners", all you need is regular cameras. The 3D informat…

You are correct; most of these new techniques are using a camera. In my line of work I consider a camera sensor a scanner of sorts, as we do a lot of photogrammetry and “scan” with a 45MP full frame. The inferred 3D from cameras is pretty bad when it comes to accuracy, especially from dimly lit areas or where you dip into a closet or closed space that doesn’t have a good structural tie back to the main space you are…

Both my iPhone and my Apple Vision Pro both have lidar scanners, fwiw.

Frankly I’m surprised that I can’t easily make crude 3D models of spaces with a simple app presently. It seems well within the capabilities of the hardware and software.

Post reply on HN