Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HybridNeRF: Efficient Neural Rendering

haithemturki.com

Post reply on HN
Print view View original

41–50 of 53 posts

Re: HybridNeRF: Efficient Neural Rendering

#41

Post by xrd »

turkihaithem wrote: post #28
refibrillator wrote: post #22

Earlier quoted context omitted.

Congrats on the paper! Any chance the code will be released? Also I’d be curious to hear, what are you excited about in terms of future research ideas? Personally I’m excited by the trend of eliminating the need for traditional SfM preprocessing (sparse point clouds via colmap, camera pose estimation, etc).

Thank you! The code is unlikely to be released (it's built upon Meta-internal codebases that I no longer have access to post-internship), at least not in the form that we specifically used at submission time. The last time I caught up with the team someone was expressing interest in releasing some broadly useful rendering code, but I really can't speak on their behalf so no guarantees. IMHO it's a really exciting tim…

Thanks, these are very exciting.

I'm interested in using Nerf to generate interpolated frames from a set of images. I want to do a poor man's animation. I'm interested in finding Nerf with code but it feels hard to find. Do you know of a good starting point? I tried running nerfstudio and the results weren't great.

Re: HybridNeRF: Efficient Neural Rendering

#42

Post by xrd »

GistNoesis wrote: post #39
lxe wrote: post #33

Absolute noob question that I'm having a hard time understading: In practice, why NeRF instead of Gaussian Splatting? I have very limited exposure to either, but a very cursory search on the subject yields a "it depends on the context" answer. What exact context?

There are two aspects in the difference between NeRF and Gaussian Splatting : - The first aspect concern how they solve the light rendering equation : NeRF has more potential for rendering physical quality but is slower. NeRF use raycasting. Gaussian Splatting project and draw gaussians directly in screen space. Each have various rendering artefacts. One distinction is in handling light reflections. When you use rayc…

This is a beautiful explanation, thanks so much!

Re: HybridNeRF: Efficient Neural Rendering

#43

Post by smusamashah »

totalview wrote: post #12
ttul wrote: post #3

Does anyone else look forward to a game that lets you transform your house or neighbor into a playable level with destructible objects? How far are we from recognizing the “car” and making it drivable, or the “tree” and making it choppable?

I work in the rendering and gaming industry and also run a 3D scanning company. I have similarly wished for this capability, especially the destructability part. What you speak of is still pretty far off for several reasons: -No Collision/poor collision on NERFs and GS: to have a proper interactive world, you usually need accurate character collision so that your character or vehicle can move along the floor/ground (…

Is there a teardown mod that uses reality captured models? Or is there any video even? I have played the game once, destruction was awesome. I want to see how it looks like the way you said.

Re: HybridNeRF: Efficient Neural Rendering

#44

Post by turkihaithem »

xrd wrote: post #41
turkihaithem wrote: post #28

Earlier quoted context omitted.

Thank you! The code is unlikely to be released (it's built upon Meta-internal codebases that I no longer have access to post-internship), at least not in the form that we specifically used at submission time. The last time I caught up with the team someone was expressing interest in releasing some broadly useful rendering code, but I really can't speak on their behalf so no guarantees. IMHO it's a really exciting tim…

Thanks, these are very exciting. I'm interested in using Nerf to generate interpolated frames from a set of images. I want to do a poor man's animation. I'm interested in finding Nerf with code but it feels hard to find. Do you know of a good starting point? I tried running nerfstudio and the results weren't great.

I might be misunderstanding your use case, but are you trying to interpolate movement between frames (ie: are you trying to reconstruct a dynamic 4D scene or is the capture fully static?)

If you are trying to capture dynamics, most of the Nerfstudio methods are geared towards static captures and will give poor results for scenes with movement. There are many dynamic NeRF works out there - for example https://dynamic3dgaussians.github.io/ and https://github.com/andrewsonga/Total-Recon both provide code if you want to play around. With that being said, robust 4D reconstruction is still very much an open research problem (especially when limited to monocular RGB data / casual phone captures). I'd expect a lot of movement in the space in the months/years to come!

Re: HybridNeRF: Efficient Neural Rendering

#45

Post by xrd »

turkihaithem wrote: post #44
xrd wrote: post #41

Earlier quoted context omitted.

Thanks, these are very exciting. I'm interested in using Nerf to generate interpolated frames from a set of images. I want to do a poor man's animation. I'm interested in finding Nerf with code but it feels hard to find. Do you know of a good starting point? I tried running nerfstudio and the results weren't great.

I might be misunderstanding your use case, but are you trying to interpolate movement between frames (ie: are you trying to reconstruct a dynamic 4D scene or is the capture fully static?) If you are trying to capture dynamics, most of the Nerfstudio methods are geared towards static captures and will give poor results for scenes with movement. There are many dynamic NeRF works out there - for example https://dynamic3…

Really appreciate your response.

I'm trying to recreate my experiences with claymation when I was a kid. I want to take a picture of an object, like a lego figure, and then move it slightly, take another picture, then move the figure again slightly. Once I have some frames, I want to use a NERFs to interpolate between those frames.

When I was young and doing claymation, I would move the figure, shoot two frames, and do that 12 times per second of film. And, the lights I used were often so hot, that my clay would melt and alter the figure. It was a chore.

I thought I could capture fewer in-between frames and let the NERF figure out the interpolation, and perhaps get some weird side effects. Especially if it hallucinates.

I'm not sure if a NERF is the right approach, but it seems like a good starting point.

Thank you.

Re: HybridNeRF: Efficient Neural Rendering

#46

Post by turkihaithem »

xrd wrote: post #45
turkihaithem wrote: post #44

Earlier quoted context omitted.

I might be misunderstanding your use case, but are you trying to interpolate movement between frames (ie: are you trying to reconstruct a dynamic 4D scene or is the capture fully static?) If you are trying to capture dynamics, most of the Nerfstudio methods are geared towards static captures and will give poor results for scenes with movement. There are many dynamic NeRF works out there - for example https://dynamic3…

Really appreciate your response. I'm trying to recreate my experiences with claymation when I was a kid. I want to take a picture of an object, like a lego figure, and then move it slightly, take another picture, then move the figure again slightly. Once I have some frames, I want to use a NERFs to interpolate between those frames. When I was young and doing claymation, I would move the figure, shoot two frames, and…

You might also want to take a look at diffusion models: https://vidim-interpolation.github.io/

In terms of publicly available code, I think Stable Video Diffusion can do frame interpolation (https://stability.ai/news/introducing-stable-video-diffusion...), but I haven't tried it myself.

Re: HybridNeRF: Efficient Neural Rendering

#47

Post by xrd »

turkihaithem wrote: post #46
xrd wrote: post #45

Earlier quoted context omitted.

Really appreciate your response. I'm trying to recreate my experiences with claymation when I was a kid. I want to take a picture of an object, like a lego figure, and then move it slightly, take another picture, then move the figure again slightly. Once I have some frames, I want to use a NERFs to interpolate between those frames. When I was young and doing claymation, I would move the figure, shoot two frames, and…

You might also want to take a look at diffusion models: https://vidim-interpolation.github.io/ In terms of publicly available code, I think Stable Video Diffusion can do frame interpolation ( https://stability.ai/news/introducing-stable-video-diffusion... ), but I haven't tried it myself.

That stuff is perfect. Thanks. I will definitely play with this.

Re: HybridNeRF: Efficient Neural Rendering

#48

Post by choppaface »

lxe wrote: post #33

Absolute noob question that I'm having a hard time understading: In practice, why NeRF instead of Gaussian Splatting? I have very limited exposure to either, but a very cursory search on the subject yields a "it depends on the context" answer. What exact context?

NeRF does glass, fog, reflections, and some furs better than gsplat. Gsplat does normal surfaces as well or better than NeRF, and gsplat also provides explicit geometry (a point cloud). NeRF models the entire image rendering function while gsplats only model typical surfaces.

This work (and others e.g. https://creiser.github.io/binary_opacity_grid/ ) attempt to blend the raytracing aspect of NeRF with the explicit surface aspect of gsplats.

One key non-research problem is that gsplats can render on mobile devices / headsets using vanilla WebGL APIs. But approaches like this paper require CUDA (and also apparently a top-shelf desktop GPU). If Apple and others (mostly Apple has been sandbagging tho) provided better support for WebGPU or an alternative API then NeRF research would be dramatically more impactful versus gsplats. The popularity of gsplats is largely due to its accessibility.

Re: HybridNeRF: Efficient Neural Rendering

#49

Post by choppaface »

turkihaithem wrote: post #17

One of the paper authors here - happy to answer any questions about the work or chat about neural rendering in general!

Re: the 4FPS example, if one renders a VR180 view as an equirectangular image and lets the headset handle the 6DOF head movement, then 4FPS is plenty. Especially if there’s one render per-eye. 110% if there are no objects within a meter of the ego camera.

So your motivating problem does not exist.

More FPS is better, and yes we all do want to find a hybrid of NeRF and splats that works well, but then you should emphasize your theoretical and experimental contributions. Flatly claiming 4FPS doesn’t work is specious to most readers. Even Deva knows this is being too aggressive for a paper like this.

Re: HybridNeRF: Efficient Neural Rendering

#50

Post by eloisius »

modeless wrote: post #20
totalview wrote: post #12

Earlier quoted context omitted.

I work in the rendering and gaming industry and also run a 3D scanning company. I have similarly wished for this capability, especially the destructability part. What you speak of is still pretty far off for several reasons: -No Collision/poor collision on NERFs and GS: to have a proper interactive world, you usually need accurate character collision so that your character or vehicle can move along the floor/ground (…

I think collision detection is solvable. And the scanning process should be no harder than 3D modeling to the same quality level. Probably much easier, honestly. Modeling is labor intensive. I'm not sure why you say "there’s no scanner available that provides both good 3-D information and good photo realistic textures" because these new techniques don't use "scanners", all you need is regular cameras. The 3D informat…

There’s some exciting research on recovering light sources https://dorverbin.github.io/eclipse/
Post reply on HN