Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RenderFormer: Neural rendering of triangle meshes with global illumination

microsoft.github.io

Post reply on HN
Print view View original

21–30 of 59 posts

Re: RenderFormer: Neural rendering of triangle meshes with global illumination

#21

Post by jiggawatts »

timhigins wrote: post #8

The coolest thing here might be the speed: for a given scene RenderFormer takes 0.0760 seconds while Blender Cycles takes 3.97 seconds (or 12.05 secs at a higher setting), while retaining a 0.9526 Structural Similarity Index Measure (0-1 where 1 is an identical image). See tables 2 and 1 in the paper. This could possibly enable higher quality instant render previews for 3D designers in web or native apps using on-dev…

I wonder if the model could be refined on the fly by rendering small test patches using traditional methods and using that as the feedback for a LoRA tuning layer or some such.

Re: RenderFormer: Neural rendering of triangle meshes with global illumination

#23

Post by buildartefact »

timhigins wrote: post #8

The coolest thing here might be the speed: for a given scene RenderFormer takes 0.0760 seconds while Blender Cycles takes 3.97 seconds (or 12.05 secs at a higher setting), while retaining a 0.9526 Structural Similarity Index Measure (0-1 where 1 is an identical image). See tables 2 and 1 in the paper. This could possibly enable higher quality instant render previews for 3D designers in web or native apps using on-dev…

For the scenes that they’re showing, 76ms is an eternity. Granted, it will get (a lot) faster but this being better than traditional rendering is a way off yet.

Re: RenderFormer: Neural rendering of triangle meshes with global illumination

#24

Post by mhl47 »

timhigins wrote: post #10
johnb231 wrote: post #9

Earlier quoted context omitted.

It's not pointless. This is research. Here's something interesting from the conclusion of the paper: "An interesting and promising direction for future work that leverages the inherent differentiability, would be to apply RenderFormer to inverse rendering applications." That means generate a 3D scene from 2D images.

This has been researched quite a lot with the adjacent task of structure from motion (2d videos) - see NeRFs, Gaussian Splatting - and now there are a few models that do 3d from single 2d images fairly well.

And interestingly it's still not even close to what a good human artist could do in this task having the same data.

Re: RenderFormer: Neural rendering of triangle meshes with global illumination

#25

Post by cubefox »

timhigins wrote: post #8

The coolest thing here might be the speed: for a given scene RenderFormer takes 0.0760 seconds while Blender Cycles takes 3.97 seconds (or 12.05 secs at a higher setting), while retaining a 0.9526 Structural Similarity Index Measure (0-1 where 1 is an identical image). See tables 2 and 1 in the paper. This could possibly enable higher quality instant render previews for 3D designers in web or native apps using on-dev…

> The runtime-complexity of attention layers scales quadratically with the number of tokens, and thus triangles in our case. As a result, we limit the total number of triangles in our scenes to 4,096;

Re: RenderFormer: Neural rendering of triangle meshes with global illumination

#26

Post by jsheard »

buildartefact wrote: post #23
timhigins wrote: post #8

The coolest thing here might be the speed: for a given scene RenderFormer takes 0.0760 seconds while Blender Cycles takes 3.97 seconds (or 12.05 secs at a higher setting), while retaining a 0.9526 Structural Similarity Index Measure (0-1 where 1 is an identical image). See tables 2 and 1 in the paper. This could possibly enable higher quality instant render previews for 3D designers in web or native apps using on-dev…

For the scenes that they’re showing, 76ms is an eternity. Granted, it will get (a lot) faster but this being better than traditional rendering is a way off yet.

Yeah, and the big caveat with this approach is that it scales quadratically with scene complexity, as opposed to the usual methods which are logarithmic. Their examples only have 4096 triangles at most for that reason. It's a cool potential direction for future research but there's a long way to go before it can wrangle real production scenes with hundreds of millions of triangles.

Re: RenderFormer: Neural rendering of triangle meshes with global illumination

#27

Post by cubefox »

dclowd9901 wrote: post #6

Forgive my ignorance: are these scenes rendered based on how a scene is expected to be rendered? If so, why would we use this over more direct methods (since I assume this is not faster than direct methods)?

Presumably because it is Cool Research (TM). It's not useful, since the cost increases quadratically with the number of triangles. Which is why they only had 4096 per scene.

Re: RenderFormer: Neural rendering of triangle meshes with global illumination

#28

Post by bemmu »

dclowd9901 wrote: post #6

Forgive my ignorance: are these scenes rendered based on how a scene is expected to be rendered? If so, why would we use this over more direct methods (since I assume this is not faster than direct methods)?

This will probably have some cool non-obvious benefits.

For instance if the scenes are a blob of input weights, what would it look like to add some noise to those, could you get some cool output that wouldn't otherwise be possible?

Would it look interesting if you took two different scene representations and interpolated between them? Etc. etc.

Re: RenderFormer: Neural rendering of triangle meshes with global illumination

#29

Post by kilpikaarna »

timhigins wrote: post #8

The coolest thing here might be the speed: for a given scene RenderFormer takes 0.0760 seconds while Blender Cycles takes 3.97 seconds (or 12.05 secs at a higher setting), while retaining a 0.9526 Structural Similarity Index Measure (0-1 where 1 is an identical image). See tables 2 and 1 in the paper. This could possibly enable higher quality instant render previews for 3D designers in web or native apps using on-dev…

> The coolest thing here might be the speed: for a given scene RenderFormer takes 0.0760 seconds while Blender Cycles takes 3.97 seconds (or 12.05 secs at a higher setting), while retaining a 0.9526 Structural Similarity Index Measure (0-1 where 1 is an identical image). See tables 2 and 1 in the paper.

This sounds pretty wild to me. Scanned through it quickly but I couldn't find any details on how they set this up. Do they use the CPU or the Cuda kernel on an A100 for Cycles? Also, if this is doing single frames an appreciable fraction of the 3.97s might go into firing up the renderer. Time-per-frame would drop off if rendering a sequence.

And the complexity scaling per triangle mentioned in a sibling comment. Ouch!

Re: RenderFormer: Neural rendering of triangle meshes with global illumination

#30

Post by fulafel »

kilpikaarna wrote: post #29
timhigins wrote: post #8

The coolest thing here might be the speed: for a given scene RenderFormer takes 0.0760 seconds while Blender Cycles takes 3.97 seconds (or 12.05 secs at a higher setting), while retaining a 0.9526 Structural Similarity Index Measure (0-1 where 1 is an identical image). See tables 2 and 1 in the paper. This could possibly enable higher quality instant render previews for 3D designers in web or native apps using on-dev…

> The coolest thing here might be the speed: for a given scene RenderFormer takes 0.0760 seconds while Blender Cycles takes 3.97 seconds (or 12.05 secs at a higher setting), while retaining a 0.9526 Structural Similarity Index Measure (0-1 where 1 is an identical image). See tables 2 and 1 in the paper. This sounds pretty wild to me. Scanned through it quickly but I couldn't find any details on how they set this up.…

This reads like they used the GPU with Cycles:

  "Table 2 compares the timings on the four scenes in Figure 1 of our
  unoptimized RenderFormer (pure PyTorch implementation without
  DNN compilation, but with pre-caching of kernels) and Blender Cy-
  cles with 4,096 samples per pixel (matching RenderFormer’s training
  data) at 512 × 512 resolution on a single NVIDIA A100 GPU."
Post reply on HN