RenderFormer: Neural rendering of triangle meshes with global illumination

Wow. The loop is closed with GPUs then. Rendering to compute to rendering.

Raytracing, The Matrix edition. Feels like an odd round about we're in.

Forgive my ignorance: are these scenes rendered based on how a scene is expected to be rendered? If so, why would we use this over more direct methods (since I assume this is not faster than direct methods)?

The coolest thing here might be the speed: for a given scene RenderFormer takes 0.0760 seconds while Blender Cycles takes 3.97 seconds (or 12.05 secs at a higher setting), while retaining a 0.9526 Structural Similarity Index Measure (0-1 where 1 is an identical image). See tables 2 and 1 in the paper.

This could possibly enable higher quality instant render previews for 3D designers in web or native apps using on-device transformer models.

Note the timings above were on an A100 with an unoptimized PyTorch version of the model. Obviously the average user's GPU is much less powerful, and for 3D designers it might be still powerful enough to see significant speedups over traditional rendering. Or for a web-based system it could even connect to A100s on the backend and stream the images to the browser.

Limitations are that it's not fully accurate especially as scene complexity scales, e.g. with shadows of complex shapes (plus I imagine particles or strands), so the final renders will probably still be done traditionally to avoid any of the nasty visual artifacts common in many AI-generated images/videos today. But who knows, it might be "good enough" and bring enough of a speed increase to justify use by big animation studios who need to render full movie-length previews to use for music, story review, etc etc.

This has been researched quite a lot with the adjacent task of structure from motion (2d videos) - see NeRFs, Gaussian Splatting - and now there are a few models that do 3d from single 2d images fairly well.