Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gaussian Splatting – A$AP Rocky "Helicopter" music video

radiancefields.com

Post reply on HN
Print view View original

261–270 of 278 posts

Re: Gaussian Splatting – A$AP Rocky "Helicopter" music video

#261

Post by jeffsipko »

Hi all! If you watched the video and thought you'd love to make stuff like that, but then you looked at the cost of a volumetric capture and nope'ed out, we should talk!

I've been developing a solution to make the cost of 4D capture an order of magnitude cheaper by using a small number of off-the-shelf cameras. Here's the proof-of-concept demo using 4x GoPros: https://youtube.com/shorts/Y56l0FlLlAg (yes, lots of room to improve quality). You can also see the interactive version (with XR support) at https://gaussplay.lovable.app

Re: Gaussian Splatting – A$AP Rocky "Helicopter" music video

#263

Post by HaZeust »

wahnfrieden wrote: post #191
HaZeust wrote: post #187

Earlier quoted context omitted.

r/playboicarti is one of my favorite places to go to just turn my brain off and see shitposts that have a certain reminiscence to me, almost a "high school class when the teacher didn't show up" vibe.

95% of its posters are in high school or lower, and are in class during daytime hours, so that's a part of why it makes you feel like that

Indeed. It's a good vibe when you want to turn your brain off sometimes though, same reason why Beavis and Butthead succeeds and is re-aired.

Re: Gaussian Splatting – A$AP Rocky "Helicopter" music video

#264

Post by make3 »

meindnoch wrote: post #78
pleurotus wrote: post #8

Super cool to read but can someone eli5 what Gaussian splatting is (and/or radiance fields?) specifically to how the article talks about it finally being "mature enough"? What's changed that this is now possible?

1. Create a point cloud from a scene (either via lidar, or via photogrammetry from multiple images) 2. Replace each point of the point cloud with a fuzzy ellipsoid, that has a bunch of parameters for its position + size + orientation + view-dependent color (via spherical harmonics up to some low order) 3. If you render these ellipsoids using a differentiable renderer, then you can subtract the resulting image from th…

I assume that the differentiable renderer is only given its position and viewing angle at any one time (in order to be able to generalize to new viewing angles)?

Is it a fully connected NN?

Re: Gaussian Splatting – A$AP Rocky "Helicopter" music video

#265

Post by tokymegz »

origamiarmy wrote: post #158
tokymegz wrote: post #155

Hello! I’m Chris Rutledge, the post EP / cg supervisor at Grin Machine. Happy to answer any questions. Glad people are enjoying this video, was so fun to get to play with this technique and help break it into some mainstream production

Awesome work, incredibly well done! What was the process like for setting the direction on use of these techniques with Rakim? Were you basically just trusted to make something great or did they have a lot of opinions on the technicalities?

I didn’t interface much with Rocky outside of the shoot, our director Dan was talking to him regularly and he certainly had options and great ideas but mostly left it up to us. All of the creative came from Dan and bouncing ideas off us / trying things and seeing what was possible with this tech. By the end of the process it was awesome to get Dan also into blender helping set up camera moves himself, in addition to finessing the edit and animatic to help point us in the right direction.

Re: Gaussian Splatting – A$AP Rocky "Helicopter" music video

#266

Post by suzzer99 »

two_handfuls wrote: post #170
suzzer99 wrote: post #38

Earlier quoted context omitted.

Can you put "Gaussing splats" in some kind of real world metaphor so I can understand what it means? Either that or explain why "Gaussian" and why "splat". I am vaguely aware of stuff like Gaussian blur on Photoshop. But I never really knew what it does.

> explain why "Gaussian" and why "splat". Happily. Gaussian splats are a technique for 3D images, related to point clouds. They do the same job (take a 3D capture of reality and generate pictures later from any point of view "close enough" to the original). The key idea is that instead of a bunch of points, it stores a bunch of semi-transparent blobs - or "splats". The transparency increases quickly with distance, fo…

Somehow this hit right in the sweet spot at my level of knowledge. Thanks!

Re: Gaussian Splatting – A$AP Rocky "Helicopter" music video

#267

Post by meindnoch »

make3 wrote: post #264
meindnoch wrote: post #78

Earlier quoted context omitted.

1. Create a point cloud from a scene (either via lidar, or via photogrammetry from multiple images) 2. Replace each point of the point cloud with a fuzzy ellipsoid, that has a bunch of parameters for its position + size + orientation + view-dependent color (via spherical harmonics up to some low order) 3. If you render these ellipsoids using a differentiable renderer, then you can subtract the resulting image from th…

I assume that the differentiable renderer is only given its position and viewing angle at any one time (in order to be able to generalize to new viewing angles)? Is it a fully connected NN?

No. There are no neural networks here. The renderer is just a function that takes a bunch of ellipsoid parameters and outputs a bunch of pixels. You render the scene, then subtract the ground truth pixels from the result, and sum the squared differences to get the total error. Then you ask the question "how would the error change if the X position of ellipsoid #1 was changed slightly?" (then repeat for all ellipsoid parameters, not just the X position, and all ellipsoids, not just ellipsoid #1). In other words, compute the partial derivative of the error with respect to each ellipsoid parameter. This gives you a gradient, that you can use to adjust the ellipsoids to decrease the error (i.e. get closer to the ground truth image).

Re: Gaussian Splatting – A$AP Rocky "Helicopter" music video

#268

Post by numpad0 »

itishappy wrote: post #240
numpad0 wrote: post #165

Earlier quoted context omitted.

A LIDAR point cloud from a single point of view is a mono-ocular depth map. Unless the LIDAR in question is like, using supernova level gamma rays or neutrino generators for the laser part to get density and albedo volumetric data for its whole distance range. You just can't see the back of a thing by knowing the shape of the front side with current technologies.

Right! My terminology may be imprecise here, but I believe there is still an important distinction: The depth map stored for image processing is image metadata, meaning it calculates one depth per pixel from a single position in space. Note that it doesn't have the ability to measure that many depth values, so it measures what it can using LIDAR and focus information and estimates the rest. On the other hand, a point…

No, LIDAR data are necessarily taken from a single position. They are 3D, but literally single eyed. You can't tell from LIDAR data if you're looking at a half-cut apple or an intact one. This becomes obvious the moment you tried to rotate a LIDAR capture - it's just the skin. You need depth maps from all angles to reconstruct the complete skin.

So you have to have minimum two for front and back of a dancer. Actually, the seams are kind of dubious so let's say three 120 degrees apart. Well we need ones looking down as well as up for baggy clothing, so more like nine, 30 degrees apart vertically and 120 degrees horizontally, ...

and ^ this will go far down enough that installing few dozens of identical non-Apple cameras in a monstrous sci-fi cage starts making a lot more sense than an iPhone, for a video.

Re: Gaussian Splatting – A$AP Rocky "Helicopter" music video

#270

Post by antidamage »

numpad0 wrote: post #112
secretsatan wrote: post #55

Earlier quoted context omitted.

Couldn’t you just use iphone pros for this? I developed an app specifically for photogrammetry capture using AR and the depth sensor as it seemed like a cheap alternative. EDIT: I realize a phone is not on the same level as a red camera, but i just saw iphones as a massively cheaper option to alternatives in the field i worked in.

A single camera only captures the side of the object facing the camera. Knowing how far away that camera facing side of a Rubik's Cube help if you were making educated guesses(novel view synthesis), but it won't solve the problem of actually photographing the backside. There are usually six sides on a cube, which means you need minimum six iPhone around an object to capture all sides of it to be able to then freely m…

The minimum is four RGB-only cameras (if you want RGB data) but adding lidar really helps.

The standard pipeline can infer a huge amount of data, and there are a few AI tools now for hallucinating missing geometry and backfaces based on context recognition, which can then be converted back into a splat for fast, smooth rendering.

Post reply on HN