Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It Took Me 30 Years to Solve This VFX Problem – Green Screen Problem [video]

youtube.com

Post reply on HN
Print view View original

11–20 of 119 posts

Re: It Took Me 30 Years to Solve This VFX Problem – Green Screen Problem [video]

#11

Post by diacritical »

somat wrote: post #10
Springtime wrote: post #3

In an earlier video they made a couple years back about Disney's sodium vapor technique Paul Debevec suggested he was considering creating a dataset using a similar premise: filming enough perfectly masked references to be able to train models to achieve better keying. So it was interesting seeing Corridor tackle this by instead using synthetic data.

With regards to the sodium vapor process, an idea has been percolating in the back of my head ever since I saw that video. But I don't really have the budget to try it out. theory: make the mask out of non-visable light illuminate the backing screen in near Infra-Red light. (after a bit of thought I chose near-IR as opposed to near-UV for hopefully obvious reasons) point two cameras at a splitting prism with a near I…

Don't humans and other warm objects also radiate IR?

Re: It Took Me 30 Years to Solve This VFX Problem – Green Screen Problem [video]

#12

Post by overvale »

somat wrote: post #10
Springtime wrote: post #3

In an earlier video they made a couple years back about Disney's sodium vapor technique Paul Debevec suggested he was considering creating a dataset using a similar premise: filming enough perfectly masked references to be able to train models to achieve better keying. So it was interesting seeing Corridor tackle this by instead using synthetic data.

With regards to the sodium vapor process, an idea has been percolating in the back of my head ever since I saw that video. But I don't really have the budget to try it out. theory: make the mask out of non-visable light illuminate the backing screen in near Infra-Red light. (after a bit of thought I chose near-IR as opposed to near-UV for hopefully obvious reasons) point two cameras at a splitting prism with a near I…

Debevec tried a version of this: https://arxiv.org/abs/2306.13702

Re: It Took Me 30 Years to Solve This VFX Problem – Green Screen Problem [video]

#13

Post by somat »

diacritical wrote: post #11
somat wrote: post #10

Earlier quoted context omitted.

With regards to the sodium vapor process, an idea has been percolating in the back of my head ever since I saw that video. But I don't really have the budget to try it out. theory: make the mask out of non-visable light illuminate the backing screen in near Infra-Red light. (after a bit of thought I chose near-IR as opposed to near-UV for hopefully obvious reasons) point two cameras at a splitting prism with a near I…

Don't humans and other warm objects also radiate IR?

That is far-IR, thermal stuff, Near-IR, 700 nanometer-ish is right below red in human vision.

Camera sensors can pick up a little near-IR so they have have a filter to block it. If that filter was removed and a filter to block visable light was used in place you would have a camera that can only see non-visable light. Poorly, the camera was not engineered to operate in this bandwidth, but it might be good enough for a mask. A mask that does not interfere with any visible colors.

Re: It Took Me 30 Years to Solve This VFX Problem – Green Screen Problem [video]

#14

Post by jasonwatkinspdx »

diacritical wrote: post #7

From ~04:10 till 05:00 they talk about sodium-vapor lights and how Disney has the exclusive rights to use it. From what I read the knowledge on how to make them is a trade secret, so it's not patented. Seems weird that it would be hard to recreate something from the 1950's. I also wonder how many hours were wasted by people who had to use inferior technology because Disney kept it secret. Cutting out animals and obje…

Yeah, that's just nonsense. We used sodium vapor monochromatic bulbs in my high school physics class to duplicate the double slit experiment.

I suspect the real reason is that digital green screen in the hands of experienced people is "good enough" vs the complication of needing a double camera and beam splitting prism rig and such.

Re: It Took Me 30 Years to Solve This VFX Problem – Green Screen Problem [video]

#16

Post by actionfromafar »

somat wrote: post #10
Springtime wrote: post #3

In an earlier video they made a couple years back about Disney's sodium vapor technique Paul Debevec suggested he was considering creating a dataset using a similar premise: filming enough perfectly masked references to be able to train models to achieve better keying. So it was interesting seeing Corridor tackle this by instead using synthetic data.

With regards to the sodium vapor process, an idea has been percolating in the back of my head ever since I saw that video. But I don't really have the budget to try it out. theory: make the mask out of non-visable light illuminate the backing screen in near Infra-Red light. (after a bit of thought I chose near-IR as opposed to near-UV for hopefully obvious reasons) point two cameras at a splitting prism with a near I…

I'll do you one better, which requires no special cameras (most have IR filters) nor double cameras or prisms.

Shoot the scene in 48 or 96 fps. Sync the set lighting to odd frames. Every odd frame, the set lights are on. Every even frame, set lights are off.

For the backing screen, do the reverse. Even frames, the backing screen is on. Odd frames, backing screen is off.

There you go. Mask / normal shot / Mask normal shot / Mask ... you get the idea.

Of course, motion will cause normal image and mask go out of sync, but I bet that can be remedied by interpolating a new frame between every mask frame. Plus, when you mix it down to 24fps you can introduce as much motion blur and shutter angle "emulation" as you want.

Re: It Took Me 30 Years to Solve This VFX Problem – Green Screen Problem [video]

#17

Post by jrm4 »

ralusek wrote: post #9

I'm a software engineer that, like the vast majority of you, uses AI/agents in my workflow every day. That being said, I have to admit that it feels a little weird to hear someone who does not write code say that they built something, without even mentioning that they had an agent build it (unless I missed that).

I mean, the heading of the video says "he solved the problem," which I think is wise to pay a lot of attention to.

Re: It Took Me 30 Years to Solve This VFX Problem – Green Screen Problem [video]

#18

Post by ryandamm »

actionfromafar wrote: post #16
somat wrote: post #10

Earlier quoted context omitted.

With regards to the sodium vapor process, an idea has been percolating in the back of my head ever since I saw that video. But I don't really have the budget to try it out. theory: make the mask out of non-visable light illuminate the backing screen in near Infra-Red light. (after a bit of thought I chose near-IR as opposed to near-UV for hopefully obvious reasons) point two cameras at a splitting prism with a near I…

I'll do you one better, which requires no special cameras (most have IR filters) nor double cameras or prisms. Shoot the scene in 48 or 96 fps. Sync the set lighting to odd frames. Every odd frame, the set lights are on. Every even frame, set lights are off. For the backing screen, do the reverse. Even frames, the backing screen is on. Odd frames, backing screen is off. There you go. Mask / normal shot / Mask normal…

This is called “ghost frame” and already exists in Red cameras and virtual production wall tools like Disguise.

Re: It Took Me 30 Years to Solve This VFX Problem – Green Screen Problem [video]

#20

Post by rhdunn »

superjan wrote: post #2

Watched this a few days ago. The video is light on technical details, except maybe that they used CGI to generate training data.

The idea behind a greenscreen is that you can make that green colour transparent in the frames of footage allowing you to blend that with some other background or other layered footage. This has issues like not always having a uniform colour, difficulty with things like hair, and lighting affecting some edges. These have to be manually cleaned up frame-by-frame, which takes a lot of time that is mostly busy work.

An alternative approach (such as that used by the sodium lighting on Mary Poppins) is that you create two images per frame -- the core image and a mask. The mask is a black and white image where the white pixels are the pixels to keep and the black pixels the ones to discard. Shades of gray indicate blended pixels.

For the mask approach you are filming a perfect alpha channel to apply to the footage that doesn't have the issues of greenscreen. The problem is that this requires specialist, licensed equipment and perfect filming conditions.

The new approach is to take advantage of image/video models to train a model that can produce the alpha channel mask for a given frame (and thus an entire recording) when just given greenscreen footage.

The use of CGI in the training data allows the input image and mask to be perfect without having to spend hundreds of hours creating that data. It's also easier to modify and create variations to test different cases such as reflective or soft edges.

Thus, you have the greenscreen input footage, the expected processed output and alpha channel mask. You can then apply traditional neural net training techniques on the data using the expected image/alpha channel as the target. For example, you can compute the difference on each of the alpha channel output neurons from the expected result, then apply backpropagation to compute the differences through the neural network, and then nudge the neuron weights in the computed gradient direction. Repeat that process across a distribution of the test images over multiple passes until the network no longer changes significantly between passes.

Post reply on HN