Background Matting: The World is Your Green Screen • Hacker News Board

Re: Background Matting: The World is Your Green Screen

#41

Post by Joeboy » Sat, Apr 25, 2020, 12:35 PM UTC

web007 wrote: post #13

What's novel about this? If you have a background picture, you have all the info you need to identify your subject - just plain subtraction. I think this is what the Photo Booth app on my circa-2012 MacBook does, quite effectively.

> If you have a background picture, you have all the info you need to identify your subject - just plain subtraction

It's not really "just plain subtraction", it's keying. Which AIUI basically means setting the alpha according to the difference between the image and the reference.

Green screen works well for this because, excepting Zoe Soldana, people tend to hang out around the opposite side of the colour wheel, so there tends to be a good distance between foreground colour and background colour. If you're trying to do this against arbitrary backgrounds, you seemingly need to augment keying with additional techniques like image segmentation to get good results.

Re: Background Matting: The World is Your Green Screen

#42

Post by asdf10 » Sat, Apr 25, 2020, 12:50 PM UTC

fock wrote: post #28

asdf10 wrote: post #21

Earlier quoted context omitted.

This new method works well for partially transparent regions (hair) and allows slightly larger background movement and color overlap between foreground and background.

I think they baited you with their "loook, movement" replacement videos. As far as I can tell, their inputs have a fixed background and are of constant exposure and camera position.

No, the camera is indeed allowed to change a tiny bit. For example, you do not need a tripod. Taking photos with a handheld camera works fine (although a tripod works even better). They explain it in greater detail in their paper: https://arxiv.org/pdf/2004.00626.pdf

Background subtraction methods on the other hand usually fail if the camera moves even a tiny bit or the lighting changes slightly. More advanced methods can recover eventually, but you still get a few frames with improperly removed background.

Re: Background Matting: The World is Your Green Screen

#43

Post by ollifi » Sat, Apr 25, 2020, 1:10 PM UTC

Anybody know of any work done to improve greenscreen keying? The current old school techniques work quite poorly and require so much manual work. I would imagine with the new work coming out with neural nets etc. there would be possibilities for improvement. This is very cool work and good for certain applications, but seems to produce similar problems like greenscreen to some edges.

Re: Background Matting: The World is Your Green Screen

#44

Post by Joeboy » Sat, Apr 25, 2020, 1:20 PM UTC

e_carra wrote: post #29

web007 wrote: post #13

What's novel about this? If you have a background picture, you have all the info you need to identify your subject - just plain subtraction. I think this is what the Photo Booth app on my circa-2012 MacBook does, quite effectively.

I thought the same thing. I remember vividly playing with PhotoBoot on my aunt's MacBook, I really enjoyed the rollercoaster!

That would be https://www.youtube.com/watch?v=588kZu1JeFw ?

I mean, it's trying to do the same kind of thing, but this looks to be a lot better at it.

Re: Background Matting: The World is Your Green Screen

#45

Post by shubidubi » Sat, Apr 25, 2020, 1:29 PM UTC

Is there an easy way to stream it to Zoom/Slack etc...? Will be nice to use it as a virtual camera source

Re: Background Matting: The World is Your Green Screen

#46

Post by kmfrk » Sat, Apr 25, 2020, 1:30 PM UTC

ollifi wrote: post #43

Anybody know of any work done to improve greenscreen keying? The current old school techniques work quite poorly and require so much manual work. I would imagine with the new work coming out with neural nets etc. there would be possibilities for improvement. This is very cool work and good for certain applications, but seems to produce similar problems like greenscreen to some edges.

Nvidia have had automatic "greenscreen keying" without a greenscreen in some sort of beta for a long time, but it still hasn't moved beyond that: https://blogs.nvidia.com/blog/2019/09/26/nvidia-rtx-broadcas....

Re: Background Matting: The World is Your Green Screen

#47

Post by cricalix » Sat, Apr 25, 2020, 2:28 PM UTC

myridium wrote: post #2

Is it fast enough to operate on a live video feed from, say, a webcam?

XSplit's VCam can do background removal without chroma-key. It's reasonably good, and emits a virtual webcam that VC clients or OBS can use as an input. Think it's 40 USD for a lifetime license. Has a bit of ghosting when you move quickly.

Re: Background Matting: The World is Your Green Screen

#48

Post by pierrebai » Sat, Apr 25, 2020, 2:46 PM UTC

KaiserPro wrote: post #25

web007 wrote: post #13

What's novel about this? If you have a background picture, you have all the info you need to identify your subject - just plain subtraction. I think this is what the Photo Booth app on my circa-2012 MacBook does, quite effectively.

Ex-VFX person here. Green screens are crap with hair, because its translucent the green/blue bleeds through which means that it has to be cleaned up by hand. Then there are the situations where there isn't a green screen. Again manual cleanup is required. Each frame needs to be cut out by hand. 24 times a second. The same with a difference matte. Cameras are noisy, so there is constant noise in the alpha channel. Thi…

Ex-VFX part-time software team member here.

I can assure you they read these papers. If it is good, it will be part of a future version of the software.

Re: Background Matting: The World is Your Green Screen

#49

Post by aledthemathguy » Sat, Apr 25, 2020, 2:59 PM UTC

That's @!#!@$ magical!

Re: Background Matting: The World is Your Green Screen

#50

Post by illumanaughty » Sat, Apr 25, 2020, 3:03 PM UTC

laumars wrote: post #27

This does look like an interesting step forward however I’ve found the biggest limitations of previous techniques to be - people who are using occupying a wider z-axis (for example leaning forwards in the camera or who have arms in front of them) - people holding objects like cups How well do your method handle those kind of situations?

I'm not sure how either of those situations would trip up the system they're using. For a system trained on the background image, what difference does it make if the subject is holding a cup? The cup is not the background image, and it would be obvious in the same way that it's obvious the subject isn't the background image.