Open-Source Virtual Background

Post by BenTheElder » Fri, Apr 10, 2020, 7:45 PM UTC

renewiltord wrote: post #68

TeMPOraL wrote: post #35

Earlier quoted context omitted.

I guess I must be completely miscalibrated wrt. performance of newer technologies, because I'd imagine it's the opposite. In particular, I'd be surprised to get a Python+Node loop passing large amounts of data around like that to run 30+ FPS, unless everything Python-side is carefully written to do everything on C side. At the same time, I'd assume the inference/ML part is the fastest one, because, as far as I unders…

A modern laptop will run Bodypix at about 30 fps. There could be additional bottlenecks but the deep (and wide) NNs are usually not super fast, they're just fast for the wondrous things they do. You can usually alter performance (with Bodypix that's an accuracy/speed tradeoff) or do something silly like downscale, run, and upscale the mask. I'd like to try this.

Post by BenTheElder » Fri, Apr 10, 2020, 7:52 PM UTC

dh2424 wrote: post #49

This is pretty awesome work, but I just wanted to point out that Zoom doesn't actually require a green screen. If you uncheck the "I have a green screen" button, choosing your own virtual background still looks really good, although of course you're not going to get any crazy effects like this script adds.

Post by xyzzy_plugh » Fri, Apr 10, 2020, 9:11 PM UTC

bronco21016 wrote: post #71

wdfx wrote: post #64

Earlier quoted context omitted.

Thanks for the info, but I already have all of this part working. The problem is that v4l2loopback only provides a virtual _webcam_ (video source), not a virtual _screen_ - the two are different and are handled differently both by browsers (webrtc) and desktop conference apps (Slack, Teams, etc). I guess the other issue is that conference apps treat webcam and screen capture differently; usually if someone is sharing…

OBS will handle the screen grabbing. At least on Ubuntu 18.04 I can select individual windows or even the whole screen.

Post by manbash » Sat, Apr 11, 2020, 8:39 AM UTC

asveikau wrote: post #77

manbash wrote: post #59

Great read! I am curious then why the Linux client doesn't supports this, if all it takes is to send out our webcam stream data to be processed server-side? P.S. What happens when they do e2ee on the webcam stream?

I think server side processing is really not a good idea for this. Zoom is really seeing a lot of use right now. It would not be sustainable for them to not take advantage of all the computing power of the clients.

Post by rogierhofboer » Sat, Apr 11, 2020, 5:25 PM UTC

BenTheElder wrote: post #30

sigwinch28 wrote: post #25

You mention the ~10FPS performance. It seems like moving all that data backwards and forwards between Python and Node might be a bottleneck, no?

I need to profile it more closely, I actually don't remember the exact FPS etc. but I don't expect that to be the limiting factor. The inference / ml is expensive (which I did profile initially...), and I suspect not really optimized on this backend. It appears to be faster with webGL in the browser. I sorta stopped worrying about it once it was "good enough" to show up to a few meetings with, but with all the attent…

    from keras.models import load_model


    model = load_model('models/transpose_seg/deconv_bnoptimized_munet.h5', compile=False)


    def get_mask(frame):
        # Preprocess
        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

        simg = cv2.resize(frame, (128, 128), interpolation=cv2.INTER_AREA)
        simg = simg.reshape((1, 128, 128, 3)) / 255.0

        # Predict
        out = model.predict(simg)

        # Postprocess
        msk = out.reshape((128, 128, 1))
        mask = cv2.resize(msk, (frame.shape[1], frame.shape[0]))

        return mask