Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-Source Virtual Background

elder.dev

Post reply on HN
Print view View original

21–30 of 85 posts

Re: Open-Source Virtual Background

#21

Post by BenTheElder »

alexellisuk wrote: post #17

Fascinating write-up Ben, who would have known that you were a genius with image processing as well as running containers :-) Love the gory details and I didn't know about pyfakewebcam either. Do you have a live video recorded showing how quickly it can process a stream?

The demo at the end of the page is a video (webm), but there's not a ton of motion to reference besides the blinking.

IIRC it's something like 10FPS currently which is sufficient enough for meetings so far (about 1/3 what you might get with sufficient bandwidth in most video conference tools).

There's definitely room to improve it.

Re: Open-Source Virtual Background

#23

Post by BenTheElder »

knotty66 wrote: post #18

Good article, thanks. BTW, this is the method Jitsi-meet uses. They also use BodyPix. https://github.com/jitsi/jitsi-meet/blob/master/react/featur...

Thanks for sharing -- With very similar settings too! (Not far from the defaults unsurprisingly).

Injecting this into a web client seems like the sweet spot effort wise.

Re: Open-Source Virtual Background

#24

Post by BenTheElder »

villgax wrote: post #22

Pretty insane to use Docker & web-requests for high frame rate video chat with something which runs just fine even on CPU.

Docker made it easier to package up the dependencies (especially aligning cuda etc.), and containers are my dayjob :-)

The web requests are just an easy mode of IPC to pass around some bags of bytes, "high frame rate" is at most 30 qps ... that part isn't really interesting performance wise and this isn't a production tool :-)

I'm not sure I'd be so confident about tensorflow.js being so fast on the CPU ... you can see a marked difference in the backends httpss://www.tensorflow.org/js/guide/platform_environment

Re: Open-Source Virtual Background

#26

Post by sneak »

gmueckl wrote: post #19
peer2pay wrote: post #16

Since it looks like your webcam is mounted to a stationary PC and you're only really writing this for yourself, wouldn't it be a lot easier to just subtract out a static picture of your background from the feed?

This breaks down in daylight (brightness shifts due to sun moving/clouds) or whenever anything shifts ever so slightly. Even shadows may make things appear where they shouldn't. Chroma keying is the stable version of this idea: the single background color of a separately and well lit background removes these issues.

https://www.amazon.com/Webaround-Portable-Webcam-Background-...

Re: Open-Source Virtual Background

#28

Post by BenTheElder »

sneak wrote: post #26
gmueckl wrote: post #19

Earlier quoted context omitted.

This breaks down in daylight (brightness shifts due to sun moving/clouds) or whenever anything shifts ever so slightly. Even shadows may make things appear where they shouldn't. Chroma keying is the stable version of this idea: the single background color of a separately and well lit background removes these issues.

https://www.amazon.com/Webaround-Portable-Webcam-Background-...

I'm in a 1 bedroom apartment with my fiance who is also work from home now, there isn't really space to put a physical green screen behind me.

If you have the space, and good lighting it's a much simpler approach.

It's also less fun though, and I could do this with what I had on hand pretty quickly.

Re: Open-Source Virtual Background

#30

Post by BenTheElder »

sigwinch28 wrote: post #25

You mention the ~10FPS performance. It seems like moving all that data backwards and forwards between Python and Node might be a bottleneck, no?

I need to profile it more closely, I actually don't remember the exact FPS etc. but I don't expect that to be the limiting factor.

The inference / ml is expensive (which I did profile initially...), and I suspect not really optimized on this backend. It appears to be faster with webGL in the browser.

I sorta stopped worrying about it once it was "good enough" to show up to a few meetings with, but with all the attention I'll probably take another look.

It does look like someone ported bodypix to python, I'll probably try that next.

https://github.com/ajaichemmanam/simple_bodypix_python

Post reply on HN