Bokehlicious Selfies
rahulrav.com
Bokehlicious Selfies
1–10 of 53 posts
Re: Bokehlicious Selfies
#2This is a quick proof of concept of an idea. This won't work for every use-case but it worked surprisingly well for a use-case I had in mind. Happy to answer any questions.
Re: Bokehlicious Selfies
#3This is a quick proof of concept of an idea. This won't work for every use-case but it worked surprisingly well for a use-case I had in mind. Happy to answer any questions.
So, is the ML part just the segmentation, correct?
Re: Bokehlicious Selfies
#4This is really cool, but it isn't bokeh, it's just blur.
Re: Bokehlicious Selfies
#5It would be nice if there was some more information about where the mask comes from. We want a segmentation map for people, so this technique basically takes the layer activation map for the "person" class - which is id 0 in COCO - and you can threshold that to get foreground/background. If you changed the mask index, this would respond to other object types (so as written this code will only work for humans).
--
By the way, Google does some absolutely nuts stuff with this on the Pixel 3 and 4 - they actually calculate a stereo depth map using the two autofocus sites in individual pixels. Essentially some modern CMOS sensors use a technology called dual pixel autofocus (DPAF), by measuring the response from two photodiodes in the same pixel, you adjust focus until each pixel has the same intensity (more or less). If the camera is out of focus, the two photodiodes will have different intensities.
However what this gives you is two separate images with an extremely small (but detectable) parallax which can be used to give coarse 3D reconstruction, and you can segment foreground and background. It's nice because you get a strong physical prior, rather than having to worry about using a convnet to identify fore/background regions. (They of course apply a convnet anyway to refine the result).
https://ai.googleblog.com/2018/11/learning-to-predict-depth-...
https://ai.googleblog.com/2019/12/improvements-to-portrait-m...
Re: Bokehlicious Selfies
#6Awesome! Computational photography is fascinating. A while back, I wrote some OpenGL shaders to do the same thing, and got stuck when I wanted variable bokeh based on scene depth. This is easy when you have the depth map (say, in a renderer). I recall reading about how Apple achieved this using stereo disparity mapping, which is great if you have the hardware. I hadn't considered a neural net approach like yours for auto-generating a depth map, although I now that I look into it, Facebook recently published research on just that!
https://ai.facebook.com/blog/-powered-by-ai-turning-any-2d-p...
Re: Bokehlicious Selfies
#7This is really cool, but it isn't bokeh, it's just blur.
It’s a nice proof of concept though. I think that further applying the second part of the article (masking) to lighter parts of the image at a different intensity could provide a more accurate simulation.
Edit: I really have no idea what I’m talking about. I’m just guessing it can definitely be improved upon. I probably should’ve stopped at it’s a nice proof of concept (like most free content) :)
Re: Bokehlicious Selfies
#8It would be nice if there was some more information about where the mask comes from. We want a segmentation map for people, so this technique basically takes the layer activation map for the "person" class - which is id 0 in COCO - and you can threshold that to get foreground/background. If you changed the mask index, this would respond to other object types (so as written this code will only work for humans). -- By…
It’s a simple Gaussian kernel multiplied with a triangular mask.
You can do better by playing with intensities of pixel values as suggested in the article I linked.
Re: Bokehlicious Selfies
#9This is really cool, but it isn't bokeh, it's just blur.
Yes. That is correct. Just wanted to see if the idea in my head actually worked. I was surprised by how decent it turned out especially given that I did not do anything “clever”.
Re: Bokehlicious Selfies
#10It would be nice if there was some more information about where the mask comes from. We want a segmentation map for people, so this technique basically takes the layer activation map for the "person" class - which is id 0 in COCO - and you can threshold that to get foreground/background. If you changed the mask index, this would respond to other object types (so as written this code will only work for humans). -- By…
It’s a simple Gaussian kernel multiplied with a triangular mask. You can do better by playing with intensities of pixel values as suggested in the article I linked.
I meant about where this line comes from:
mask = masks[0][0]
Also recent related reading: https://bartwronski.com/2020/03/15/using-jax-numpy-and-optim...
HN Discussion: https://news.ycombinator.com/item?id=22590360&ref=hvper.com&...