Live data from Hacker News

Lytro Unlock – Making a bad camera slightly better

github.com

71–74 of 74 posts

Re: Lytro Unlock – Making a bad camera slightly better

#71
post #59

Earlier quoted context omitted.

What's wild is that it turns out we were only a few years away from technology that could completely replace the purpose of the cinema camera, with off the shelf GPUs on a normal desktop (NeRFs)

NeRFs are cool and someday will do what the Lytro cinema camera claimed. But today it’s nowhere close. See how Lytro took 10 years to go from lab science to a cinema quality camera. NeRFs have a similar journey ahead of them. Today they kinda mostly sorta work in the lab.

Also just from a fidelity perspective it seems better to fully capture all the light rays than to capture part of the scene and try to recompute it later.

Re: Lytro Unlock – Making a bad camera slightly better

#72
post #32

Earlier quoted context omitted.

"just performing predictable mathematical transformations" Just linear algebra, to be precise. A lens is just a matrix and rays are simply vectors. Combining lenses is merely matrix multiplication and complex optical systems can be represented by the result, which is a plain simple matrix again. The Eigenvalues tell you interesting properties of your optical system. I've always enjoyed this part of physics because it…

Going on a metaphysical tangent, it is a bit weird that A LOT of physical processes can be modelled with linear algebra, and don't require something way more advanced ...

I think it’s more that we developed linear algebra that made us approximate physical processes by linear models. If you look close enough, almost nothing is linear.

For example, we happily draw a linear scale on a mercury thermometer, even though we know that not to be correct, even if we incorrectly assume that the coefficient of expansion of mercury is independent of its temperature. Also, try explaining why the conversions between Celsius, Fahrenheit and Kelvin are all linear (answer for Kelvin and Celsius: they technically only are since 2019, when we redefined them (https://en.wikipedia.org/wiki/2019_redefinition_of_the_SI_ba...). I think the Fahrenheit scale was redefined as a shift on the Kelvin one around the same time)

Re: Lytro Unlock – Making a bad camera slightly better

#73
post #59

Earlier quoted context omitted.

NeRFs are cool and someday will do what the Lytro cinema camera claimed. But today it’s nowhere close. See how Lytro took 10 years to go from lab science to a cinema quality camera. NeRFs have a similar journey ahead of them. Today they kinda mostly sorta work in the lab.

Also just from a fidelity perspective it seems better to fully capture all the light rays than to capture part of the scene and try to recompute it later.

Indeed. NeRFs are impressive from an interpolation standpoint, but they can't magically beat Nyquist - any component in the radiance field that with a higher frequency than the sampling rate will be missed, and probably aliased. In practical terms, this means things like highly specular reflections, "glittery" or "twinkly" surfaces, and magnifying optics - anything which could "project" an obvious pattern onto the capture plane.

An example would be the surface of a swimming pool - reflection of light off a swimming pool will project a pattern like this [0] onto the nearest wall, and if that's where your camera positions are they are effectively sampling that image - there is no data for what happens in between. The resulting NeRF reconstruction will look dull and flat, with the water taking on a matte or satin texture.

Another example would be a laser beam - all the cameras see is a weak translucent line, if they see anything at all, and you cannot infer that you will receive a blinding amount of light if you put your eye right in the beam unless you happen to know about lasers. Even if you do know about lasers, if the laser itself is not in shot, you cannot determine which way the beam is going.

Assuming that nothing in the scene is emitting perversely high-frequency signals like a laser, you could in theory make a guess at some of this information by inferring a lot more about the scene - calculating the position of lights, computing normal maps, guessing that certain sets of points are made from the same material and therefore likely have the same BSDF. But current NeRFs don't do any of that, they just run volume rendering on a set of emissive points.

[0] https://www.google.com/search?q=swimming+pool+reflection

Re: Lytro Unlock – Making a bad camera slightly better

#74

Could you construct a NeRF from a single Lytro image?

In all likelihood yes, but it wouldn't do what you likely want. NeRF is better for interpolation than extrapolation, because it can't confabulate things that were occluded in all of the input data. The main lens of the Lytro is small, so it's basically like having all your input images confined to a circle a few inches big. As soon as you tried to reconstruct anything outside that, you'd see gaps.

It might be useful for boosting its mediocre resolution, though.

Post reply on HN