Earlier quoted context omitted.
What's wild is that it turns out we were only a few years away from technology that could completely replace the purpose of the cinema camera, with off the shelf GPUs on a normal desktop (NeRFs)
NeRFs are cool and someday will do what the Lytro cinema camera claimed. But today it’s nowhere close. See how Lytro took 10 years to go from lab science to a cinema quality camera. NeRFs have a similar journey ahead of them. Today they kinda mostly sorta work in the lab.
Lytro Unlock – Making a bad camera slightly better
71–74 of 74 posts
Re: Lytro Unlock – Making a bad camera slightly better
#72Earlier quoted context omitted.
"just performing predictable mathematical transformations" Just linear algebra, to be precise. A lens is just a matrix and rays are simply vectors. Combining lenses is merely matrix multiplication and complex optical systems can be represented by the result, which is a plain simple matrix again. The Eigenvalues tell you interesting properties of your optical system. I've always enjoyed this part of physics because it…
Going on a metaphysical tangent, it is a bit weird that A LOT of physical processes can be modelled with linear algebra, and don't require something way more advanced ...
For example, we happily draw a linear scale on a mercury thermometer, even though we know that not to be correct, even if we incorrectly assume that the coefficient of expansion of mercury is independent of its temperature. Also, try explaining why the conversions between Celsius, Fahrenheit and Kelvin are all linear (answer for Kelvin and Celsius: they technically only are since 2019, when we redefined them (https://en.wikipedia.org/wiki/2019_redefinition_of_the_SI_ba...). I think the Fahrenheit scale was redefined as a shift on the Kelvin one around the same time)
Re: Lytro Unlock – Making a bad camera slightly better
#73Earlier quoted context omitted.
NeRFs are cool and someday will do what the Lytro cinema camera claimed. But today it’s nowhere close. See how Lytro took 10 years to go from lab science to a cinema quality camera. NeRFs have a similar journey ahead of them. Today they kinda mostly sorta work in the lab.
Also just from a fidelity perspective it seems better to fully capture all the light rays than to capture part of the scene and try to recompute it later.
An example would be the surface of a swimming pool - reflection of light off a swimming pool will project a pattern like this [0] onto the nearest wall, and if that's where your camera positions are they are effectively sampling that image - there is no data for what happens in between. The resulting NeRF reconstruction will look dull and flat, with the water taking on a matte or satin texture.
Another example would be a laser beam - all the cameras see is a weak translucent line, if they see anything at all, and you cannot infer that you will receive a blinding amount of light if you put your eye right in the beam unless you happen to know about lasers. Even if you do know about lasers, if the laser itself is not in shot, you cannot determine which way the beam is going.
Assuming that nothing in the scene is emitting perversely high-frequency signals like a laser, you could in theory make a guess at some of this information by inferring a lot more about the scene - calculating the position of lights, computing normal maps, guessing that certain sets of points are made from the same material and therefore likely have the same BSDF. But current NeRFs don't do any of that, they just run volume rendering on a set of emissive points.
[0] https://www.google.com/search?q=swimming+pool+reflection
Re: Lytro Unlock – Making a bad camera slightly better
#74Could you construct a NeRF from a single Lytro image?
It might be useful for boosting its mediocre resolution, though.