Live data from Hacker News

Unbounded High Dynamic Range Photography Using a Modulo Camera

media.mit.edu

21–30 of 98 posts

Re: Unbounded High Dynamic Range Photography Using a Modulo Camera

#21

Looks to be more related to exposure than saturation but still, it's an interesting new take on dynamic range. I'm more a hobbyist than an expert but current tone mapping techniques (at least anything mostly automated) can often leave you with "halos" around objects where the software has feathered the edges between lighter and darker areas. Then the article mentions the issues with using multiple shots for tone mapp…

Halos are a problem, but they can mostly be removed with good post-processing. Note they're not a fundamental aspect of tone mapping. Tone mapping is just whatever algorithm you chose to compress the HDR information into a low dynamic range we have for displays. The great thing about this camera compared to multi-exposure HDR (apart from the halo effect you mentioned) is you can actually take pictures of moving objects or even make HDR movies (the temporal variation should actually help with better phase unwrapping). That would be amazing for the film industry I think.

Re: Unbounded High Dynamic Range Photography Using a Modulo Camera

#22

Earlier quoted context omitted.

Exactly. This is huge, and I was wondering why people didn't do it already. Or, for example, why don't cameras currently take multiple snapshots of the sensor? For example, if I'm exposing for 2 seconds, read the sensor at 0.1 sec (for the very bright lights) and again at 2 seconds, for the darker parts. That way, I can make an HDR image without having to take another photo. Is reading the sensor destructive to the d…

Readout of a CCD is destructive. There are types of sensors that have non-destructive readout (Charge Injection Devices being one example: https://www.cis.rit.edu/research/CID/a_cid_is.htm ).

A sensor with per-pixel reset to zero on readout would seem to be ideal — then you could get cumulative counts without wasting valuable photon landing space on the die.

Re: Unbounded High Dynamic Range Photography Using a Modulo Camera

#23

> No more will photographers or even ordinary people have to fumble with aperture size and exposure length. This is a "computer scientists" understanding of photography, and this phrase alone can even be seen as "dangerous" by photographers. There's more to aperture size and exposure length than "how much light reaches the sensor", like focus, depth of field, motion blur and bokeh, to name a few, that is not aknowled…

> is this implementation very different from having a 16-bit ADC in normal sensors

16-bit ADCs increases tonal granularity in the readout, which can help in the darker tones. A sensor site may be capable of differentiating about 12 bits worth of meaningful information, hence there's a good argument for overshooting that in the ADC. Our eyes, even if not sensitive to more than 8 bits worth of tonal gradation, are perceiving in a logarithmic space (vs. the sensor recording in a linear space), hence extra information in the shadows can be quite good. Plus the extra bits give more room in which to tune a captured image. An article such as http://www.scantips.com/basics14.html gives a rundown on this.

The extra bits don't help if the incoming light exceeds the sensor's recording capacity, hence the interest of this "modulo" discussion for over-exposed images.

Re: Unbounded High Dynamic Range Photography Using a Modulo Camera

#24

> No more will photographers or even ordinary people have to fumble with aperture size and exposure length. This is a "computer scientists" understanding of photography, and this phrase alone can even be seen as "dangerous" by photographers. There's more to aperture size and exposure length than "how much light reaches the sensor", like focus, depth of field, motion blur and bokeh, to name a few, that is not aknowled…

It sounds from the implementation summary that it's equivalent to a higher bit-depth ADC, but that the higher-value bits are inferred by software somehow. I don't think it would properly recover an image with a dark field with the exception of a single pixel with a value of 10^7 above the dark noise.

What would truly solve the dynamic-range problem is a logarithmic readout CCD.

(Thanks for the Foveon pointer; it's an elegant approach to color.)

Re: Unbounded High Dynamic Range Photography Using a Modulo Camera

#25
post #20
post #11

Earlier quoted context omitted.

Sure, photographs will still control aperture and exposure time — this just removes a constraint at one end: if you want shallow depth of field and motion blur, you could use this method (and get extra bits) rather than an ND filter.

That's just really a completely misguided statement. No mater how much you fumble with aperture and shutter speed there's simply no way to expand the camera sensor dynamic range without taking multiple exposures. Besides I have never seen "ordinary people" fumble with aperture size and exposure length so that's a solved problem. Sometimes it gives "wrong" results, like exposing for the highlights rather than the shad…

  > there's simply no way to expand the camera sensor dynamic range without taking multiple exposures
That's the point of this paper†. They expand dynamic range by having the detectors wrap around (discarding high bits) and then recover the high bits computationally.

† here http://web.media.mit.edu/~hangzhao/modulo.html (linked by putterson in another comment)

Re: Unbounded High Dynamic Range Photography Using a Modulo Camera

#26

> No more will photographers or even ordinary people have to fumble with aperture size and exposure length. This is a "computer scientists" understanding of photography, and this phrase alone can even be seen as "dangerous" by photographers. There's more to aperture size and exposure length than "how much light reaches the sensor", like focus, depth of field, motion blur and bokeh, to name a few, that is not aknowled…

The Foveon sensor is different - the claimed benefit there has to do with stacked silicon and lack of debayering, and the benefit is having better resolving power with the same pixel count. Sony is now researching similar stacked chips so we'll probably see this idea return.

The innovation with the tech in the article is in the dynamic range, current CMOS sensors have a fairly limited maximum dynamic range because of the well saturation described in the article, usually around 12-14ev DR. For high contrast situations that isn't enough, and photographers will definitely appreciate increased dynamic range, not to mention the other applications they listed, like astronomy.

On the display side most displays also have fairly limited DR and cannot reproduce the dynamic range in HDR images, so we end up having to tonemap images to get the perceived dynamic range back - however I think HDR monitors will break through in the next couple of years as many HDR TVs are hitting the market this year.

Re: Unbounded High Dynamic Range Photography Using a Modulo Camera

#27
post #25
post #20

Earlier quoted context omitted.

That's just really a completely misguided statement. No mater how much you fumble with aperture and shutter speed there's simply no way to expand the camera sensor dynamic range without taking multiple exposures. Besides I have never seen "ordinary people" fumble with aperture size and exposure length so that's a solved problem. Sometimes it gives "wrong" results, like exposing for the highlights rather than the shad…

> there's simply no way to expand the camera sensor dynamic range without taking multiple exposures That's the point of this paper†. They expand dynamic range by having the detectors wrap around (discarding high bits) and then recover the high bits computationally. † here http://web.media.mit.edu/~hangzhao/modulo.html (linked by putterson in another comment)

I know, I just don't see how it solves this issue. You couldn't solve the issue before "fumbling with aperture size and exposure length"

The only thing I would consider that solves "fumble with aperture size and exposure length" would be huge ISO range (both ways) without any noise.

> No more will photographers or even ordinary people have to fumble with aperture size and exposure length.

Re: Unbounded High Dynamic Range Photography Using a Modulo Camera

#28
Here's an idea, instead of separately tracking the number of "resets" we just add those bits on to the left hand side of the measurement. We could call it... having more bits in the ADC. Expose for the highlights and then pull more detail out of the shadows with your extra bits of ADC precision.

There are not 1 million ADC units in a 1-megapixel camera, there's a few ADCs (a Canon 7D has 2 image processors with 4 ADCs each) that are iterated across the CCD sites to progressively read them out. To make this new sensor, you need to cram 50 million sets (50mp is state of the art) of voltage comparator/charge reset/digital counter circuits onto the CCD, and for N bits of rollover accuracy they need to be able to trigger, erase the charge, and resume exposure at least N times during the exposure time E (which is say 1/500th of a second, since they're complaining about movement during multiple HDR exposures). Practically speaking the trigger/operation/resume period must be significantly less than E/N since I see no way to retain the exposure during the period when the charge well is being drained to zero. If this time is non-trivial, that translates to losing the fine bits of your ADC accuracy.

In their image they compare a 13-stop exposure (the current state of the art) to an 8-stop exposure (state of the art in 1900). So assuming they didn't just pull a photoshop out of their ass, they are overall claiming a 5 stop increase from state of the art. That's 5 extra bits of recovery (1 stop is double the range, i.e. an extra bit), so they think they can trigger the reset circuit 5 times during an exposure. That implies this circuit must have a minimum cycle time of 1/2500th of a second (practically speaking a lot less to give time for the actual exposure). With this math I'm also assuming that the comparator is perfect and doesn't lose any accuracy - variance in trigger threshold or trigger time translates to losing some of your accuracy again. They also need to generate little enough waste heat to avoid hot pixels (this is a major reason the ADC is on the image processor rather than the CCD), and you need 50 million sets of these on the CCD.

If you can do it then go for it. It's a nice idea on paper but I think there's a lot of physical obstacles to overcome. If it were that easy someone would have done it already. It's especially difficult given that reading out the sensor is destructive - measuring it wipes the charge, so I'm not sure how any of this would work at all given that.

Now what would actually be interesting is to apply the image-processing techniques to dual-DR imaging, as they mention in their "related works" section. The open-source Magic Lantern firmware for Canon DSLRs allows you to scan alternating rows at different ISOs, so effectively you can capture a lot more dynamic range at the expense of your vertical resolution. It's all volunteers and they probably haven't applied all the fancy image-processing magic with the convolutions and the hippity-hop rap music. How about reconstructing those overexposed lines instead? Or working on some of the sensors with physical implementations of dual-DR?

Re: Unbounded High Dynamic Range Photography Using a Modulo Camera

#29
post #12

The title here is terrible because over-saturation is an effect of poor tone mapping, which is inevitable if you want to display an HDR image in a low dynamic range display. An accurate title would be the end of over- exposed images, but why not just keep the original?

Two different senses of the word ‘saturation’. The subtitle of the article (current title here) is unfortunate.

Yes. I expected to see an article about stock images being overused across lots of publications, and whatever the fix for that is.

Re: Unbounded High Dynamic Range Photography Using a Modulo Camera

#30
post #26

> No more will photographers or even ordinary people have to fumble with aperture size and exposure length. This is a "computer scientists" understanding of photography, and this phrase alone can even be seen as "dangerous" by photographers. There's more to aperture size and exposure length than "how much light reaches the sensor", like focus, depth of field, motion blur and bokeh, to name a few, that is not aknowled…

The Foveon sensor is different - the claimed benefit there has to do with stacked silicon and lack of debayering, and the benefit is having better resolving power with the same pixel count. Sony is now researching similar stacked chips so we'll probably see this idea return. The innovation with the tech in the article is in the dynamic range, current CMOS sensors have a fairly limited maximum dynamic range because of…

Worth noting that those "HDR" televisions are still only Rec709 which is limited to about 6 stops of dynamic range. Sure some are approaching rec2020 color gamut, but we're still cramming all that gamma down into 6-9 stops. Even a consumer DSLR can shoot 12.

I work in post and have moved our workflow over to the Academy Color Encoding Specification (ACES) which supports up to 27 stops (more than the eye can see which is about 24 best case scenario). ACES allows us to dynamically change our output to rec709, rec2020, P3 etc and while 2020 has better colors, it's definitely still a highly compressed image (in terms of dynamic range).

Post reply on HN