Show HN: Brawshot – Basic temporal denoise for videos in BRAW format • Hacker News Board

Show HN: Brawshot – Basic temporal denoise for videos in BRAW format

#1

Post by an-unknown » Sat, May 18, 2024, 10:47 PM UTC

I wanted to record the aurora last weekend, but I only have a Blackmagic Design video camera which is clearly not made for this purpose. Recording a video of the night sky results in extreme noise to the point that you don't really see anything, so I wrote a tool to significantly reduce noise in such video recordings. Essentially it computes a moving average across video frames which significantly reduces the random sensor noise. This works because aurora changes very slowly, and it's roughly comparable to a long exposure time computed out of a video file where the individual frames have a very short exposure time. But unlike a photo camera with long exposure time, this produces a video at full frame rate again. The window size of the moving average has no influence on the computation time, so even large window sizes of e.g. 100 frames or more are no problem.

If you want to use this tool for artistic purposes, it will produce extreme motion blur depending on the window size you choose.

I am aware that tools like ffmpeg or the paid version of DaVinci Resolve have denoising features, but at least ffmpeg's denoising filters are extremely slow and memory intensive, and it's fun to implement this as fast GPU accelerated open source software for this non-standard use case anyway.

To use this tool, you'll need a Blackmagic Design camera which records in BRAW format (or you could hack the video decoder in the main.cpp file to decode some other source video file format).

If someone has a good idea how to remove the remaining noise pattern which seems to be inherent in the image sensor and very slowly changes over time, I'd be very interested!

Show HN: Brawshot – Basic temporal denoise for videos in BRAW format
github.com

Re: Show HN: Brawshot – Basic temporal denoise for videos in BRAW format

#2

Post by macnetic » Sun, May 19, 2024, 4:27 PM UTC

You should look up image sensor dark noise. One way to reduce dark noise is taking an exposure without any light on the sensor, it should give you the fixed pattern which can be subtracted from your scene image. Dark noise is sensitive to temperature and exposure time, so you should try to do this under similar conditions.

Re: Show HN: Brawshot – Basic temporal denoise for videos in BRAW format

#3

Post by mgraczyk » Sun, May 19, 2024, 7:58 PM UTC

Looks like the output is significantly darker than the input. This can happen if you don't unbias the noise near the black level. You may want to experiment with adjusting the noise estimate taking into account the black level

Re: Show HN: Brawshot – Basic temporal denoise for videos in BRAW format

#4

Post by sorenjan » Sun, May 19, 2024, 8:32 PM UTC

Try using the median or mode instead. Averaging multiple data points only works well when the data distribution is symmetrical around the true value, but the closer you get to the endpoints (black and white) the less true that assumption is. If a pixel's true value is 5, and you get some samples with values 0 to 10, and one with value 100 that outlier will not get compensated by an outlier in the opposite direction.

You could also average just the middle quantiles or similar, if you still want to average over several frames.

Re: Show HN: Brawshot – Basic temporal denoise for videos in BRAW format

#5

Post by an-unknown » Sun, May 19, 2024, 8:42 PM UTC

sorenjan wrote: post #4

Try using the median or mode instead. Averaging multiple data points only works well when the data distribution is symmetrical around the true value, but the closer you get to the endpoints (black and white) the less true that assumption is. If a pixel's true value is 5, and you get some samples with values 0 to 10, and one with value 100 that outlier will not get compensated by an outlier in the opposite direction.…

You are indeed right. I'm not sure how to efficiently implement a median here though, because a single frame is around 120MB large already. I assume I could/should compute the median over only a few frames to limit memory consumption and feed the result into the moving (arithmetic) average?

I'm also not entirely sure if computing the average of the non-linear "raw" sensor data which is what I did so far is a good or a bad idea / how it interacts at the endpoints of the value range.

Re: Show HN: Brawshot – Basic temporal denoise for videos in BRAW format

#6

Post by sorenjan » Sun, May 19, 2024, 9:41 PM UTC

an-unknown wrote: post #5

sorenjan wrote: post #4

Try using the median or mode instead. Averaging multiple data points only works well when the data distribution is symmetrical around the true value, but the closer you get to the endpoints (black and white) the less true that assumption is. If a pixel's true value is 5, and you get some samples with values 0 to 10, and one with value 100 that outlier will not get compensated by an outlier in the opposite direction.…

You are indeed right. I'm not sure how to efficiently implement a median here though, because a single frame is around 120MB large already. I assume I could/should compute the median over only a few frames to limit memory consumption and feed the result into the moving (arithmetic) average? I'm also not entirely sure if computing the average of the non-linear "raw" sensor data which is what I did so far is a good or…

I haven't done any image filtering a a good while, so I'm pretty much just speculating here. But maybe you can work on patches instead of the whole frames to save memory, since it's not a real time application anyway? I think a moving truncated mean would be better than an average of medians. Or maybe you can do one pass over the video and calculate the mean and variance for each pixel over a moving window, and then in the next pass filter out pixel values that are more than one standard deviation (or whatever you choose) from the current estimated mean and average the remaining values.

Doing linear combinations of values that aren't linear does indeed sound problematic, values from one end will have a disproportional influence over the result, although maybe it's not a big deal since the same pixel in neighboring frames will probably only vary slightly in an approximately locally linear fashion.

Re: Show HN: Brawshot – Basic temporal denoise for videos in BRAW format

#7

Post by an-unknown » Mon, May 20, 2024, 1:43 AM UTC

macnetic wrote: post #2

You should look up image sensor dark noise. One way to reduce dark noise is taking an exposure without any light on the sensor, it should give you the fixed pattern which can be subtracted from your scene image. Dark noise is sensitive to temperature and exposure time, so you should try to do this under similar conditions.

I implemented it just now. Turns out when performing the subtraction on the raw data before applying the color space conversion LUT, it's also necessary to add the mean of the noise image's pixel values because otherwise some bias is lost and the entire image's brightness changes. With this in place, it seems to work. Interestingly enough, this even results in noticeable noise reduction when the noise image was not recorded in the same environment but only with the same camera settings!

When performing the subtraction after applying the LUT (that is, result = apply_lut(raw) - apply_lut(raw_noise) instead of result = apply_lut(raw - raw_noise + mean(raw_noise))), the result is quite different for reasons that I don't fully understand yet, but the noise is reduced in the same way. This will need some further investigation.