Restoration of defocused and blurred images (2012)

Strangely enough, the author only mentions total variation denoising in passing as a feature of SmartBlur. I would say this method is one of the most common, especially when your image has sharp transitions and lots of solid regions of color (e.g. pictures of buildings). I wrote what is effectively one of two of the fastest TV denoising algorithms and implementations out there: https://github.com/tansey/gfl

The way to use it in image processing is to basically think of each pixel as a point on a graph, with edges between its adjacent pixels. Then you apply a least squares penalty with an additional regularization penalty on the absolute value of the difference between neighboring pixels. The result is regions of constant pixel color.

I have tried the referenced implementation (SmartDeblur) on a few (similar) slightly de-focused images. From memory, I had the artefacts without any noticeable improvement. I don't think I even saved the results. YMMV and I would almost certainly try it on photographs with a larger blur than mine (approximately a disc kernel with radius of 0.8px). Since then I've been working on other methods with mixed results.

I've found a few things out: Firstly, many articles mention some form of Richardson Lucy with regularization as if de-convolution with a known Point Spread Function is a solved problem. The regularization can indeed produce better results, but usually introduces new parameters. I've found, with real photographs, I have had far better results using less statistically sound methods. This is especially true as I've found that RL, more often than not, begins to diverge quite early.

The reality is that there are papers surrounding deconvolution and blind deconvolution that get apparently good results and work poorly in practice. It is only once implemented that the flaws of the algorithms proposed are exposed. This is compounded by the fact that we have a wonderful error metric with this inverse problem: the euclidean distance between 1. the solution convolved with the point-spread function used for deconvolution; and 2. the original (blurred) image. With many algorithms, lowering the euclidean distance decreases the perceived quality by introducing more artefacts. Nonetheless, I have only seen a handful of papers that include this metric when considering a proposed regularisation scheme, and this is one of the only times when this particular metric makes any sense in computer vision -- if your deconvolution is working, and your PSF is correct (which it is in many cases as the blur is synthetic) then the euclidean distance is the measure you want.

I have since switched to least squares using L-BFGS-B for fitting which has lowered again the Euclidean distance and produced the best, most natural-looking results by far on real photographic images. Unfortunately, estimating the PSF is difficult for small out-of-focus blurs. Optics applied to an ideal lens would suggest a disc kernel, but at this scale other factors are playing a part.

I'll probably do a writeup of this when I'm done -- though I don't know when that will be. I'll also be putting any code on Github in due course. In the meantime if anybody has any pointers or is interested in further details or source code please grab my email address from my profile.

Are there security implications? If the last example was a blurred out license key or address for instance, this technique might be able to restore it.

How does an upscaler in your TV work? How does it create information out of nothing?

Imagine that you are learning a deep neural network on a huge amount of movies. You have access to all of the lovely Hollywood movies. You downscale them to a 480p resolution, and then try to learn a deep neural network to upscale the thing, maybe upscaling only 16x16 blocks of the image.

It works amazingly well, and looks like magic.

Maybe there was no visibility of pores on the face in the 480p downscale, but your model can learn to reproduce them faithfully.

Sony has access to billions of movie frames in extremely large resolutions. Their engineers are definitely using this large amount of information to create statistical filters which upscale your non-hd, or maybe your HD to 4k HD. These filters work better than deterministic methods in this article. Why? Because the filters know much more about the distribution of the source (distribution of values of each individual pixel). They have exact information that one instead tries to assume (author in the article assumed that something in the source - be it noise or something else - behaves according the to Gaussian). If you know how to find the proper distribution, instead of assuming it, you can move closer to the information theoretical limits.

Just imagine how fast these filters can be if you put them on an FPGA, it also explains why TV sometimes cost more than $2k.

If you knew that your images would only contain car registration plates, you could definitely learn a filter that would be very precise in reconstructing the image when zoomed, you'd now find CSI zooming a little bit more realistic :D

In focus-blurred images, the blur spreads the information about certain pixels throughout a region of the image. (Objects appear larger and fainter because of this when they are blurred. It's why you can shoot through a mesh-wire fence with a wide lens aperture, for example, and not see the fence.) This 'spread out' information is being used to reconstruct an approximation of the original image.

However, when images are downscaled, all the information from 'subpixels' is kept within the reduced-resolution-image-pixel, and all replaced with a single RGB value. There is no 'spread out' information from those pixels. The 'blur' only applies to a region with sharp edges that exactly coincide with the unit of information. So, none of the data from the sub-pixels remain. To reconstruct the pixels that were within you have to essentially guess based on the context of the surrounding pixels.