Show HN: Neural Image Compression Demo

Hi everyone, I've been working on an implementation of a model for learnable image compression together with general support for neural image compression in PyTorch. You can try it out directly and compress your own images in Google Colab [1] or checkout the source on Github [2].

This project is based on the paper "High-Fidelity Image Compression" by Mentzer et. al. [3] - this was one of the most interesting papers I've read this year! The model is capable of compressing images of arbitrary size and resolution to bitrates competitive with state-of-the-art compression methods while maintaining a very high perceptual quality. At a high-level, the model jointly trains an autoencoding architecture together with a GAN-like component to encourage faithful reconstructions, combined with a hierarchical probability model to perform the entropy coding.

What's interesting is that the model avoids compression artifacts associated with standard image codecs by subsampling high-frequency detail in the image while preserving the global features of the image very well - for example, the model learns to sacrifice faithful reconstruction of e.g. faces and writing and use these 'bits' in other places to keep the overall bitrate low.

The overall model is around 700MB - so transmitting the model wouldn't be particularly feasible, and the idea is that both the sender and receiver have access to the model, and can transmit the compressed messages between themselves.

If you have any questions or notice something weird I'd be more than happy to address them.

---

[1] Colab Demo: https://colab.research.google.com/github/Justin-Tan/high-fid...

[2]: Github: https://github.com/Justin-Tan/high-fidelity-generative-compr...

[3]: Original paper: https://hific.github.io/

[4]: Sample reconstructions: https://github.com/Justin-Tan/high-fidelity-generative-compr...

Not an compression expert, but my eyes have been trained to ignore color gradient issues and minor pixelation as long as the outline of the shapes is clearly defined. This approach while doing better job on preserving detail in colors and avoids pixelation, it distorts significantly the shapes themselves (see the clock on the last example). It makes the images seem like google map 3D renders of shorts. How finely can you tune the target compression ratio ? Maybe with a less aggressive target these would not be that evident ?

I would be interested in a comparision with a neural JPEG decoder that tries to restore the original image as much as possible.

It's incredibly hard to change the default file format on the web, but there's an opportunity to switch libjpeg to a decoder with much more realistic output images.

During training, you can set a target bitrate by heavily penalizing examples which exceed the target rate in the rate-distortion objective - so the model should learn to produce compressed representations at or below this bitrate. However, this constraint is only enforced on aggregate throughout the entire dataset - like many ML systems, there is no guarantee of behaviour for individual examples, either within or outside the training set. Despite this, the model appears to respect the target rate well, even on out-of-sample images.

One shortcoming is that this current model is non-adaptive - which means that the target rate is fixed. So to achieve different target compression rates you would have to train multiple models in different rate regimes. In the Colab demo there is the option to select between 3 different models trained with a target bits-per-pixel (bpp) rate at 0.14bpp, 0.30bpp, and 0.45bpp, respectively - higher rates correspond to more higher-fidelity reconstructions, at the expense of a lower compression ratio. The default is the `HiFIC-med` model (and this is what the all samples in the README were generated with), but the model trained at the highest bitrate should have less obvious imperfections.

There's also an aspect to the distortion that can be attributed to the entropy coding process rather than the model itself - currently the system clips values outside a certain probability range, resulting in artificial distortion - a fix is in the pipeline though.

What was the error? I tried to make the demo notebook as robust as possible - you should be able to execute all cells in sequence once then execute cells out of sequence etc. without trouble, but it's hard to legislate for errors in Jupyter-like notebooks sometimes.