Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lepton image compression: saving 22% losslessly from images at 15MB/s

blogs.dropbox.com

Post reply on HN
Print view View original

71–80 of 196 posts

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#71

Post by niftich »

are595 wrote: post #55

Interesting that they are using VP8 to compress the JPEGs. This is one degree of separation away from Google's WebP [1]. It would be interesting to see how they stack up (WebP has a lossy and lossless mode). [1]. https://developers.google.com/speed/webp/

(EDIT: removed wording that Lepton produces files that conform to the JPEG spec. It doesn't. It losslessly compresses into a custom format, that losslessly decompresses into a JPEG)

Lepton uses the arithmetic coder [1] from VP8. Using arithmetic coding instead of Huffman encoding to get better compression was always an option in JPEG, but it has been historically avoided due to patents [2].

Compared to VP8-Intra, the compression used in lossy WebP, JPEG is missing the prediction step, usually called 'filtering' [3], which is the single largest contributor of WebP's compression outperforming JPEG.

Reading through the Lepton blog post, it seems they're using a different method of prediction, based on observations about typical gradients and correlations between AC and DC coefficients. VP8 uses a more 'traditional' approach of predicting your neighboring pixels, which was borne out of run-length encoding, but also very applicable to video's moving macroblocks. A comparison would indeed be enlightening.

[1] https://tools.ietf.org/html/rfc6386#section-7

[2] https://en.wikipedia.org/wiki/Arithmetic_coding#US_patents

[3] https://medium.com/@duhroach/how-webp-works-lossly-mode-33bd...

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#72

Post by sliverstorm »

aidenn0 wrote: post #44
the_duke wrote: post #22

Earlier quoted context omitted.

I wonder if they use a rolling checksum too, to avoid duplicating a complete file if only a view bytes shifted (for example adding a line of text in the beginning of a file) The backup tool bup ( https://github.com/bup/bup ) does this.

They almost certainly do not, mostly because of how slow doing so is.

It probably wouldn't hit the most important cases either, dedup is typically most powerful & valuable on large media files, software packages, disk ISO's, and the like which do not frequently have arbitrary text inserted at the start of the file!

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#73

Post by guelo »

colechristensen wrote: post #30
ausjke wrote: post #21

This is very impressive for archiving images. For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG. If someone can do this over video files that will be PiedPiper comes into real life.

The reason this can be done with JPEGs is the compression hasn't been updated. There are folks that have used h264 for compressing images and webp uses vp8 compression, both with much better results than JPEG. Lepton is cool because it helps make existing technology a whole lot better, but what we actually need is a better image format. You wouldn't see the same leap for videos because people have been working hard t…

Indeed. In one project where I converted jpegs to webp I saw around 70% savings at similar quality. This was on Android where it's fully supported. On the web pretty much only Chrome supports it, I'm not sure why Firefox, IE and Safari are hesitant.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#74

Post by Veratyr »

ausjke wrote: post #21

This is very impressive for archiving images. For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG. If someone can do this over video files that will be PiedPiper comes into real life.

I'm running over several terabytes and on the 5k images I've done so far (3.6GB), the compression has been 0.78x, which is damn good for lossless compression!

EDIT: 17GB now (my server is kinda slow) and it's still holding at 0.78x.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#75

Post by jsingleton »

This looks very useful for archiving, but as others have pointed out, less useful for web development. I did lots of research on image compression for a book recently and found quite a few helpful tools.

jpeg-archive [^1] is designed for long term storage and you can still serve the images over the web. imageflow [^2] has just been kickstarted and looks really promising for use with ASP.NET Core.

mozjpeg is also showing progress and if FLIF takes off then that will be great. Scalable images would be fantastic. No more resizing and all the security issues that brings [^3].

[^1]: https://github.com/danielgtaylor/jpeg-archive

[^2]: https://www.imageflow.io

[^3]: https://imagetragick.com

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#76

Post by noahkim11 »

daniel_rh wrote: post #5
mbreese wrote: post #3

So, am I to read this as to mean that when I send Dropbox a JPEG, they are behind the scenes further compressing it using Lepton? Then when I request it back, they are re-converting it back to JPEG?

That's the idea behind the algorithm, yes. And since it's lossless, every original bit is preserved. The same idea could be applied on the Desktop client instead of on the server, which would save 22% of the bandwidth as well and make syncing faster.

Any word on when that would be implemented?

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#77

Post by halomru »

sliverstorm wrote: post #70
colechristensen wrote: post #30

Earlier quoted context omitted.

The reason this can be done with JPEGs is the compression hasn't been updated. There are folks that have used h264 for compressing images and webp uses vp8 compression, both with much better results than JPEG. Lepton is cool because it helps make existing technology a whole lot better, but what we actually need is a better image format. You wouldn't see the same leap for videos because people have been working hard t…

Is JPEG, PNG, etc a container format that could use arbitrary compression, like MKV? Or do we need a brand new format.

PNG is a very nice, extensible container format. In principle you can use it to store arbitrary image data, and still make use of all established metadata fields.

In practice you probably shouldn't do that without changing the file extension and the magic bytes to avoid confusion among users and poorly written software.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#78

Post by antisthenes »

tedunangst wrote: post #65
dublinben wrote: post #60

Earlier quoted context omitted.

Why bother trying to losslessly compress your DVDs when they're already a low quality, compressed MPEG2 source? The small amount of content that isn't available in higher quality won't take up that much space left as is.

This is why I come to HN. To find out what's in my video archive and how easy it is to replace.

In the case you're being facetious, maybe you could clarify how big is your video archive exactly and what sort of compression you're trying to achieve on it?

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#79

Post by jszymborski »

SapphireSun wrote: post #66

This is amazing, we've been struggling with JPG storage and fast delivery at my lab (terabytes and petabytes of microscopy images). We'll be running tests and giving this a shot!

jpg is a weird format to be storing microscopy images, no? Usually end up in some sort of bitmap TIFF (or their Zeiss/etc. proprietary format) from what I've seen.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#80

Post by Sinaf »

guelo wrote: post #73
colechristensen wrote: post #30

Earlier quoted context omitted.

The reason this can be done with JPEGs is the compression hasn't been updated. There are folks that have used h264 for compressing images and webp uses vp8 compression, both with much better results than JPEG. Lepton is cool because it helps make existing technology a whole lot better, but what we actually need is a better image format. You wouldn't see the same leap for videos because people have been working hard t…

Indeed. In one project where I converted jpegs to webp I saw around 70% savings at similar quality. This was on Android where it's fully supported. On the web pretty much only Chrome supports it, I'm not sure why Firefox, IE and Safari are hesitant.

This [1] is from 2013, so I don't know how it holds up, but I found it in the criticism section of the Wikipedia article for webp [2].

edit: This [3] is a follow-up from 2014 to the article/study from 2014.

TL;DR: "We consider this study to be inconclusive when it comes to the question of whether WebP and/or JPEG XR outperform JPEG by any significant margin. We are not rejecting the possibility of including support for any format in this study on the basis of the study’s results. We will continue to evaluate the formats by other means and will take any feedback we receive from these results into account."

[1]https://blog.mozilla.org/research/2013/10/17/studying-lossy-...

[2]https://en.wikipedia.org/wiki/WebP#Criticism

[3]https://blog.mozilla.org/research/2014/07/15/mozilla-advance...

Post reply on HN