Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lepton image compression: saving 22% losslessly from images at 15MB/s

blogs.dropbox.com

Post reply on HN
Print view View original

61–70 of 196 posts

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#61

Post by CJefferson »

personjerry wrote: post #27

Could someone explain to me how compression algorithms work? Shouldn't they not work very consistently, by pigeonhole principle?

No compression algorithm works on completely random data. Fortunately (for compressors) there is in practice almost no completely random data sets people care about in the world. Almost all data is corrolated in some way. In pictures you can do a good job of predicting the colours of neighbouring pixels (they will, most of the time, be a similar colour).

In this case they are making use of the fact that JPEGs store a certain mathematical formulation of a picture. It turns out JPEG doesn't store that mathematical formulation very well, so you can squash it loselessly into a better formulation, then later turn it back into the original JPEG.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#62

Post by are595 »

daniel_rh wrote: post #26
ausjke wrote: post #21

This is very impressive for archiving images. For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG. If someone can do this over video files that will be PiedPiper comes into real life.

Thanks for testing this! For archival purposes, be sure to check the exit code of the lepton binary after compressing each JPEG: the default parameters only support a subset of JPEGs (you need to pass -allowprogressive and -memory=2048M -threadmemory=256M to support a wider variety of large or progressive JPEG files). Lepton will not write images that it is unable to compress using the settings provided. In those cas…

Hi Daniel, very cool project! I am interested in hearing your thoughts on alternative image formats / compression methods.

WebP is very promising (also based on VP8) for lossless and lossy compression. Have you considered using it to compress PNGs in the same way Lepton is compressing JPGs? Odds are it wouldn't be bit-perfect though (despite being pixel-perfect).

Also interested in hearing about the tradeoffs between server-side decoding and client-side. Not to keep focusing on it, but WebP has native support in Chrome and javascript decoders for everything else.

I think it is a very exciting time for image formats with several promising new ones on the way (WebP, FLIF, maybe BPG but possible legal issues).

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#63

Post by ryanlm »

ausjke wrote: post #21

This is very impressive for archiving images. For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG. If someone can do this over video files that will be PiedPiper comes into real life.

Someone just had to mention that _stupid_ show. Seriously. By empowering that show, you're just making fun of yourself and all the other programmers on here.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#64

Post by lj3 »

Jabbles wrote: post #49
BrunoJo wrote: post #45

Earlier quoted context omitted.

Space is cheap but the bandwidth isn't. At Backblaze the download traffic costs $0.05/GB = $50k/PB.

I don't think they're compressing this and decompressing it client side. At least I didn't get that impression from the article. You're correct ofc, download costs = 10 months of storage.

From the article:

Lepton can decompress significantly faster than line-speed for typical consumer and business connections. Lepton is a fully streamable format, meaning the decompression can be applied to any file as that file is being transferred over the network. Hence, streaming overlaps the computational work of the decompression with the file transfer itself, hiding latency from the user.

In my humble opinion, dropbox's image gallery webapp is considerably faster than any other I've seen, especially when compared to imgur.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#65

Post by tedunangst »

dublinben wrote: post #60
neuronexmachina wrote: post #32

Earlier quoted context omitted.

I'd love to have something like this for archiving DVD ISOs though, where the VOBs are compressed with old-school MPEG.

Why bother trying to losslessly compress your DVDs when they're already a low quality, compressed MPEG2 source? The small amount of content that isn't available in higher quality won't take up that much space left as is.

This is why I come to HN. To find out what's in my video archive and how easy it is to replace.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#68

Post by daniel_rh »

GrantS wrote: post #53
daniel_rh wrote: post #28

Earlier quoted context omitted.

It's very easy to make a JPEG file that will not compress at all. Luckily it might look like snow from a television set rather than a typical image produced by a camera. We live in a world where it is common for blue sky to occupy a portion of the frame and green grass to occupy another portion of the frame. Since images captured of our world exhibit repetition and patterns, there are opportunities for lossless compr…

Impressive work, Daniel! Do I understand correctly that any image prediction for which the deltas are smaller in absolute value than the full JPEG/DCT coefficients would offer continued compression benefits? As in, if you could "name that tune" to predict the rest of the entire image from the first few pixels, the rest of the image would be stored for close to free (and if not, it essentially falls back to regular JP…

Hi GrantS: That's pretty close to how it works. We always use the new prediction, even where it's worse than JPEG though (very rarely), to stay consistent.

As for having the mega-model that predicts all images better: well it turns out with the lepton model out you only lose a few tenths of a percent by training the model from scratch on each images individually. We have a test case for training a global model in the archive (it's https://github.com/dropbox/lepton/blob/master/src/lepton/tes... ) That trains the "perfect" lepton model on the current image then uses that same model to compress the image (It's not meant to be a fair test, but it gives us a best-case scenario for potential gains from a model that has been trained from a lot of images) and in this case it doesn't gain much, even in a controlled situation like the test suite.

However the idea you mention here may still be a good idea for a hypothetical model--but we haven't identified that model yet.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#69

Post by matznerd »

rhaps0dy wrote: post #46
XorNot wrote: post #23

Earlier quoted context omitted.

They will once Silicon Valley takes them to task for it. Just like season 1 pretty much wiped "making the changes world a better place" from the lingo of SV companies.

> season 1 pretty much wiped "making the changes world a better place" from the lingo of SV companies Did it really?

Yes, the show is making the world a better place.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#70

Post by sliverstorm »

colechristensen wrote: post #30
ausjke wrote: post #21

This is very impressive for archiving images. For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG. If someone can do this over video files that will be PiedPiper comes into real life.

The reason this can be done with JPEGs is the compression hasn't been updated. There are folks that have used h264 for compressing images and webp uses vp8 compression, both with much better results than JPEG. Lepton is cool because it helps make existing technology a whole lot better, but what we actually need is a better image format. You wouldn't see the same leap for videos because people have been working hard t…

Is JPEG, PNG, etc a container format that could use arbitrary compression, like MKV? Or do we need a brand new format.
Post reply on HN