Live data from Hacker News

Lepton image compression: saving 22% losslessly from images at 15MB/s

blogs.dropbox.com

81–90 of 196 posts

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#81
post #52

Earlier quoted context omitted.

Can you run those losslessly?

https://trac.ffmpeg.org/wiki/Encode/H.264#LosslessH.264

While there's a lossless H.264, it doesn't fit the implied use-case proposed in the thread. Using lossless H.264 to recompress lossy MPEG-2 would be akin to using PNG to recompress a JPEG -- silly and wasteful.

Sure, it's an inexact analogy because PNG is not a block-based DCT coder, but all lossless H.264 does is set the quantizer as absurdly high as it needs to go to losslessly encode a particular frame. Staring from a compressed source, this is not a good recipe for achieving a space-saving result.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#84

I'm interested in a 'super lossy' (deep learning based?) compression. You should be able to compress movies down to a screenplay and a few stage directions. A picture should of course be worth 1000 words.

Google Photos already has an incredibly lossy compression, right? Images organized under Dog, Cat, etc. :) Might take a bit more effort to get the extra 999 words.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#85
post #13
post #11

That's nothing compared to LenPEG 3 http://www.dangermouse.net/esoteric/lenpeg.html >For the standard test image the new LenPEG 3 compresses the image so efficiently that data storage space is actually freed on the computer right up to the entire capacity of the storage devices

don't get this one, where is the code and how can I try it out? if it's so good I would assume dropbox have found it instead of doing lepton.

LMAO - yes, Dropbox, please run LenPEG compression on your production env!

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#86
post #21

This is very impressive for archiving images. For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG. If someone can do this over video files that will be PiedPiper comes into real life.

The reason this can be done with JPEGs is the compression hasn't been updated. There are folks that have used h264 for compressing images and webp uses vp8 compression, both with much better results than JPEG. Lepton is cool because it helps make existing technology a whole lot better, but what we actually need is a better image format. You wouldn't see the same leap for videos because people have been working hard t…

All that is true, but it misses out a key point: Lepton is risk-free.

I could re-encode all of my JPEG photos with a better codec, but the problem is that then they have gone through a lossy compression scheme twice: once with JPEG, and once with the new codec. This could ruin the image quality on some photos, giving them nasty artefacts.

The newer codecs might even be hindered further by the fact that they are being asked to compress an image that's been JPEG'd. I don't think any of the codecs you mentioned are tuned to work their best with image data that already has been butchered by JPEG. They expect to be given the 'pure' original image. This may well cause the codec to perform worse than expected.

Using Lepton gives none of these risks, since jpeglepton is lossless. I could throw away all the JPEGs afterwards, safe in the knowledge that if I want to go back, I can.

Also, it's not really true that JPEG has been left to rot. There are lots of programs available that greatly improve upon the basic JPEG compression, while still outputting a JPEG file that any compliant decoder can read. And there's an even simpler way to shrink your photo file sizes - just drop the JPEG quality level slightly. Most cameras and programs default to a very high quality value. Dropping the default by even a tiny amount can produce remarkably smaller files, and you'll probably never notice the miniscule image quality loss.

I suspect that JPEG is so popular because it is 'good enough'. Most people and programs don't care so much about squeezing out smaller files, so they'll keep using JPEG regardless.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#87

Earlier quoted context omitted.

Then why do you admire them? Would you also admire them if you were their investor? Dropbox management is obligated by law to act in the best interests of their shareholders, i.e. to make them as much profit as possible. It's more likely that they have released it because of some profit-seeking interest. They are not charity.

I admire them because they certainly considered the pros and cons of doing so, and decided that they feel secure enough in their market position and prefer to give this back to the open source community Pretty much every modern company safes tremendous amounts of money from open source (Linux and upwards in the stack), and so the OSS community should rightly be considered a STAKEholder. The share vs stakeholder obses…

There's also the upside of attracting other talented developers to come work at Dropbox if Lepton is representative of the kind of project they might be working on.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#88

Earlier quoted context omitted.

The reason this can be done with JPEGs is the compression hasn't been updated. There are folks that have used h264 for compressing images and webp uses vp8 compression, both with much better results than JPEG. Lepton is cool because it helps make existing technology a whole lot better, but what we actually need is a better image format. You wouldn't see the same leap for videos because people have been working hard t…

Is JPEG, PNG, etc a container format that could use arbitrary compression, like MKV? Or do we need a brand new format.

The JPEG file format is pretty detailed, there are lots of places in the file where you could indicate that the image has been compressed with a different algorithm. But it wouldn't really help, because older decompressors will still not be able to interpret the new codec. They would be able to parse the file and read the metadata in it - like image width and height, timestamps, camera details, and so on, but there's no way it can decompress the data.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#89
post #39

Could someone explain to me how compression algorithms work? Shouldn't they not work very consistently, by pigeonhole principle?

They work because while most possible data is unstructured, most real-world data is highly structured. On average over all possible inputs, any compression scheme must have 1x compression. Consider images of slides for a presentation that are text on a flat background. If you know the value of the pixel just to the left of the current pixel, then if you guess that the current pixel will be the same, you will be right…

On average, any lossless compression scheme must have >= 1x compression. Lossy compression schemes can have less (since they're discarding information), and lossless compression schemes certainly can average out to > 1x (this is trivially observable by taking any 1x-average scheme and adding a fixed-length header to the output).

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#90

I really admire Dropox for open sourcing this, it shows their commitment. Saving almost a quarter of space for most images stored is something that truly gives a competitive edge. (I say most because people probably primarily have JPEG images). Especially considering how many images are probably stored on services like Dropbox. And they just gave it away to their competitors.

IMO, it shows how they don't plan to make any money from this, don't consider it a large competitive advantage, and think that the largest advantage of open sourcing it are the 'free' open source volunteers will improve the software.

It's interesting that they don't consider it a large enough competitive advantage.

Either that or they are using this to attract engineers.

Post reply on HN