Lepton image compression: saving 22% losslessly from images at 15MB/s • Hacker News Board

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#171

Post by animal531 » Fri, Jul 15, 2016, 12:50 PM UTC

halomru wrote: post #77

sliverstorm wrote: post #70

Earlier quoted context omitted.

Is JPEG, PNG, etc a container format that could use arbitrary compression, like MKV? Or do we need a brand new format.

PNG is a very nice, extensible container format. In principle you can use it to store arbitrary image data, and still make use of all established metadata fields. In practice you probably shouldn't do that without changing the file extension and the magic bytes to avoid confusion among users and poorly written software.

If I remember correctly PNG can use different compression algorithms per scanline, although I doubt it's ever used as such.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#172

Post by iopq » Fri, Jul 15, 2016, 1:02 PM UTC

colechristensen wrote: post #30

ausjke wrote: post #21

This is very impressive for archiving images. For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG. If someone can do this over video files that will be PiedPiper comes into real life.

The reason this can be done with JPEGs is the compression hasn't been updated. There are folks that have used h264 for compressing images and webp uses vp8 compression, both with much better results than JPEG. Lepton is cool because it helps make existing technology a whole lot better, but what we actually need is a better image format. You wouldn't see the same leap for videos because people have been working hard t…

Actually, high quality JPEGs are just as good as BPG, VP8, etc:

https://people.xiph.org/~jm/daala/revisiting/subset1_psnrhvs...

so when you use a better encoder like mozjpeg, JPEG is actually competitive with the newest formats, given enough quality

it gets slightly grainier results with blocking artifacts, but BPG and WebM look like they had a painting algorithm blur out all of the details

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#173

Post by S3curityPlu5 » Fri, Jul 15, 2016, 1:02 PM UTC

wheres the code?

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#174

Post by prirun » Fri, Jul 15, 2016, 1:20 PM UTC

mahyarm wrote: post #105

randyrand wrote: post #90

Earlier quoted context omitted.

IMO, it shows how they don't plan to make any money from this, don't consider it a large competitive advantage, and think that the largest advantage of open sourcing it are the 'free' open source volunteers will improve the software. It's interesting that they don't consider it a large enough competitive advantage. Either that or they are using this to attract engineers.

You also open source if it's in your best interest for something to become the 'standard'. In google there is a regret that they didn't open source a lot of stuff, because it ends up being reproduced outside of google in some form. The open source companies have an advantage in hiring and overall advancement of their product by using the open source version whatever their internal version was. You also see it in face…

Without taking anything away from their generous announcement, it's very much in their interest for .lep files to become a standard so that they do not have to transcode to jpeg when users are viewing files on DropBox. I'm happy they released it.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#175

Post by are595 » Fri, Jul 15, 2016, 2:28 PM UTC

SapphireSun wrote: post #155

manigandham wrote: post #137

Earlier quoted context omitted.

Store it as FLIF then dynamically serve up whatever is needed for the web interface...

We're storing them as RAWs for our backups, but it's cheaper and faster to store them compressed and transmit them without a transformation step. I'll keep FLIF in mind, but it looks kind of unstable reading the website?

Yes, the FLIF format is still being fully fleshed out. I would actually give WebP a shot instead of JPG. It offers better compression and already has native implementations in Chrome and on Android.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#176

Post by joelg236 » Fri, Jul 15, 2016, 2:55 PM UTC

daniel_rh wrote: post #160

nachtigall wrote: post #157

I would be interested why C++ was chosen for https://github.com/dropbox/lepton instead of Rust? Given the recent usage of Rust for the implementation of Brotli compression ( https://blogs.dropbox.com/tech/2016/06/lossless-compression-... ) and that it's used for data storage ( http://www.wired.com/2016/03/epic-story-dropboxs-exodus-amaz... ) this somewhat surprises me. The reasons given for Rust as stated in https://…

SIMD support in Rust is still very early and required unsafe mode for SSE, and lepton makes heavy use of SSE intrinsics. Now I do see http://huonw.github.io/simd/simd/ was being developed in August 2015, but it seems to be gathering dust of late. I really do wish that Rust would provide nice alignment guarantees (eg 32 byte) without depending on customizing the allocator, and builtin, safe, SIMD instructions

> I really do wish that Rust would provide nice alignment guarantees (eg 32 byte) without depending on customizing the allocator, and builtin, safe, SIMD instructions

Same could be said about C/C++. I'm guessing the answer is much simpler: the author(s) are comfortable with C++. And they probably don't deploy much rust code yet.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#177

Post by ptspts » Fri, Jul 15, 2016, 3:13 PM UTC

Portability notes for the Lepton implementation (https://github.com/dropbox/lepton):

* Implemented in C++ (-std=c++0x and -std=c++11 work, -std=c++98 doesn't work).

* Needs a recent g++ to compile (g++-4.8 works, g++-4.4 doesn't work).

* Runs on Linux and Windows.

* Runs on i386 (-m32) and amd64 (-m64) architectures. Doesn't work on other architectures, because it uses SSE4.1 instructions.

* Can be compiled without autotools (http://ptspts.blogspot.ch/2016/07/how-to-compile-lepton-jpeg...).

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#178

Post by odbol_ » Fri, Jul 15, 2016, 4:04 PM UTC

ausjke wrote: post #21

This is very impressive for archiving images. For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG. If someone can do this over video files that will be PiedPiper comes into real life.

It's not really "lossless" if it only works with a lossy format though, right?

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#179

Post by are595 » Fri, Jul 15, 2016, 4:08 PM UTC

jacobolus wrote: post #135

rattray wrote: post #128

Earlier quoted context omitted.

> FLIF is a novel lossless image format which outperforms PNG, lossless WebP, lossless BPG, lossless JPEG2000, and lossless JPEG XR in terms of compression ratio. > 74% smaller than lossless JPEG XR compression. > Works on any kind of image

> outperforms PNG, lossless WebP, lossless BPG, lossless JPEG2000, and lossless JPEG XR PNG was not designed to be used for photograph-like images. The rest of those were not designed to be lossless formats, the lossless version is just a tacked-on afterthought. Very unsurprising to find a codec that can beat those.

In my own tests, I found FLIF generally beats PNG (PNG Crush/Optipng both in brute-force mode) for comics (greyscale, majority white) as well, but by a less significant margin.

It's also worth noting that there aren't many other lossless formats, so it's still a valid comparison. I'm sure neither TIFF nor RAW outperform FLIF either.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#180

Post by mark-r » Fri, Jul 15, 2016, 6:59 PM UTC

odbol_ wrote: post #178

ausjke wrote: post #21

This is very impressive for archiving images. For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG. If someone can do this over video files that will be PiedPiper comes into real life.

It's not really "lossless" if it only works with a lossy format though, right?

Yes and no. It's not a lossless way to encode an image, but the Lepton step itself doesn't add any loss that wasn't already present in the JPEG to begin with, and it generates a smaller file overall.