Lepton image compression: saving 22% losslessly from images at 15MB/s • Hacker News Board

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#151

Post by bnolsen » Fri, Jul 15, 2016, 4:19 AM UTC

SapphireSun wrote: post #122

jszymborski wrote: post #79

Earlier quoted context omitted.

jpg is a weird format to be storing microscopy images, no? Usually end up in some sort of bitmap TIFF (or their Zeiss/etc. proprietary format) from what I've seen.

So the weird thing is that we're not only a lab, but also a web tool. We have the files backed up in a standard format in one place, but delivering 1024x1024x128 cubes of images over the internet has been tricky. We don't need people to always view them at full fidelity, just good enough. We tried JPEG2000, which was better quality per a file size, but the web worker decoder was slower than the JPEG one adding second…

Yeah jpeg2k sucks. It doesn't seem to do anything very well. Design by committee ruined it by making it waay to complex.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#152

Post by mrcactu5 » Fri, Jul 15, 2016, 5:07 AM UTC

are there public data sets of images the scale of 15MB/s ?

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#153

Post by Stanleyc23 » Fri, Jul 15, 2016, 5:28 AM UTC

sapphireblue wrote: post #141

Stanleyc23 wrote: post #134

Earlier quoted context omitted.

Very cool! if anyone who comes across this particular thread knows about papers/research being written about this topic, I'd be very interested to learn more.

There is an award-winning compressor that uses many statistical models (and a 3-layered dense neural network) to compress the data losslessly: http://www.byronknoll.com/cmix.html Also there is a free book that contains descriptions of various common and exotic compression formats http://www.mattmahoney.net/dc/dce.html Overall I think we are yet to see the full potential of deep learning unleashed on data compression.…

ah thanks. really appreciate the lead. do you yourself happen to have much experience in the space of machine learning or compression?

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#154

Post by raverbashing » Fri, Jul 15, 2016, 6:36 AM UTC

Very interesting. But it's not clear to me how estimating the DC components makes it guaranteed lossless

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#155

Post by SapphireSun » Fri, Jul 15, 2016, 6:40 AM UTC

manigandham wrote: post #137

SapphireSun wrote: post #123

Earlier quoted context omitted.

Thanks for the info, see my response below.

Store it as FLIF then dynamically serve up whatever is needed for the web interface...

We're storing them as RAWs for our backups, but it's cheaper and faster to store them compressed and transmit them without a transformation step. I'll keep FLIF in mind, but it looks kind of unstable reading the website?

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#156

Post by PepeGomez » Fri, Jul 15, 2016, 6:55 AM UTC

daniel_rh wrote: post #26

ausjke wrote: post #21

This is very impressive for archiving images. For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG. If someone can do this over video files that will be PiedPiper comes into real life.

Thanks for testing this! For archival purposes, be sure to check the exit code of the lepton binary after compressing each JPEG: the default parameters only support a subset of JPEGs (you need to pass -allowprogressive and -memory=2048M -threadmemory=256M to support a wider variety of large or progressive JPEG files). Lepton will not write images that it is unable to compress using the settings provided. In those cas…

How much better do you think it could get with JPEGs encoded with Lepton in mind?

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#157

Post by nachtigall » Fri, Jul 15, 2016, 7:39 AM UTC

I would be interested why C++ was chosen for https://github.com/dropbox/lepton instead of Rust?

Given the recent usage of Rust for the implementation of Brotli compression (https://blogs.dropbox.com/tech/2016/06/lossless-compression-...) and that it's used for data storage (http://www.wired.com/2016/03/epic-story-dropboxs-exodus-amaz...) this somewhat surprises me.

The reasons given for Rust as stated in https://blogs.dropbox.com/tech/2016/06/lossless-compression-... would seem valid here too: > For Dropbox, any decompressor must exhibit three properties: > > 1. it must be safe and secure, even against bytes crafted by modified or hostile clients, > 2. it must be deterministic—the same bytes must result in the same output, > 3. it must be fast.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#158

Post by djmips » Fri, Jul 15, 2016, 8:24 AM UTC

fake-name wrote: post #127

Am I the only person who spent the first paragraph or so very confused why dropbox was messing with thermal cameras? The FLIR Lepton is thermal imaging sensor ( http://www.flir.com/cores/content/?id=66257 ), and has been for a few years now. They really should have used a different name.

just because it messed with your own personal namespace - you know that lepton is a particle and not a thermal camera right. Just because camera marketers got to you first. The real problem is polluting using useful words to market things in the first place.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#159

Post by therealmarv » Fri, Jul 15, 2016, 8:39 AM UTC

Strom wrote: post #114

therealmarv wrote: post #112

I want this working seamlessly on the file system. That means: I see JPGs but they are lepton compressed in reality. Would be a great use case for existing file servers. How could this be (theoretically) achieved on a linux machine?

One way would be to write a FUSE [1] that does this. [1] https://en.wikipedia.org/wiki/Filesystem_in_Userspace

Now I get very interested in writing a FUSE for that http://www.cs.nmsu.edu/~pfeiffer/fuse-tutorial/

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#160

Post by daniel_rh » Fri, Jul 15, 2016, 8:40 AM UTC

nachtigall wrote: post #157

I would be interested why C++ was chosen for https://github.com/dropbox/lepton instead of Rust? Given the recent usage of Rust for the implementation of Brotli compression ( https://blogs.dropbox.com/tech/2016/06/lossless-compression-... ) and that it's used for data storage ( http://www.wired.com/2016/03/epic-story-dropboxs-exodus-amaz... ) this somewhat surprises me. The reasons given for Rust as stated in https://…

SIMD support in Rust is still very early and required unsafe mode for SSE, and lepton makes heavy use of SSE intrinsics.

Now I do see http://huonw.github.io/simd/simd/ was being developed in August 2015, but it seems to be gathering dust of late.

I really do wish that Rust would provide nice alignment guarantees (eg 32 byte) without depending on customizing the allocator, and builtin, safe, SIMD instructions