Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lepton image compression: saving 22% losslessly from images at 15MB/s

blogs.dropbox.com

Post reply on HN
Print view View original

121–130 of 196 posts

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#121

Post by tetrep »

randyrand wrote: post #90
the_duke wrote: post #20

I really admire Dropox for open sourcing this, it shows their commitment. Saving almost a quarter of space for most images stored is something that truly gives a competitive edge. (I say most because people probably primarily have JPEG images). Especially considering how many images are probably stored on services like Dropbox. And they just gave it away to their competitors.

IMO, it shows how they don't plan to make any money from this, don't consider it a large competitive advantage, and think that the largest advantage of open sourcing it are the 'free' open source volunteers will improve the software. It's interesting that they don't consider it a large enough competitive advantage. Either that or they are using this to attract engineers.

> Either that or they are using this to attract engineers.

On a related note (I can't speak for Dropbox specifically) there are many engineers who desire their work to be open source for their own motivations, and when it's not a significant business risk to do so, nice companies will allow it.

So it works on two fronts, as a "hey, we'll let you open source your stuff" and a "hey, we've got people here who care about and contribute to open source."

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#122

Post by SapphireSun »

jszymborski wrote: post #79
SapphireSun wrote: post #66

This is amazing, we've been struggling with JPG storage and fast delivery at my lab (terabytes and petabytes of microscopy images). We'll be running tests and giving this a shot!

jpg is a weird format to be storing microscopy images, no? Usually end up in some sort of bitmap TIFF (or their Zeiss/etc. proprietary format) from what I've seen.

So the weird thing is that we're not only a lab, but also a web tool. We have the files backed up in a standard format in one place, but delivering 1024x1024x128 cubes of images over the internet has been tricky. We don't need people to always view them at full fidelity, just good enough.

We tried JPEG2000, which was better quality per a file size, but the web worker decoder was slower than the JPEG one adding seconds to the total download/decode time.

EDIT: We're currently doing 256x256x256 (equivalent to a 4k image) on eyewire.org. We're speeding things up to handle bigger 3D images.

EDIT2: If you check out Eyewire right now, you might notice some slowdown when you load cubes, that's because we're decoding on the main thread. We'll be changing that up next week.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#123

Post by SapphireSun »

manigandham wrote: post #118
SapphireSun wrote: post #66

This is amazing, we've been struggling with JPG storage and fast delivery at my lab (terabytes and petabytes of microscopy images). We'll be running tests and giving this a shot!

Why not use a better format to begin with? Like FLIF: http://flif.info/

Thanks for the info, see my response below.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#124

Post by SapphireSun »

anc84 wrote: post #117
SapphireSun wrote: post #66

This is amazing, we've been struggling with JPG storage and fast delivery at my lab (terabytes and petabytes of microscopy images). We'll be running tests and giving this a shot!

JPEG2000 might be a nice choice, it is often used in scientific environments.

Thanks for the info! See my comment above.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#125

Post by versteegen »

placeybordeaux wrote: post #33
jacobsladder wrote: post #24

Earlier quoted context omitted.

Then why do you admire them? Would you also admire them if you were their investor? Dropbox management is obligated by law to act in the best interests of their shareholders, i.e. to make them as much profit as possible. It's more likely that they have released it because of some profit-seeking interest. They are not charity.

They are a private company, they can actually ask their shareholders how they wish them to act.

Does it make a difference? Can't public companies do the same at shareholder meetings, or declare what the intentions of the company are before going public? For (a poor) example, Google declared at IPO they would never pay dividends.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#126

Post by Stanleyc23 »

Just a random idea: could machine learning algorithms that do object recognition help to improve the compression of images or videos? Maybe a lossy algorithm could compress away "irrelevant" things. This way a high resolution frame might have lower resolution objects inside but it would he ok because the important part of the content is preserved.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#127

Post by fake-name »

Am I the only person who spent the first paragraph or so very confused why dropbox was messing with thermal cameras?

The FLIR Lepton is thermal imaging sensor (http://www.flir.com/cores/content/?id=66257), and has been for a few years now. They really should have used a different name.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#128

Post by rattray »

manigandham wrote: post #119
colechristensen wrote: post #30

Earlier quoted context omitted.

The reason this can be done with JPEGs is the compression hasn't been updated. There are folks that have used h264 for compressing images and webp uses vp8 compression, both with much better results than JPEG. Lepton is cool because it helps make existing technology a whole lot better, but what we actually need is a better image format. You wouldn't see the same leap for videos because people have been working hard t…

Yes, it should all just switch to FLIF http://flif.info/

> FLIF is a novel lossless image format which outperforms PNG, lossless WebP, lossless BPG, lossless JPEG2000, and lossless JPEG XR in terms of compression ratio.

> 74% smaller than lossless JPEG XR compression.

> Works on any kind of image

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#129

Post by Florin_Andrei »

Stanleyc23 wrote: post #126

Just a random idea: could machine learning algorithms that do object recognition help to improve the compression of images or videos? Maybe a lossy algorithm could compress away "irrelevant" things. This way a high resolution frame might have lower resolution objects inside but it would he ok because the important part of the content is preserved.

I think a fully trained neural network is, in a sense, a "lossy" (possibly very lossy) description of the training set.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#130

Post by eon1 »

e12e wrote: post #120
jszymborski wrote: post #79

Earlier quoted context omitted.

jpg is a weird format to be storing microscopy images, no? Usually end up in some sort of bitmap TIFF (or their Zeiss/etc. proprietary format) from what I've seen.

The size difference between "high quality" jpeg and lossless compressed TIFFs can be quite significant - so I it might not be feasible to use eg. TIFF for archiving. And jpegs might very well be "good enough". Eg. just tested on a small 10MP Sony ARW raw image - the raw file is 7MB, the camera jpeg is 2.3MB, and a tiff compressed with LZW is 20MB (uncompressed 29MB). The raw tiff run through lzma is 9.3MB. But either…

How on earth did a 7MB raw blow out to a 20MB TIFF? Isn't that going from uncompressed RGBG to losslessly compressed RGB?
Post reply on HN