Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lepton image compression: saving 22% losslessly from images at 15MB/s

blogs.dropbox.com

Post reply on HN
Print view View original

31–40 of 196 posts

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#31

Post by Jabbles »

It's interesting to do a cost analysis here:

It's saved "multiple petabytes" of space.

Backblaze storage is $0.005/GB/Month = $5k/PB/Month.

The GitHub repo has 7 authors, perhaps costing Dropbox $200k/year each and taking most of a year ~ $1M to develop this system.

So this might pay for itself after 200PB*Months, assuming Dropbox's storage costs are the same as Backblaze's prices, and assuming CPU time is free. (TODO: estimate CPU costs...)

Of course, advancing the state of the art has intrinsic advantages, but again, it's interesting to look at the purely financial point.

https://www.backblaze.com/b2/cloud-storage-pricing.html

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#32

Post by neuronexmachina »

colechristensen wrote: post #30
ausjke wrote: post #21

This is very impressive for archiving images. For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG. If someone can do this over video files that will be PiedPiper comes into real life.

The reason this can be done with JPEGs is the compression hasn't been updated. There are folks that have used h264 for compressing images and webp uses vp8 compression, both with much better results than JPEG. Lepton is cool because it helps make existing technology a whole lot better, but what we actually need is a better image format. You wouldn't see the same leap for videos because people have been working hard t…

I'd love to have something like this for archiving DVD ISOs though, where the VOBs are compressed with old-school MPEG.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#33

Post by placeybordeaux »

jacobsladder wrote: post #24
the_duke wrote: post #20

I really admire Dropox for open sourcing this, it shows their commitment. Saving almost a quarter of space for most images stored is something that truly gives a competitive edge. (I say most because people probably primarily have JPEG images). Especially considering how many images are probably stored on services like Dropbox. And they just gave it away to their competitors.

Then why do you admire them? Would you also admire them if you were their investor? Dropbox management is obligated by law to act in the best interests of their shareholders, i.e. to make them as much profit as possible. It's more likely that they have released it because of some profit-seeking interest. They are not charity.

They are a private company, they can actually ask their shareholders how they wish them to act.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#34

Post by verytrivial »

daniel_rh wrote: post #28
personjerry wrote: post #27

Could someone explain to me how compression algorithms work? Shouldn't they not work very consistently, by pigeonhole principle?

It's very easy to make a JPEG file that will not compress at all. Luckily it might look like snow from a television set rather than a typical image produced by a camera. We live in a world where it is common for blue sky to occupy a portion of the frame and green grass to occupy another portion of the frame. Since images captured of our world exhibit repetition and patterns, there are opportunities for lossless compr…

I think you are explaining how both lossy and lossless compression work without explaining how this does it differently.

I can't read the article due to technical constraints, but understand that e.g. JPEG has a lossy quantisation pass followed by a lossless encoding/compression pass of the result of the first stage. If they're reproducing bit-identical result to the input JPEG, it must be a (very good) optimisation of the latter stage. [How'd I do?]

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#35

Post by placeybordeaux »

Jabbles wrote: post #31

It's interesting to do a cost analysis here: It's saved "multiple petabytes" of space. Backblaze storage is $0.005/GB/Month = $5k/PB/Month. The GitHub repo has 7 authors, perhaps costing Dropbox $200k/year each and taking most of a year ~ $1M to develop this system. So this might pay for itself after 200PB*Months, assuming Dropbox's storage costs are the same as Backblaze's prices, and assuming CPU time is free. (TOD…

I only see 3 authors.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#36

Post by Jabbles »

placeybordeaux wrote: post #35
Jabbles wrote: post #31

It's interesting to do a cost analysis here: It's saved "multiple petabytes" of space. Backblaze storage is $0.005/GB/Month = $5k/PB/Month. The GitHub repo has 7 authors, perhaps costing Dropbox $200k/year each and taking most of a year ~ $1M to develop this system. So this might pay for itself after 200PB*Months, assuming Dropbox's storage costs are the same as Backblaze's prices, and assuming CPU time is free. (TOD…

I only see 3 authors.

https://github.com/dropbox/lepton/blob/master/AUTHORS

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#38

Post by shuntress »

jacobsladder wrote: post #24
the_duke wrote: post #20

I really admire Dropox for open sourcing this, it shows their commitment. Saving almost a quarter of space for most images stored is something that truly gives a competitive edge. (I say most because people probably primarily have JPEG images). Especially considering how many images are probably stored on services like Dropbox. And they just gave it away to their competitors.

Then why do you admire them? Would you also admire them if you were their investor? Dropbox management is obligated by law to act in the best interests of their shareholders, i.e. to make them as much profit as possible. It's more likely that they have released it because of some profit-seeking interest. They are not charity.

"Dropbox management is obligated by law to act in the best interests of their shareholders, i.e. to make them as much profit as possible." - no they are not.

http://www.nytimes.com/roomfordebate/2015/04/16/what-are-cor...

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#39

Post by aidenn0 »

personjerry wrote: post #27

Could someone explain to me how compression algorithms work? Shouldn't they not work very consistently, by pigeonhole principle?

They work because while most possible data is unstructured, most real-world data is highly structured. On average over all possible inputs, any compression scheme must have 1x compression.

Consider images of slides for a presentation that are text on a flat background. If you know the value of the pixel just to the left of the current pixel, then if you guess that the current pixel will be the same, you will be right most of the time. This is obviously not true for random noise. Consider a really simple compression scheme where a pixel is stored as a single bit of a 1 if it is the same color as the previous pixel, and the color is stored as usual, but with an additional 0 bit prepended to the color. When you guess wrong, you pay a tax of 1 bit, but when you guess right you save N-1 bits where N is the number of bits per pixel.

For random noise, this will grow the input quite a bit, but for simple flat-shaded graphics it will shrink the input quite a bit.

Post reply on HN