Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lepton image compression: saving 22% losslessly from images at 15MB/s

blogs.dropbox.com

Post reply on HN
Print view View original

41–50 of 196 posts

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#41

Post by aidenn0 »

verytrivial wrote: post #34
daniel_rh wrote: post #28

Earlier quoted context omitted.

It's very easy to make a JPEG file that will not compress at all. Luckily it might look like snow from a television set rather than a typical image produced by a camera. We live in a world where it is common for blue sky to occupy a portion of the frame and green grass to occupy another portion of the frame. Since images captured of our world exhibit repetition and patterns, there are opportunities for lossless compr…

I think you are explaining how both lossy and lossless compression work without explaining how this does it differently. I can't read the article due to technical constraints, but understand that e.g. JPEG has a lossy quantisation pass followed by a lossless encoding/compression pass of the result of the first stage. If they're reproducing bit-identical result to the input JPEG, it must be a (very good) optimisation…

JPEG has a very poor lossless encoding stage; this is well known. Several archiving tools (most notably winRAR) already do some lossless reencoding of JPEG.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#42

Post by acdha »

jacobsladder wrote: post #24
the_duke wrote: post #20

I really admire Dropox for open sourcing this, it shows their commitment. Saving almost a quarter of space for most images stored is something that truly gives a competitive edge. (I say most because people probably primarily have JPEG images). Especially considering how many images are probably stored on services like Dropbox. And they just gave it away to their competitors.

Then why do you admire them? Would you also admire them if you were their investor? Dropbox management is obligated by law to act in the best interests of their shareholders, i.e. to make them as much profit as possible. It's more likely that they have released it because of some profit-seeking interest. They are not charity.

> Dropbox management is obligated by law to act in the best interests of their shareholders, i.e. to make them as much profit as possible.

Can you cite the law which you believe makes that obligation?

The reason you can't is because it's not actually a legal requirement and the reason is obvious: it's hard to say what the best interest is over all but the shortest time frame:

https://en.wikipedia.org/wiki/Business_judgment_rule

Dropbox's management might argue that they benefit more from open-source than they're giving away, that this kind of favorable attention will help them hire the top engineers who make far larger contributions to their bottom-line, that the pricing models are complex enough that this just doesn't matter very much, etc. Absent evidence that they're acting in bad faith, it's almost impossible to say in advance whether those arguments are right or wrong.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#44

Post by aidenn0 »

the_duke wrote: post #22
bmalehorn wrote: post #18

Earlier quoted context omitted.

Block, it deduplicates based on 4MB blocks. Source: https://news.ycombinator.com/item?id=2478595

I wonder if they use a rolling checksum too, to avoid duplicating a complete file if only a view bytes shifted (for example adding a line of text in the beginning of a file) The backup tool bup ( https://github.com/bup/bup ) does this.

They almost certainly do not, mostly because of how slow doing so is.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#45

Post by BrunoJo »

Jabbles wrote: post #31

It's interesting to do a cost analysis here: It's saved "multiple petabytes" of space. Backblaze storage is $0.005/GB/Month = $5k/PB/Month. The GitHub repo has 7 authors, perhaps costing Dropbox $200k/year each and taking most of a year ~ $1M to develop this system. So this might pay for itself after 200PB*Months, assuming Dropbox's storage costs are the same as Backblaze's prices, and assuming CPU time is free. (TOD…

Space is cheap but the bandwidth isn't. At Backblaze the download traffic costs $0.05/GB = $50k/PB.

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#46

Post by rhaps0dy »

XorNot wrote: post #23
mmastrac wrote: post #12

Great work. Ensuring bit-by-bit identical output and compressing an extra 22% is amazing. > For those familiar with Season 1 of Silicon Valley, this is essentially a “middle-out” algorithm. I really wish that every single compression-related blog post would stop referencing Silicon Valley.

They will once Silicon Valley takes them to task for it. Just like season 1 pretty much wiped "making the changes world a better place" from the lingo of SV companies.

> season 1 pretty much wiped "making the changes world a better place" from the lingo of SV companies

Did it really?

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#47

Post by stephenr »

neuronexmachina wrote: post #32
colechristensen wrote: post #30

Earlier quoted context omitted.

The reason this can be done with JPEGs is the compression hasn't been updated. There are folks that have used h264 for compressing images and webp uses vp8 compression, both with much better results than JPEG. Lepton is cool because it helps make existing technology a whole lot better, but what we actually need is a better image format. You wouldn't see the same leap for videos because people have been working hard t…

I'd love to have something like this for archiving DVD ISOs though, where the VOBs are compressed with old-school MPEG.

How does h.264/5 not fit the bill?

Re: Lepton image compression: saving 22% losslessly from images at 15MB/s

#49

Post by Jabbles »

BrunoJo wrote: post #45
Jabbles wrote: post #31

It's interesting to do a cost analysis here: It's saved "multiple petabytes" of space. Backblaze storage is $0.005/GB/Month = $5k/PB/Month. The GitHub repo has 7 authors, perhaps costing Dropbox $200k/year each and taking most of a year ~ $1M to develop this system. So this might pay for itself after 200PB*Months, assuming Dropbox's storage costs are the same as Backblaze's prices, and assuming CPU time is free. (TOD…

Space is cheap but the bandwidth isn't. At Backblaze the download traffic costs $0.05/GB = $50k/PB.

I don't think they're compressing this and decompressing it client side. At least I didn't get that impression from the article.

You're correct ofc, download costs = 10 months of storage.

Post reply on HN