Very cool - but what's the Weissman Score of this algorithm?
Lepton image compression: saving 22% losslessly from images at 15MB/s
51–60 of 196 posts
Re: Lepton image compression: saving 22% losslessly from images at 15MB/s
#52Re: Lepton image compression: saving 22% losslessly from images at 15MB/s
#53Could someone explain to me how compression algorithms work? Shouldn't they not work very consistently, by pigeonhole principle?
It's very easy to make a JPEG file that will not compress at all. Luckily it might look like snow from a television set rather than a typical image produced by a camera. We live in a world where it is common for blue sky to occupy a portion of the frame and green grass to occupy another portion of the frame. Since images captured of our world exhibit repetition and patterns, there are opportunities for lossless compr…
Impressive work, Daniel! Do I understand correctly that any image prediction for which the deltas are smaller in absolute value than the full JPEG/DCT coefficients would offer continued compression benefits? As in, if you could "name that tune" to predict the rest of the entire image from the first few pixels, the rest of the image would be stored for close to free (and if not, it essentially falls back to regular JPEG encoding).
If that's the case, then not only could we rely on the results of everything we've decompressed so far to use for prediction (which is like one-sided image in-painting), but we also could store a few bits of semantic information (e.g. from an image-net-based CNN, from face detection) about the content of the original image before re-compression, and use that semantic information for prediction as well via some generative model. All of this would obviously be trading computation for storage/bandwidth, but it this seems like an exciting direction to me. Again, nice work.
Re: Lepton image compression: saving 22% losslessly from images at 15MB/s
#54Earlier quoted context omitted.
I'd love to have something like this for archiving DVD ISOs though, where the VOBs are compressed with old-school MPEG.
How does h.264/5 not fit the bill?
I don't think h.264/5 is bit-for-bit retrievable is it? Neuronexmachina might be wanting to be able to get back the 1:1 version maybe?
Re: Lepton image compression: saving 22% losslessly from images at 15MB/s
#55Interesting that they are using VP8 to compress the JPEGs. This is one degree of separation away from Google's WebP [1]. It would be interesting to see how they stack up (WebP has a lossy and lossless mode).
Re: Lepton image compression: saving 22% losslessly from images at 15MB/s
#56Re: Lepton image compression: saving 22% losslessly from images at 15MB/s
#57Earlier quoted context omitted.
Space is cheap but the bandwidth isn't. At Backblaze the download traffic costs $0.05/GB = $50k/PB.
I don't think they're compressing this and decompressing it client side. At least I didn't get that impression from the article. You're correct ofc, download costs = 10 months of storage.
> I don't think they're compressing this and decompressing it client side.
The speed quotes made it sound like client-side was a concern. Why would you go to all the effort of devising a new image compression format saving 20%+ storage and on the wire, and not have it decompressed client-side, especially when you control the client?
Re: Lepton image compression: saving 22% losslessly from images at 15MB/s
#58I really admire Dropox for open sourcing this, it shows their commitment. Saving almost a quarter of space for most images stored is something that truly gives a competitive edge. (I say most because people probably primarily have JPEG images). Especially considering how many images are probably stored on services like Dropbox. And they just gave it away to their competitors.
Then why do you admire them? Would you also admire them if you were their investor? Dropbox management is obligated by law to act in the best interests of their shareholders, i.e. to make them as much profit as possible. It's more likely that they have released it because of some profit-seeking interest. They are not charity.
Even if that was their obligation (which it's not, as others have pointed out), there's no reason to believe that keeping the algorithm proprietary would be more profitable. They're not handing a critical capability to their competitors (competing with Dropbox isn't fundamentally about cheap storage). They don't have the infrastructure to licence the algorithm (who should they licence it to? For how much? Under which terms? Those are not simple questions to answer.)
Releasing it freely allows it to work its way into browsers, allowing Dropbox to serve up the smaller images directly to users, saving money on bandwidth. It allows other users to contribute improvements. It markets Dropbox as a desirable place to work for engineers.
Not obviously less profitable that the opposite.
Re: Lepton image compression: saving 22% losslessly from images at 15MB/s
#59Earlier quoted context omitted.
Space is cheap but the bandwidth isn't. At Backblaze the download traffic costs $0.05/GB = $50k/PB.
I don't think they're compressing this and decompressing it client side. At least I didn't get that impression from the article. You're correct ofc, download costs = 10 months of storage.
Why wouldn't they (at least, plan to)? They control the client-software.
Re: Lepton image compression: saving 22% losslessly from images at 15MB/s
#60Earlier quoted context omitted.
The reason this can be done with JPEGs is the compression hasn't been updated. There are folks that have used h264 for compressing images and webp uses vp8 compression, both with much better results than JPEG. Lepton is cool because it helps make existing technology a whole lot better, but what we actually need is a better image format. You wouldn't see the same leap for videos because people have been working hard t…
I'd love to have something like this for archiving DVD ISOs though, where the VOBs are compressed with old-school MPEG.
Why bother trying to losslessly compress your DVDs when they're already a low quality, compressed MPEG2 source? The small amount of content that isn't available in higher quality won't take up that much space left as is.