21–30 of 196 posts
For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG.
If someone can do this over video files that will be PiedPiper comes into real life.
Does Dropbox do block or file-level deduping?
Block, it deduplicates based on 4MB blocks. Source: https://news.ycombinator.com/item?id=2478595
The backup tool bup (https://github.com/bup/bup) does this.
Great work. Ensuring bit-by-bit identical output and compressing an extra 22% is amazing. > For those familiar with Season 1 of Silicon Valley, this is essentially a “middle-out” algorithm. I really wish that every single compression-related blog post would stop referencing Silicon Valley.
I really admire Dropox for open sourcing this, it shows their commitment. Saving almost a quarter of space for most images stored is something that truly gives a competitive edge. (I say most because people probably primarily have JPEG images). Especially considering how many images are probably stored on services like Dropbox. And they just gave it away to their competitors.
It's more likely that they have released it because of some profit-seeking interest. They are not charity.
This is very impressive for archiving images. For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG. If someone can do this over video files that will be PiedPiper comes into real life.
This is very impressive for archiving images. For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG. If someone can do this over video files that will be PiedPiper comes into real life.
Could someone explain to me how compression algorithms work? Shouldn't they not work very consistently, by pigeonhole principle?
We live in a world where it is common for blue sky to occupy a portion of the frame and green grass to occupy another portion of the frame. Since images captured of our world exhibit repetition and patterns, there are opportunities for lossless compression that focuses on serializing deviations from the patterns.
I really admire Dropox for open sourcing this, it shows their commitment. Saving almost a quarter of space for most images stored is something that truly gives a competitive edge. (I say most because people probably primarily have JPEG images). Especially considering how many images are probably stored on services like Dropbox. And they just gave it away to their competitors.
Then why do you admire them? Would you also admire them if you were their investor? Dropbox management is obligated by law to act in the best interests of their shareholders, i.e. to make them as much profit as possible. It's more likely that they have released it because of some profit-seeking interest. They are not charity.
Pretty much every modern company safes tremendous amounts of money from open source (Linux and upwards in the stack), and so the OSS community should rightly be considered a STAKEholder.
The share vs stakeholder obsession in the space of large companies and corporations represents a lot that's wrong with our current markets.
--
Also, the only upside to open sourcing this is getting other involved in development. I just tested on 10k images, and the promise both on compression rate and bit parity after decompression holds true.
Seems to be a pretty stable product, so the that motivation is probably only miniscule.
This is very impressive for archiving images. For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG. If someone can do this over video files that will be PiedPiper comes into real life.
Lepton is cool because it helps make existing technology a whole lot better, but what we actually need is a better image format.
You wouldn't see the same leap for videos because people have been working hard to make those great for a while while JPEG has been left to rot.